Heartbleed
SparkFun has reacted quickly to protect ourselves and our customers against the Heartbleed vulnerability
Heartbleed, or CVE-2014-0160, is a pretty serious vulnerability in OpenSSL, one of the more popular libraries for encrypting communications on the internet, and its exposure this week has the internet on high alert.
You can read about the details behind the bug at Heartbleed.com, but here's how it works in a nutshell: a couple years ago OpenSSL got a "heartbeat" feature which allows computers and servers with secure connections to each other to "ping" each other regularly to keep the secure connection open. A bug in the heartbeat feature allowed for computers without secure connections to still get a response this way, and that response could be exploited to include chunks of data from the server's RAM, which could include all sorts of recently-decrypted stuff like passwords.
What We Are Responsible For
As soon as the vulnerability was exposed to the greater internet on April 7th we sprang into action along with the administrators of sites great and small, worldwide. It's our responsibility to first patch all SSL libraries on all servers (to "stop the bleeding" as it were). Next we revoke and reissue all SSL certificates as the private keys may have been compromised. With some pressure on our certificate authority we've got fresh certs inbound and as soon as those are in place, likely sometime this afternoon, we'll dump all sessions on SparkFun.com as those are also at risk of having been compromised.
This means every SparkFun customer will be logged out and any customers who have built a cart without signing in will lose that cart. However from that point on when you sign in to SparkFun.com you'll be doing so over SSL using a new certificate where there's no risk of the private keys having been leaked. But there's still more to be done...
What YOU Are Responsible For
First of all, don't log into any websites until you know they're patched against this bug. Most major websites have already responded, and the patch isn't very complicated so there's no excuse not to respond. There's a utility here that can help you determine if a given website has taken some of the necessary steps for protection.
Secondly, it's about time to reset all of your passwords. Seriously. This vulnerability existed for the better part of two years and was only just exposed on Monday. If your account credentials somewhere were slurped up using this exploit at some time in that past and yesterday that site patched against Heartbleed those attackers still have your credentials. Change your password, and also consider setting up two-factor authentication for any sites or services that offer it.
What Happens Now?
This story has a lot of interesting angles. It's arguably one of the biggest security vulnerabilities in the entire history of the internet. It's technical but not terribly so, so I'm curious to see how it is covered in the main-stream media.
In the netsec community Heartbleed is already the highlight of the year. Flurries of discussions on various fora are churning right now on how this happened, how to react, what the long-term impacts are, etc. A family member who works as a penetration tester (someone who's paid to steal your data and tell you how it was done) summarized the reaction from the offensive side of the network security community thusly.
The Business Side
Another angle worth mentioning is the certificate authorities, or CAs. SSL certificates can be generated for free using open source tools, but when done so they are "self-signed." The entire SSL model relies on certificates to verify a website is who they say they are, and a self-signed certificate (while functional) doesn't provide any confidence in that. This is why browsers warn you when a cert is self-signed. Certificate authorities are companies like Verisign and Comodo that build a business on confirming people are who they say they are and then signing their certificates. This can be expensive depending on how iron-clad you want that certification to be. At SparkFun we pop for the Extended Verification, or EV certificates which can require months of investigation on the CA's behalf to confirm who SparkFun really is.
The curious thing about this vulnerability is that it requires an estimated 60-70% of all certificate holding websites across the internet to revoke+reissue or renew their SSL certificates. Revoke+reissue is free with our CA but renewal isn't. If any sites unaware of their ability to revoke+reissue, or if a CA charges for that service, this could be a huge pay day for the CAs. That raises ethical questions about where their incentives lie... does it make good business sense for Comodo or Verisign to quietly encourage similar vulnerabilities in the future? Is a system like that ultimately sustainable? In fairness it's also potentially a burden on the CAs as their volume for issuing certs has skyrocketed overnight. Time will tell how they react in the wake of Heartbleed.
The Open Source Side
We harp a lot about the virtues of open source here. This vulnerability, having appeared in an open source SSL library (OpenSSL) allowed for the netsec community to provide line-by-line diagnoses of the flaw within hours of its general exposure. Furthermore, open source is not about things being open now but things being open over time, so it's been possible to peer deep into OpenSSL's history to see how the flaw was introduced and evolved over time.
I've seen some chatter already about how this was the net effect of poor programming from an amateur open source development team. The quality of the programming (and arguably the review process) and the open source nature of the library are two completely different aspects that should not be conflated, however. A proprietary SSL library developed behind closed doors could have easily introduced the same flaws. The open source nature of the library may have made it easier for attackers to craft exploits against the heartbeat feature, but it's likely that a similar feature+flaw in a proprietary library would have been compromised the same way. The internet's most skilled and nefarious are never slowed down much by working with compiled binaries as opposed to source, and security through obscurity is widely stigmatized for good reason.
Ultimately the open source nature of the library that introduced the flaw has vastly aided the community in assessing the damages inflicted and mount a swift response. Regardless, the pessimist in me still expects to see "open source" taking some undue blame for this fiasco.
Protect Thyself!
So that's Heartbleed in a nutshell from SparkFun's perspective. I'll close with a reminder to protect yourself. Use strong passwords that don't repeat, pay attention when your browser is warning you about an insecure website, and use 2-factor authentication wherever you can. Stay safe and have fun. =)
UPDATE: 2014-04-09 15:00 MST
We've received and installed our reissued certificates, so we've dropped all browser sessions as an extra precautionary measure.
Congratulations Sparkfun. You are the FIRST site I've come across that has posted a notice on their heartbleed status, described the actions they have taken, etc.
As usual, You Folks Rock.
I concur, this is the first I have heard about it!! No contact from Google saying their servers may have issues etc.
This was the first place I heard about heartbleed as well. This video is a good explanation of what it exactly is.
or
todays XKCD
incase you don't like watching videos
+1
Not changing all my passwords. If they had my passwords they would have hacked my accounts already... This thing has been going around for two years...
Last night I wanted to see just how serious this was in the wild, and I can confirm with absolute certainty that it is the worst I've ever seen. It's extremely trivial for an attacker to pick up your credentials from POST data off the heap from nginx/apache, and I'd run under the assumption that there was widespread datamining happening hours, if not minutes after the CVE was released.
It's been a bug for quite some time. It remains an open question whether it was an exploit known to anyone in the wild before Monday's disclosure.
I'm not changing every credential I have on the web; there are so many at this point in my life that I probably don't even remember half of the places I have accounts. I am changing the stuff that touches my working life, finances, or public identity and might conceivably have been vulnerable.
This is a serious one. A little paranoia may be warranted.
Press about the threat makes it more of a threat. It is possible that a few select individuals or groups knew about this before, but I expect there was more activity in the few hours after it became wildly known than there was in the previous two years.
This is one of the best write-ups I have seen on this yet. Straight to the point and practical, yet enough technical description to be understood by people who can. You guys at SF have done a great job of staying on top of this. Thanks!
I wanted a Facebook button to share your awesomeness with my friends :(
i still see you certificate issued on 08/01/12... this means it still unsafe, or i'm missing something?
That's not actually the issue date— it's the "not valid before" date. (A lot of browsers describe it as an issue date, since it's usually set to the day the cert was issued, but the browsers are wrong: gory technical details here.) It's 100% reasonable, for example, to get a new cert today that isn't valid until next week when I plan to install it, or something like that. Certificates don't actually contain an indication of when they were issued. They definitely don't contain an indication of when their underlying keypair was generated, which is what we really want to know.
With all the heartbleed-related revocations it's become clear that different CAs do things differently when replacing a compromised key's cert. Comodo and DigiCert seem to produce new certs which have the same validity period as the old cert; Thawte produces new certs which are only valid from the time of reissue but have the same end-date as the old cert. (Thawte's behavior seems more correct to me, but I'm sure the folks at DigiCert have a reason for doing it the way they do.)
To actually know whether a vulnerable organization has responded safely, you'd probably have to find their old certificate via something like the SSL Observatory, and then verify that the old cert has been revoked and replaced by a cert using a different keypair.
This is normal - our certificate was reissued, and the old one revoked, which means the issue date and expiry date will stay the same.
Edit: We normally utilize what's known as an EV certificate (extended validation), however the reissue for it takes several days to several weeks, so we've fallen back to a traditional, non-EV one. Security wise, they're the same, but EV will give you the green bar that you see on some sites with the organization's name. Once Comodo grants our reissue and revocation request, we'll replace our certs once more.
ah ok, but if i can't trust the "issued on" date, how can i check if the site owner's had reissued their certificate? iknow some site doesn't need a reissue as wan't using openssh, but... i feel beeter knowing it has been done
Not a SparkFun deficiency directly and Indeed it shows 'Tuesday, July 31, 2012 5:00:00 PM' at this time (4/11/14 PM). Would have expected these things would be internally consistent and accurate - any such information has to be correct or it diminishes the point I ever validating it or blindly trusting it.
Though I did just follow up on another shopping site and it has a certificate dated 4/8/14 which suggests they corrected other issues as well - even though they don't have a banner of details.
just checked that little store 'amazon' and saw this issue date: Wednesday, February 26, 2014 5:00:00 PM
Unfortunately, you can't without having a copy of the old certificate to check against the revocation list. In my findings, it was actually extremely difficult, and usually impossible to get this exploit to disclose any key material unless exactly the right circumstances existed. More than anything, what was at risk is the encrypted data that was being sent to and received from servers.
Over the last two years, it's somewhat unlikely that this was actively being exploited, but just about anything you've done in the last 3 days anywhere on the internet should be considered entirely compromised. If you've had any active sessions on any sites, you should logout (someone can 'assume' your session to get into your account), and change any passwords you may have used in the last week. This includes major sites such as Facebook, GitHub, Indiegogo, etc, although I know most large organizations are aware of this threat, and have manually reset all active sessions in order to mitigate it.
You must have been checking mere minutes before we installed our new certificates (see the update above).
i STILL see it, even in anonimous mode..
edit: tested another browser, still issued on 08/01/12..
The Open Source communities are Big Boys now. They surely can take the FUD like anyone else. Thanks for getting the word out. I took some measures to protect some stuff of mine as a result. I think big credit card processors need to get the hint that storing credit cards is a bad idea. Paypal and Amazon are two of the ones I am thinking of. I exercise the option to delete that and deal with having to manually put it in every time.
On sites I use infrequently I ask not to save as well. But the nature of this SSL exposure means it was actually more at risk the more often you typed it in. If 'they' keep it safe in their 'database' then it is never transferred for display on each web transaction.