Back in 2014, Yahoo! servers were breached leveraging the Shellshock vulnerability. The breach was found by Jonathan Hall, then President of Future South Technologies which was a technology consulting firm focusing on multiple areas. One of those areas was security.
What Happened?
The breach of several large entities was noticed when servers from WinZip, Lycos, and Altavista appeared in the server logs of the public-facing servers for Future South Technologies. The IDS signatures picked up on attempt to exploit the Shellshock vulnerability through various vectors and triggered an alert to the engineering staff.
From there, investigation began and identified that WinZip.com had a vulnerable script that allowed command execution via the Shellshock vulnerability with a custom-crafted URL. This vulnerability was then used to identify a control script running on the WinZip servers that was responsible for the fuzzing attempts. Analysis on that script revealed that the compromised servers were part of a botnet on IRC.
At this point we connected to the IRC server and disguised ourselves with a username generated from the bot script the hackers were using. We also modified our IRC scripting to reply to certain ctcp queries so we would appear to be one of the infected machines and we began monitoring the channel.
We previously notified WinZip and they had already contacted us and began remediating the breach. They were already in the process of patching their hosts for the vulnerability and the servers impacted were just internal development machines with no real sensitive material on them - but they responded quickly, nonetheless, and had the issue resolved on the same day they were notified.
After just a few hours of monitoring the IRC channel our IDS alerted us to fuzzing attempts coming from several hosts from Yahoo.com, at which point we also noticed bots joining the hackers channel coming from their domain.
Notification was sent to Yahoo!
We immediately notified Yahoo! via email of the breach along with hostnames and evidence. We received a very shocking response that the discovery was not eligible for the bounty for bugs program which triggered a slightly agitated response from us. We expected no compensation to begin with, merely a response indicating they were taking the findings seriously. After hours passed without any change, and the channel communications beginning to indicate that the hackers were now targeting the Yahoo! games servers, the concerns of a widespread impact began to grow at this point. so we emailed the then CEO Marissa Meyer to inform her of the breach.
Things heated up.
Once we reached out to Marissa Meyer she immediately sent the mail over to her then CISO - Alex Stamos - who responded publicly in a manner that we didn't take very well. At that point in time they were very understandably focused on reputational damage control while simultaneously trying to understand the scope of the impact, coupled with the fact that every media outlet on the internet was running the story in a way that was pouring fuel on the fire and both Stamos and myself began to engage in a war of words in the public eye.
None of this helped either party. Our servers immediately became under full-blown attacks - DDoS and fuzzing attempts - from a slew of angry people from the Y!combinator forums. Yahoo! became a target of public scrutiny, berating their responses and treating the situation like a public boxing match.
So, could things have been done better?
The most important thing to understand in technology is that security is a constantly changing battlefield. At the time that Yahoo! was compromised, the Shellshock vulnerability was so new that countless other organizations were silently being hit by it and rapidly scrambling to repair the damages. We're talking about a vulnerability that was used in the most widely distributed binary on the planet, which had been sitting there dormant for more than a decade. There's absolutely zero doubt in anyones mind that select groups of hackers had long-known about this vulnerability and were actively using it in the wild.
In other words, while Yahoo! and WinZip may have been the ones caught with their pants around their ankles - much larger organizations out there most certainly hid and buried potential breaches from this vulnerability out of sight of the public eye.
The visibility of this issue did a substantial amount of reputational damage to all parties involved. Yahoo! took further reputational damage from both the incident and their response, and I personally suffered reputational damage because of the aggressive way I responded to their responses publicly, creating a visible argument on a topic that the communications could have been better handled until both parties had a more solid grip on what happened and what the actual impact was.
When a breach occurs, first understand the impact.
It takes time to analyze logs and data in order to understand fully what the impact of any breach is. When we put pressure on companies for a response then-and-there, this detracts from the ability to appropriately understand the actual impact of the breach.
Unfortunately, the way the situation blew out of proportion most probably detracted from the ability to properly analyze and understand what happened.
When something like this goes so viral and public, the pressure starts to build up for immediate answers. Those immediate answers may end up being statements that are incomplete, inaccurate and even misleading. This is not necessarily intentional but is a normal human response to being pressured and put on a spot where you have to say something even if you don't quite yet have something to say.
Issue a response appropriate to the impact level.
There's a major difference between someone hacking in to a host that merely hosts some API related scripts for news processing versus gaining access to a server or database housing sensitive customer data. The way these two scenarios would be handled by any company are vastly different.
If someone steals an employee cellphone and gains access to the mobile numbers of a few prominent figures, then this is unlikely to be publicly stated to the world that it occurred. It is likely that those impacted individuals will be notified and an apology given, but it serves no purpose to advertise the fact that it occurred to the world. This happens to companies on a daily basis and is an unfortunate event.
However, cases such as the theft of almost every American citizens social security numbers are a bit different and do require public awareness, as the breach is substantially larger in scale.
What did I learn from this?
While we still believe strongly in public disclosure, over the years we've learned that such disclosures need to be done responsibly and should also be proportionate to the problem. We've also learned that responses to any situation need to be calculated and not based off current emotion at the time.
This was a learning experience for everyone involved, and it's very unfortunate that it turned in to a public battle. It was never our intention and we solely wanted to prevent further breach due to the scale of potential impact. Over the years we've grown considerably and with that has come more professionalism and more empathy. If we could go back and do things differently, we would have kept the communications between us and Yahoo! private and given them the space and opportunity to continue their investigation uninterrupted by the constant media pressure.
For this, I personally appologize. I've grown as a person and as a businessman, noting that I was only in my extremely early twenties at the time. My exposure to larger corporate environments did not exist yet, nor my understanding of the red tape and the massive scale of data and infrastructure footprint that accompanied them. This has quite obviously changed over the past decade. With that decade, I've become wiser and I've grown up.
