09.21.10

The Race to Save Twitter

The security breach that nearly brought Twitter to its knees Tuesday sparked a guerilla effort to save the network by a handful of tech geeks who noticed it first.

It was a minute after 8:30 a.m. Tuesday morning when digital security at the White House appeared to have been breached. President Obama's Press Secretary, Robert Gibbs, had just tweeted a string of code—JavaScript, to those who recognized it, and jibberish to those who didn't. It contained words and phrases like "onmouseover," "this.innerHTML," and "submit()." Not exactly your standard Gibbs politicking fare.

His followers reacted quickly, alerting the Secretary that his account was going crazy, and recommending ways he could fix it. Moments later, Gibbs sent a follow-up tweet:

"My Twitter went haywire—absolutely no clue why it sent that message or even what it is...paging the tech guys..."

"My Twitter went haywire—absolutely no clue why it sent that message or even what it is...paging the tech guys..." said White House Press Secretary Robert Gibbs.

Unbeknownst to Gibbs, similar code-laced tweets were breaking out all across Twitter Tuesday morning, on the accounts of everyone from the barely followed to the power-users with 1,000,000+ loyal fans. The entire network, it seemed, was under some kind of malicious attack, and seemingly any and all accounts were vulnerable. In fact, according to TechCrunch, approximately 40,000 tweets sent in the space of 10 minutes contained the nasty code. The only way to stay safe—aside from shutting down and just going outside, of course—was to load up a third-party Twitter client, such as Seesmic, TweetDeck, or as Twitter later explained, its mobile site.

Soon, the news broke via CNET, Mashable, and others that the tweets were the work of a 'rollover vulnerability' within Twitter itself, meaning that your computer could be afflicted should your mouse simply roll over the infected text—you didn't even have to click to become a victim.

This "onmouseover" command, as it's called, meant people accessing Twitter could have their accounts briefly hijacked. For some users, this meant having a pop-up box launched on their screens. Other victims were involuntarily zipped over to external websites—some of them containing Japanese pornography.

Many of the offending tweets were covered in a dark square box, hiding their content. Users began ominously referring to them as the "Black Tweets."

"A new Twitter security flaw has been widely exploited on thousands of Twitter accounts, redirecting users to third-party websites without their consent," wrote Stan Schroeder of Mashable in a blog post Tuesday morning, as the tweetable terror was still rapidly spreading. "The bug is particularly nasty because it works on mouseover only, meaning pop-ups and third-party websites can open even if you just move your mouse over the offending link."

But hours before the news broke in the U.S., a few select Twitter users in the UK had already been working to stop it from spreading—of course, while playing around with its code.

It was around 11 a.m. UK time (that's 6 a.m. EST and 3 a.m. Twitter HQ Time)—three hours before Press Secretary Gibbs' sent his chaotic, infected tweet—when a Quality Assurance (QA) employee named Daniel Bennett, busily testing software in IT, first noticed a strange tweet in his feed. Alongside the code, it read simply: "this could be an issue."

Bennett's not 100 percent sure, but he believes that the individual who sent this tweet was the first person to see the attack—the so-called Patient Zero. "I think [that tweet] was the person that first found it." he told The Daily Beast.

Bennett, 20, kept a close eye on the developing situation—it wasn't very widespread yet. In fact, as far as he could tell, he was one of only three people in the world who were aware of it.

To confirm the existence of the vulnerability—and explore its potential for disaster—Bennett played around to see how bad this could be. As he tells it, it "turned out very bad."

Using a simple JavaScript command, he drafted a 110-character line of code that, when activated, launched a pop-up box with the message: "I HAX YEW <3." ("I hacked you.")

"That's an ace bug," he said.

"[I] made a few test tweets, one on mouse over for alerts, one for a Rick Astley song," -- a prank he later explained was known as "tweetrolling"—"and then one going to my blog explaining there is an issue. The others I deleted. Someone following me [retweeted] them because they found them funny. Not many though, and as I deleted the tweets, I don't think it went far."

Then, he started warning the others.

"I tried to get out there to warn people from the black tweets straight away in case it all got hectic," he explained.

He then reported the breach to Twitter by sending a message to two of their public-facing accounts—but he says that was it; there would be no confirmation or response from the California-based company for some time. "Sent a message to @ twitter and @ twitterapi saying they really need to fix it ASAP, [but] heard nothing," he told me in an email. "All was asleep in USA at the time."

But perhaps Twitter did receive the transatlantic transmission after all. In a blog post explaining the "mouseover" incident, Bob Lord, a member of the Twitter security team writes, "This morning at 2:54 am PDT (that's about 10:54am in the UK) Twitter was notified of a security exploit that surfaced about a half hour before that, and we immediately went to work on fixing it."

"To their credit," adds Beth Jones of Sophos, a UK-based software security firm, "Twitter reacted very swiftly to this attack and closed up the hole…I do hope they are actively looking to see if there's any additional scripting holes."

But the crisis was far from over. Not long after Bennett and this gang of three began warning of the exploit, Bennett noticed a more "hurtful" version of his code was being used for more malicious action. That original tweet that he had left up to warn his followers about the bug had been copied by a spammer, or worse, a hacker, and was now spreading rapidly throughout the network.

"The person had left my Twitter URL in there and basically changed it so it got a script from a remote site and ran it," he said, referencing the new malicious code which kept his website's URL in it, yet called upon another.

It was around this point that Graham Cluley, an IT security blogger at Sophos, first got wind of the vulnerability. It was about noon in the UK. "A Twitter follower asked me if I'd seen the 'onmouseover' spam," he told me in an email, referring to the code that appears in the tweet, "and I quickly realized that something unpleasant was afoot."

Initially, he explained the examples he saw as more mischievous than malicious—a claim Twitter's Bob Lord backed up in the blog post explaining the issue. But he was primarily worried the simple hack could be exploited by others. Then, he saw that Sarah Brown, the wife of ex-prime minister Gordon Brown, "and something of a Twitter celeb in the UK with over a million followers," had been hit too at her account @ SarahBrownUK.

Cluley quickly uploaded a video to YouTube documenting the outbreak, titled it "Twitter onmouseover security vulnerability widely exploited," and then posted a corresponding blog post outlining the vulnerability. This was later referenced in a majority of the early reports on the attack. In it, Cluley referenced Bennett's account—in both the video and as a screenshot on the blog -- as one of the early users exploiting the bug. The Huffington Post did the same.

Naturally, this led a number of people to assume Bennett himself was to blame.

"[It] seemed like Sophos blamed me, as their website shows my page (censored though) on the blog and my page on their video on YouTube (uncensored, which made me a bit gutted)," he said. He added Cluley now believes it wasn't him, "so thank god!"

But no good deed goes unpunished, and quite a few people still accused Bennett of being responsible. "A few followers blamed me straight away. All I could do was say 'I didn't' and give them the link to explain what it was."

As for who created the worm that spread more maliciously, The New York Times reports it may have been launched by a Norwegian programmer named Magnus Holm, whose tweet "contained only a link" that, when one's mouse went over the text, automatically tweeted it onwards, thereby spreading it throughout the network.

Holm is claiming responsibility, saying he did it as an "experiment with the flaw" in Twitter. But unlike Bennett and the others, he chose not to tell Twitter about its vulnerability. "I guess I should have reported the hole," he tweeted about an hour after allegedly launching the worm, "but when I discovered it, it had already been in the wild for some time, so I assumed they knew it."

By 10 a.m. EST, the hole had been closed. Addressing the damage, Bob Lord explained the hack was enabled by an exploit of a specific vulnerability known as cross-site scripting (xss), or "the practice of placing code from an untrusted website into another one."

In Twitter's case, Lord explained, the code was submitted as standard text in a tweet, which could then be executed in the browser of another user—hence the pop-ups & redirects. As for the virality of the attack and its spread, he writes that some users created an additional step, adding "code that caused people to retweet the original Tweet without their knowledge," possibly referring to the programmer contacted by the Times.

"There's no reason that should have ever been allowed," says Jones, of Twitter allowing JavaScript in the tweets.

"I'm truly amazed that this XSS stuff happened," agreed Bennett, the software tester. "It happened before and they fixed it. Apparently a code change made it happen again," he said.

"Do they not have QA?"

Brian Ries is a Philly-born senior editor at FREEwilliamsburg.com and tech and social media editor at The Daily Beast. He lives in Brooklyn.