More Evidence That Rupert Murdoch’s Outrage at Google Is Phony

On Friday I wrote how Rupert Murdoch and other newspaper execs are pretending to be upset that search engines like Google link to news stories on their Web sites. Murdoch claims Google is "stealing" his content, but he knows the truth is that Google sends a ton of traffic his way that he wouldn't otherwise get, and the last thing he really wants is for Google to stop doing that. Friday's posting pointed out that anyone can easily block Google. All you have to do is go to your Web site's robots.txt file and type in two short lines. After that, your site will no longer appear in Google's search results. It's so simple, Murdoch could do it himself. Google will even help him—there's a page on its Web site explaining how. Of course, Murdoch is no fool. He knows he can do this, but he doesn't because it would cost him readers, and money.

Now comes the next beat: the folks over at Mike Masnick's must-read blog Techdirt.com picked up on my piece, and a smart reader in the comments section advanced the story. He went to Murdoch's FoxNews.com and took a look at the site's robots.txt file, to see just how far Murdoch goes in his attempts to stop Google from raiding his cupboards. This is what he found:

User-agent: gsa-crawler
Allow: /printer_friendly_story
Allow: /google_search_index.xml
Allow: /google_news_index.xml
Allow: /*.xml.gz
#
Sitemap: http://www.foxnews.com/google_search_index.xml [foxnews.com]
Sitemap: http://www.foxnews.com/google_news_index.xml [foxnews.com]

You don't need to be a code monkey to understand what's going on here. As the commenter points out, not only is FoxNews.com not doing anything at all to block Google, the site explicitly points Google to its news stories. To borrow Murdoch's robbery metaphor, it's as if he left the front door of his house wide open, and planted a huge sign on the front lawn that says, "Please come in and take anything you want. The valuables are in the bedroom." So much for Murdoch's faux outrage.

What seems to get Murdoch and other legacy news execs so crazy is that on the Internet, they no longer have total control over their stuff. Murdoch made his pile when newspapers were virtually monopolies and he could extract large sums from readers and advertisers. Now innovation and competition have entered the marketplace, and readers and advertisers have a wealth of new choices—and bargaining power—that didn't exist before. Murdoch is, understandably, not happy about it.

But Murdoch's desire for old-style control is not only futile on the Web, it's bad business. The way to profit online isn't by trying to hold back the chaos of the Internet by hiding your stories or putting up paywalls—another thing he is threatening to do—but by embracing the chaos and letting it work for you.

Take the chain of events that led to the blog post you're reading right now. I wrote up that first item smacking Murdoch on Friday afternoon, and sent it into the ether. Many other Web sites and blogs picked up on the item, because that's how, you know, the Internet works.

Now, a lot of the bloggers who took note of my piece cut and pasted huge chunks of it onto their sites, sometimes with extra commentary of their own, but not always. A few of them just took the whole thing and dumped it onto their page. All of the blogs that picked up the story included a link back to the item here at Newsweek.com, but I'm sure many readers didn't bother to click on the link and read my original post, since they got the gist of my story from reading the "stolen" version on the blog where they found it. On many of the sites, my work was surrounded by their ads.

Should I be angry about this? The Murdoch way of  looking at it is that NEWSWEEK lost thousands of readers because these other sites piggybacked on NEWSWEEK's content, and even placed ads against it. They robbed us!

That would be bad, if it were true. But the better—and I think more accurate—way of looking at it is that those "thieves" made it possible for thousands and thousands more people to read my piece than otherwise would have.

If Murdoch's Web world view prevailed, I'd have posted my piece and hoped that people would come to NEWSWEEK to read it. That's the way it was when newspapers ruled. You had to buy the paper to see what was inside. But that makes no sense now. Instead, I had hundreds, if not thousands, of strangers working to promote my story for me, for free. And even though many readers who discovered my piece on some other site didn't click through to NEWSWEEK to see the original story, thousands of them did. The overwhelming majority of readers who did read my piece on Newsweek.com did so because they learned about it on a site that linked to it—including Slashdot, Techmeme, Fark, and, yes, Google.

None of those people would have read—or even known about—my story if NEWSWEEK made it hard for aggregators and search engines to find it and promote it on their sites. Do I deserve a portion of the ad dollars those other sites might have made off my story? Not unless they deserve a portion of my ad revenue as a finder's fee for sending me readers I wouldn't have had without them.

And inviting the "theft" of my work had other benefits. It led to that Techdirt commenter improving on my original story by adding interesting new information.

So let's review: I can either have 100 percent control over a small, closed market—which means everyone reads my stories on my site, but there are fewer readers coming to my site overall—or I can have far less control over a much larger market, which means many people will wind up reading my work on other people's sites, but there will be many more readers coming to my site overall. Oh, and by keeping the doors open, I have the benefit of a continuous stream of new readers coming to check out my stuff, which they learned about elsewhere—something I won't have with a locked-up site that not as many readers will find.

One last point: In this light, Murdoch's complaints seem especially ill-considered because in the case of Google, we're not even talking about other sites lifting his content. Google just links to his stories. Readers who click on those links are sent to read them on his Web sites, surrounded by his ads. So when it comes to Google, he gets all the free traffic, and all the eyeballs on his ads. And again, if he doesn't want all that rich delicious goodness, he can shut off the spigot at any time. So just what is it again that he's complaining about?