Part 1 of this story can be read here.
As good as Google is at providing information, it should not and must not be allowed to conduct business as usual. It must and will eventually be regulated, just as the phone companies and credit bureaus are regulated. Fundamental civil liberties issues are at stake.
A frustrating experience I had with Google early in 2012 illustrates in a narrow context why we need to start looking at the company in a different light. The experience opened my eyes to the real threat that an unregulated Google poses -- one so enormous that you just can't see it, like the curvature of the planet.
Here is what happened.
My Battle with Googlebots
On New Year's Day, 2012, I awoke to nine emails from Google informing me that my professional website, DrRobertEpstein.com, had been hacked. My site had been hacked before, but this was different. Last time, a few obvious lines of code had been added to a key program. The bogus code, which took only seconds to remove, had enhanced my home page with links to websites selling Viagra. This was not a big deal. Everyone gets hacked these days, even (in October 2011) Google, Facebook, and Microsoft.
This time, however, the hack appeared to be deep and deadly, affecting not only my main website but the twenty or so websites at which I provide psychological tests, such as DoYouNeedTherapy.com and MySexualOrientation.com. For all practical purposes, this hack shut down all of my websites for more than a week.
It turns out, though, that the bad-guy code was only part of the problem. The other culprit was Google itself.
Google has an awesome power, you see, that few people are aware of -- the unfettered, unregulated power to effectively shut down any website in the world by adding that website's URL to a blacklist. In my case, this power was abused.
Before I give you the details, I should point out how generally helpful I find Google to be. Google's search engine not only gets me where I want to go, it also allows people around the world to learn about my own scientific and scholarly work. As of January, 2012, when people searched for me on Google, the company pointed them to more than 450,000 web pages. Incredible! Flattering!
And powerful.
Imagine if Google suddenly erased me from its database. Relatively speaking, I would almost cease to exist. (Will this happen, I wonder, when Larry Page, Google's CEO and co-founder, learns about this article?) Information is power -- more so in today's world than ever before, and whoever provides or restricts access to large volumes of information has the ability to alter the attitudes and behavior of billions of people.
Which brings me back to New Year's Day....
Arrogant and Unresponsive
Google's missives said that my websites were redirecting visitors to a malicious website in India. In other words, my sites were not themselves malicious; they simply redirected people to a bad place, like the traffic cop who sends you off on a long detour through a scary neighborhood. Google's emails also provided links to webmaster tools that allow users to see what Google's automatic crawler program -- a Googlebot -- has found wrong with their websites.
At this point, Google was limiting access to my websites in two ways. First, if you searched for me or most anything related to me using the Google search engine, every result was tagged with a formidable warning: "This site may harm your computer." If you were foolish enough to ignore the warning and click through, you got an even stronger warning, typically a page containing a blood-curdling red box declaring that "drrobertepstein.com has been reported as an attack page" (which it was not) and featuring an irresistible button reading "Get me out of here!" Depending on your browser and security settings, you might then have been at a dead end, or you might still have been able to click through--which virtually no sane person would do, of course.
Second -- and this is important -- if you didn't use Google's search engine and simply tried to go to any of my websites directly or through a link on another website, most computers and most browsers would also block your access, again displaying that intimidating red box. This would occur even if you were just trying to download articles from my main website without actually entering the site.
Now think about that. It makes sense that Google's search engine should, on its search list, warn people about a website that its crawler has found to be dangerous, but why does Google, a private company, have the power to block access to a website that you want to visit directly, and how, technically, is Google accomplishing the block? Is someone from Google looking over your shoulder as you type?
Working with local colleagues, with my Internet service provider (Aplus.net, a division of Deluxe), as well as with the programmer who had defeated my earlier hack, I soon learned some interesting things. First of all, Google's own analytical tools indicated that my home page was clean, even while its crawler was reporting that the source of the redirect was a link to a non-existent page on my website. A non-existent page? How could a user ever get to a page that doesn't even exist? Are they supposed to use ESP to figure out the URL? We also learned that when it was possible to get past Google's blocks, visiting my websites directly never resulted in a redirect to a malicious site.
On January 3, still unable to find the hacked code and still getting contradictory information from the webmaster tools, I emailed "webmaster-central-help" at Google, asking Google to stop blocking my websites. In return, I received what appeared to be a computer-generated email referring me back to the webmaster tools. I replied with a more emphatic message, this time, in order to attract attention, with a copy to a New York Times reporter who had once interviewed me for a story on artificial intelligence.
The subsequent reply I received may have been written by a person, but it was still unsigned and contained no more information than we had found with the automated tools.
Calling Google was even more frustrating. The courteous woman who answered the phone was allowed to give me her ID number but no name, which seemed to frustrate her as much as it frustrated me. She said the department that could help me was the "web search department"--that was helpful--but that "they don't take calls." Neither, she said, does Mr. Page's office, nor her supervisor, nor even the public relations office. Why, I wondered to myself, does Google even have a phone? Just to tell you that they don't take phone calls?
That evening I was contacted by Nicole Perlroth, a New York Times blogger who specializes in computer security issues. On January 5, she posted a sympathetic article about my situation entitled, "One Man's Fight With Google Over a Security Warning." She also interviewed John Harrison, a project manager at Symantec, who acknowledged that my dilemma was becoming increasingly common, with 40 million attempted hacks every day and the websites of physicians, news organizations, and "even a Fortune 1,000 company" being successfully infected.
But comments on the article generally defended Google as a fatherly company that "protects us from the evil that lurks on websites" (actual quote). One writer accused me of "maligning" Google, which was "acting in the best interests of you and your clients." A few writers were more skeptical, however, wondering, for example, why Google won't put employees' names on emails or help people over the phone.
Fortunately, the article also drew the attention of some highly-skilled security experts, including a systems analyst in Southern California who found the troublesome code -- 20 crafty lines that had been added to a configuration file. We fixed the problem on January 6, and Google finally removed the blocks about 30 hours later--except, that is, on DrEpstein.com, my primary link. Even though it contains no content (it's just a forward to DrRobertEpstein.com), it took an additional three days for Google to clear it. Empty links that do nothing but forward to other links are apparently too subtle for Googlebots to understand.
Problem solved, right? Not at all, because this incident exposes a much larger problem, one that goes well beyond my own New Year's headaches.
Continues in Part 3: Google as Self-Appointed Internet Censor...
Robert Epstein is Senior Research Psychologist at the American Institute for Behavioral Research and Technology and the founder and Director Emeritus of the Cambridge Center for Behavioral Studies in Massachusetts. The former editor-in-chief of Psychology Today, he has published fifteen books, including a 2008 book on artificial intelligence called Parsing the Turing Test: Philosophical and Methodological Issues in the Quest for the Thinking Computer. You can view "United States of Google," a recently archived video discussion on this issue involving Dr. Epstein, Bianca Bosker (Senior Tech Editor at the Huffington Post), Pete Pachal (Tech Editor at Mashable.com), and others by visiting HuffPost Live here.