I <3 Steve McConnell*
Coding Horror
programming and human factors
by Jeff Atwood

January 11, 2007

If It's Not in Google, Does Your Website Really Exist?

Rich Skrenta, who may have written the first microcomputer virus, calls Google the start page for the Internet:

The net isn't a directed graph. It's not a tree. It's a single point labeled G connected to 10 billion destination pages.

If the Internet were a monolithic product, say the work of some alternate-future AT&T that hadn't been broken up, then you'd turn it on and it would have a start page. From there you'd be able to reach all of the destination services, however many there were.

Well, that's how the net has organized itself after all.

From this position, Google derives immense and amazing power. And they make money, but not only for themselves. Google makes advertisers money. Google makes publishers money. Google drives multi-billion dollar industries profiting from Google SEM/SEO.

Most businesses on the net get 70% of their traffic from Google. These business are not competitors with Google, they are its partners, and have an interest in driving Google's success. Google has made partners of us all.

But what happens when the start page for the internet-- the source of more than 70 percent of your traffic-- decides it will no longer index your web site?

Google blur

That's exactly what happened to JavaLobby over the christmas break:

It had been aggravating to spend holiday time cleaning up the unwanted [50,000 spam forum messages], but the real problem didn't surface until we started going through our normal morning routine yesterday, having just returned to work from our holiday break. We generally take a look at a variety of statistics in the morning before proceeding into whatever development work we're doing. Having been out of the office for almost two weeks, we had a lot of stats to look at. It took no time to see that something was wrong - traffic was down. A little more investigation revealed the problem.

We had completely disappeared from Google's main index! If you run a website, then you know how serious a problem this is. On any given day over 10,000 visitors arrive at Javalobby as a result of Google searches, and suddenly they stopped coming! We had apparently been grouped together with the spammer's viagra and casino sites, and poof! Suddenly we no longer existed in the eyes of Google, the world's largest search engine. Countless thousands of well-ranked pages gone in a blink. Perhaps you now understand why I would commit a violent crime if I caught those forum spammers? In essence, they have wiped out strategic positioning that we took years to build.

Google's response in this situation is arguably justified. They can't have query results redirecting users to sex sites; the public good requires that rogue or defaced websites get removed from their index as soon as they're discovered. Google's Matt Cutts, in response to a similar Google delisting incident posted on Slashdot, wrote an entire blog post documenting exactly how Google handles hacked websites:

But let’s take a step back. This site was hacked and stuffed with a bunch of hidden spammy porn words and links. Google detected the spam in less than 10 days; that’s faster than the site owner noticed it. We temporarily removed the site from our index so that users wouldn’t get the spammy porn back in response to queries. We made it possible for the webmaster to verify that their site was penalized. Then we emailed the site, with the exact page and the exact text that was causing problems. We provided a link to the correct place for the site owner to request reinclusion. We also made the penalty for a relatively short time (60 days), so that if the webmaster fixed the issue but didn’t contact Google, they would still be fine after a few weeks.

Ultimately, each site owner is responsible for making sure that their site isn’t spammy. If you pick a bad search engine optimizer (SEO) and they make a ton of spammy doorway pages on your domain, Google still needs to take action. Hacked sites are no different: lots of spammy/hacked sites will try to install malware on users’ computers. If your site is hacked and turns spammy, Google may need to remove your site, but we will also try to alert you via our webmaster console and even by emailing you to let you know what happened. To the best of my knowledge, no other search engine confirms any penalties to sites, nor do they email site owners.

I had completely forgotten about Google's Webmaster console until Matt mentioned it. If you own a website, you should take advantage of these tools. They'll let you diagnose and fix most Google-related problems. On top of that, they'll even give you some basic stats on the search queries people used to get to your website. All you have to do is prove ownership of your website by either uploading a specially-named file, or modifying a page to include a specific META tag.

But let's put aside, for a moment, the fact that the webmaster response to Google's delisting was a little hysterical in the face of Google's excellent webmaster tools. Can you blame them? You'd probably be upset, too, if more than 70 percent of the website to your traffic disappeared overnight.

That's the truly scary part. Google's lead over its competitors is so complete, so total, that if your website isn't in Google, it effectively doesn't exist. I'm not sure the Microsoft monopoly has ever wielded that kind of power. And even if they did, it would by definition be limited to desktops. Google has shown few signs of abusing their position so far. But I'm not sure I'm comfortable with a single company having such near-absolute power over the sum of all information on the internet, either.

Posted by Jeff Atwood    View blog reactions

 

« The Power of Defaults The Problem With C++ »

 

Comments

One more data point on the dominance of Google. Jan Goyvaerts compared Microsoft AdCenter and Google AdWords, and found that Google delivered *TEN TIMES* more ad impressions.

http://www.shareware-beach.com/2006/11/microsoft-adcenter-first-impressions/

That's not a competition. It's a sixty-to-nothing blowout, a college football team attempting to play a pro football team.

Jeff Atwood on January 13, 2007 01:44 AM

So is this really an issue? Huge monolithic companies seem to be the way of the future, some low level competion, but it is like Apple and Microsoft. Altavista and Google. Dell and Gateway. One to rule them all, and hopefully not bind them all in darkness.

Craig on January 13, 2007 07:39 AM

Google's dominance of being the front door to the entire web, is the very reason that it is a useful tool. Two sides of the same coin.

Jeff on January 13, 2007 07:51 AM

No, it's not good, regardless of how much better Google is than anyone else. And it is equally not good that after all these years no one else has matched them.

It may just be me, but when I search for certain things I get a result set that is not as good as it was some years back -- the same question over and over in various forums, etc. Perhaps that is a function of a "larger" web.

I had made a resolution to use Ask.com more often, but it's hard to break old habits...

Steve on January 13, 2007 08:32 AM

"If It's Not in Google, Does Your Website Really Exist?"

It doesn't if your site is a technical site, like JavaLobby, as a lot of techno-nerds use Google fairly exclusively. But I've found that most computer novices use MSN, AOL, or Yahoo as their search engines because either those were the defaults or they became the default when they installed software (like AIM or Yahoo Messenger or MSN Messenger).

So if your site is about the latest celebrity gossip and Google nixed you from their index, I think it would have an impact, but not nearly as profound an impact as on JavaLobby.

Scott Mitchell on January 13, 2007 09:11 AM

Five short years ago, the monolithic search engine of the web went by a different name. But then someone built a better mousetrap.

I use Google as a default search engine because it produces good results, and is non-intrusive. If it falters in either of these critical customer-satisfaction areas, some other search engine will quickly rise in popularity.

If you see this issue as a power struggle of coporate giants, then why hasn't Microsoft made anywhere near the progress they'd hoped for?

If you see the issue as countless consumers holding all the power, and simply choosing the service that serves them best, then the rise and fall of Yahoo!, the rise of Google, and the inability of Microsoft to make headway all make more sense.

John Pirie on January 13, 2007 09:21 AM

Please, Google hardly carries the power to completely erase a website from existence just because it drops the site from the index. Yes, it is arguable that Google may be a big driving force of traffic, but ultimately it's your fault as a webmaster if that's the case.

If you have to get over 70% of your traffic from people searching for you, I would think that's not good enough publicity. It's weak strategy to rely on a search engine to drop people on your doorstep. It certainly helps, but it shouldn't be the source of 70% of your traffic.

Further, Google isn't the only search engine out there. They hardly have a monopoly and they hardly control whether or not your website is seen. Yahoo, MSN, Altavista... there are plenty of alternatives out there to get your website on a search engine.

I call it poor planning to rely so heavily on just one.

Raldo on January 13, 2007 10:55 AM

Google may not be the only search engine out there, and may not be the only way to drive traffic to your website. This does not alter the facts, however, that many many businesses happen to have extremely large volumes of their traffic driven to them through Google search results. It's just the nature of things these days. Businesses don't necessarily choose to have Google drive most of their business, and sure there are ways to infbutuence change should they choose to, but sometimes one is at the whim of the world...

frank johnson on January 13, 2007 11:21 AM

<i>It may just be me, but when I search for certain things I get a result set that is not as good as it was some years back -- the same question over and over in various forums, etc. Perhaps that is a function of a "larger" web.</i>

I agree. I think it's because Google is such a large slice of pie now that people spend a lot of time and effort getting to the top of that list. So instead of getting there on merit, sometimes the top results are there because of gaming the algorithm.

<i>If you have to get over 70% of your traffic from people searching for you, I would think that's not good enough publicity. It's weak strategy to rely on a search engine to drop people on your doorstep. It certainly helps, but it shouldn't be the source of 70% of your traffic.

Further, Google isn't the only search engine out there. They hardly have a monopoly and they hardly control whether or not your website is seen. Yahoo, MSN, Altavista... there are plenty of alternatives out there to get your website on a search engine.

I call it poor planning to rely so heavily on just one.</i>

Google giving you 70% of traffic isn't because of poor marketing, relying too much on one search engine, etc, etc. It's because THAT'S how much traffic Google drives.

Google accounts for 50% of searches
Yahoo accounts for 24% of searches
MSN accounts for 10% of searches
AOL accounts for 6% of searches
Ask accounts for 2% of searches

(source: http://searchenginewatch.com/showPage.html?page=2156451)

Google is that much more popular than the competition.

engtech on January 13, 2007 01:03 PM

dang, italics got stripped when I was quoting.

Anyhow, Jeff had a really good link in the article. Google gets credited with 40-50% of search market share but almost everyone who looks at their traffic reports would say Google is responsible for more like 70-90%.

http://www.skrenta.com/2006/12/googles_true_search_market_sha.html

engtech on January 13, 2007 01:05 PM

this is a typical american reaction to any one entity having this kind of control.

philihp on January 13, 2007 03:38 PM

To conclude from this post that JavaLobby is doing a poor job at marketing its site is erroneus. Many of the searches that lead to JavaLobby are the result of a search for a subject that happens to also exist in one of the articles posted on JavaLobby. Should JavaLobby pay adversiding dollars for each article/phrase or keyword on its site?

florin on January 13, 2007 04:28 PM

Guys, keep things in perspective. Google isn't housing information that no one else has. It's just a better "phone book". There are lots of different type of phone books out there, and they are all happy to put your number in for free. If one decided to do something screwy you would see everyone jump ship and go to the next one. Everyone loved the iPod, but now it is starting to get some competition for the zune and iRiver. It's the nature of the beast, everyone has their favorite and as long as that favorite doesn't screw you over, you will continue with it as your favorite until something better comes along.

Tim on January 13, 2007 08:24 PM

The reason we don't get results like we used to years ago is that the WWW is tremendously bigger and Google has indexed who knows how much more of it than say ask.com.

Ben Kruger on January 14, 2007 09:35 AM

Nothing's making me think JavaLobby deserves all this sympathy. They neglected their site for weeks after spammers attacked it, and now Google has "punished" them by protecting the rest of the net from their lax security. It even took them hours, apparently, to fix up the forums once they discovered the problem. Short explanation: they're just not much good at running forums.

It really is a non-story.

Eric TF Bat on January 14, 2007 02:56 PM

This isn't really a big deal. Yes, everyone gets to your website through Google. No, they don't have that much power because of this.

Everyone uses Google because they were first to market with a decent search engine, and nobody has any reason to switch. If Google abused this power (by delisting legitimate sites without cause), the press would pick this up, and soon people would switch to Microsoft, Yahoo, etc. Google may have market share, but in this case it just doesn't translate to market power as much as you might think.

Aaron on January 14, 2007 11:57 PM

Clearly in the case of JavaLobby, the majority of their traffic was spawned by Google searches for help and related articles. This should be a big indicator to them that while the content may be useful, the dynamic isn't great enough to keep their brand on a users mind and browsers pointed to their site.

oftencloudy on January 15, 2007 02:00 PM

"But I'm not sure I'm comfortable with a single company having such near-absolute power over the sum of all information on the internet, either."

Ok, but what is there to do about it? If MS search or Yahoo were much good, Google's dominance wouldn't be so complete. Unless you've got something to replace PageRank, you're probably just going to have to live with discomfort...

TW Andrews on January 15, 2007 08:10 PM

ask.com is powered by google :)

mookie on January 16, 2007 05:03 AM

I think the difference between (the good) Google and say, Microsoft (the "evil empire") is that Google didn't rise to power by creating a mediocre product and then forcing everyone to use it by abusing the power of their initial monopoly position. Indeed, as someone noted above, Google was *not* the first major search engine - that belongs to AltaVista and Yahoo. Google rose to power, in the face of competition from much larger companies (the aforementioned AltaVista and Yahoo), by offering a superior product. Really, this is the free market working the way it is supposed to work (as opposed to the Microsoft case, in which a monopoly used its power to break the free market system.)

Moreover, I think the nature of search engines is such that, if Google's product were no longer superior, it would lose its dominance in the market just as quickly as it gained dominance. It's *very* easy for a consumer to switch search engines, unlike switching operating systems.

Shadow Wolf on January 16, 2007 05:42 AM

Google aside, Jeff, I think this is really a larger question.

If I'm a really smart guy and I don't take advantage of anyone at all and I build a great machine, a really great machine... so great that everyone wants one, should I be penalized for it? If I do something so much better than everyone else that I effectively control any market, whether that market is a commodity or a luxury (video games, pet collars, access to the internet, lobster bibs) is it okay to come in and tell me that I'm too good for the collective good and I have to stop now?

I would argue that that kind of attitude stifles innovation. If I am going to go out and build my mousetrap, I intend to take it all the way... not just far enough that the regulators don't notice me.

I say huzzah for Google and huzzah for Starbucks and huzzah for Microsoft (when practicing ethical business). The USA attracts the best of the best because this is the country where one can rise to the top on merit.

I hope the worlds brightest people immigrate here and build the best and keep our star shining. If the rest of the world catches on, or forces here stifle our innovators, the USA will fall by the wayside... and deservedly so.

Garret on January 16, 2007 07:38 AM

I manage the website and traffic for voicebroadcasting.us and the same thing happened to them. It was catagorized nicely among the needed keywords, then the site got hacked, poof... site vanished. I found the site doesn't return back to rankings as quickly as the article discusses. Its been over 90 days and the web spider bot hasn't visited the site yet.

Bobak R on October 28, 2007 11:19 PM

Google sure is the front door to the internet. Also TinyURL-like services are notable, because they have loads of traffic going through them. People use TinyURL to shorten long URLs into tiny ones, so people visit TinyURL home page once in a while too.

There is a dictionary in Google, but I can't find a direct link to it in the Google home page google.com google.com/dirhp

Don on October 29, 2007 03:08 AM

The easy way out if Google decides to become Godgle is the semantic web; distributed crawlers on that as of yet fictitious technology would be the End of the Enterprise Search. Or we go back to messages in bottles :)

haskellnerd on December 20, 2007 03:01 AM

Just wanted to note the irony: I found this discussion via the Goog-Meister...

anon on December 28, 2007 11:11 AM

Please cheeck out this website for the best quality in Motorcycles!

Admin on April 13, 2008 11:21 AM







(hear it spoken)


(no HTML)




Content (c) 2008 Jeff Atwood. Logo image used with permission of the author. (c) 1993 Steven C. McConnell. All Rights Reserved.