Tuesday, March 13, 2007

How Search-Engine Rules Cause Sites to Go Missing

Some entrepreneurs have built thriving businesses largely by getting search engines such as Google Inc.'s to direct customers to their Web sites. But what happens when the search engines suddenly start pointing consumers somewhere else?
That is a possibility with which Topix.net Chief Executive Rich Skrenta is struggling this month. The news site, which is majority owned by media giants Gannett Co., McClatchy Co. and Tribune Co., paid a Canadian company $1 million for the Web address Topix.com in January. Mr. Skrenta intends to switch his site over to the more popular .com Web address from .net soon to help eliminate confusion and increase credibility with consumers.
Such a simple change, Mr. Skrenta has discovered, could have disastrous short-term results. About 50% of visits to his news site come through a search engine -- and about 90% of the time, that is Google. Some companies say their sites have disappeared from top search results for weeks or months after making address switches, due to quirky rules Google and other search engines have adopted. So the same user who typed "Anna Nicole Smith news" into Google last week and saw Topix.net as a top result might not see it at all after the change to Topix.com.
Even if traffic to Topix, which gets about 10 million visitors a month, dropped just 10%, that would essentially be a 10% loss in ad revenue, Mr. Skrenta says. "Because of this little mechanical issue, it could be a catastrophe for us," he says.
Further frustrating him is that Google's response to Topix's plea for help was an email recommending that, if the switchover were to go badly, the company should post a message on an online user-support forum; a Google engineer might come along to help out. "This can't be the process," Mr. Skrenta says. "You're cast into this amusing, Kafkaesque world to run your business."
He's among a growing group of businesspeople whose fortunes are buffeted or burnished by the invisible, constantly evolving mathematical formulas at the heart of Web search engines. The influence of search engines has only grown in recent years, as they have become the de facto gateways for many of the more than 180 million American Internet users to anything they might do online.They also have become a crucial tool for businesses that depend on those users finding them.
But as a way to lure customers to a site, search rankings often aren't dependable. Overnight, sites can disappear from top results for any given search term -- say, "Miami hotels" -- and cause the sites' revenues plummet as potential customers go elsewhere.
Topix.net is moving to Topix.com, which could drop it in Google's results.
Among the most common reasons for unpredictable changes in rankings are frequent updates to search engines' algorithms. These mathematical formulas analyze billions of Web pages for dozens of factors, such as the most prominent words on the pages and what other sites link to the pages, in order to determine how to rank them for relevance to a query. Search companies change algorithms partly to frustrate people who try to inappropriately boost their sites in the results, but legitimate businesses sometimes feel they're caught in the crossfire.
Google, of Mountain View, Calif., says it offers online tools for companies to get the best, most consistent, treatment from its search engine. It also counsels that sites shouldn't become overly reliant on traffic from searches and should find other ways to get visitors, such as by setting up user forums. "We have to keep improving our algorithms and giving the best search results," says Google software engineer Matt Cutts. "We can't promise that if you're No. 1 today, you'll be No. 1 tomorrow."
The importance of appearing at the top of the results is undisputed. A JupiterResearch study sponsored by search marketing firm iProspect concluded last year that 62% of search-engine users generally clicked on a link to a site on the first page of results. That has fueled the emergence of an industry of search-engine "optimization" specialists who help businesses try to find ways for their sites to rise in the rankings, such as using more-explanitory page titles.
Companies always have the option of paying for the search advertisements that usually appear above and alongside the search-engine results, but non-advertising results can be more significant. JupiterResearch estimates that when consumers are looking for products and services, they click on non-advertising results almost 80% of the time.
While a business changing its Web address could predictably have search repercussions, unannounced changes in a search engine's algorithm can have outsized impact on a business. Marchex Inc., a Seattle company that operates more than 200,000 Web sites, says it sometimes gets snared by seemingly arbitrary shifts.
One Marchex site, bayareahotels.com, now generally ranks among the top-10 Google results for the search "Bay Area hotels." Marchex executives say the site recently disappeared without warning from the first page of Google's results, then reappeared a few weeks later. Marchex says traffic and revenue from the site fell when it dropped out of the top results (it didn't specify how much).
Peter Christothoulou, Marchex's chief strategy officer, says although changes in rankings are unpredictable, "they do a pretty fair job." He says the best way for a business to weather shifts is to have a Web site with strong content. "If you have a good site, you end up where you should be even if you [occasionally] fall out of listings," he says.
VLSI Research Inc. ran into a different problem. The Santa Clara, Calif., research company recently started a social-networking site for technical workers in semiconductor manufacturing and related industries. Chief Executive Dan Hutcheson was dismayed to find Google wasn't including the site in its search results two weeks after it launched. He says VLSI called Google to ask that it include the new site, and it bought Google search ads in the hope it would help. After VLSI contacted Google repeatedly about the issue, a Google employee threatened to blacklist VLSI sites from its results, Mr. Hutcheson says. The blacklisting never happened, and the site began showing up in Google results two weeks after it started.
"Our intent is to represent the content of the Internet fairly and accurately," Google said in a statement responding to VLSI's allegation. Google says it provides guidelines and online tools to help sites be found quickly. For example, it recommends that companies use its Webmaster Tools site to tell it which pages are most important and how often they change, so Google can more effectively find it.
Then there's the issue Topix and others have wrestled with. After closely held HomeStars.ca changed its name from HomeDirection.ca about 18 months ago to better connect with users, search-engine ranking for the site for consumer reviews of home renovators and suppliers plummeted, says Andrew Goodman, chief content producer at the Toronto-based company. For about six months, HomeStars lost roughly 60% of the visits it through search engines, about 80% of which came through Google. "I don't think anyone has ever had a changeover where they don't lose traffic for a little while," Mr. Goodman said.
When HomeStars moved its site, it placed a computer code on its old HomeDirection site to tell search engines and Web browser software to skip to its new HomeStars address. That's the equivalent of a business's forwarding its mail and putting up a sign with its new address when it moves its store.
But search engines can be skeptical of such notices, because they're wary of people buying Web addresses and forwarding the traffic to other sites that aren't as relevant to the search results.
Concerned about that strategy, Topix has run its site at both Topix.net and Topix.com for awhile. One danger with that approach is that it is unpredictable; Google will see two versions of the same page and could choose to show the Topix.net page most prominently.
Google's Mr. Cutts says the search engine should ultimately understand what is going on when a site changes its Web address. He says the best strategy is to move one section of the site to the new address and see what happens before switching the whole thing.
The Internet company is open to providing businesses with online tools to explicitly signal such a change, but in the meantime, Mr. Cutts says, posting in a Google support forum and hoping for a Google engineer to take sympathy, as Topix was counseled, is more reliable than it sounds.
Mr. Skrenta is crossing his fingers as the changeover approaches, and he says he has no animosity toward Google. "It's not that they're bad guys," he says. "It's just that they didn't set out to wield this level of influence over the Net."

No comments: