|
Search Engine Tip:
Beware of Query Strings
by Tom Dahm,
Chief Operations Officer,
NetMechanic, Inc.
Do you use Active Server Pages or Cold Fusion and have trouble getting your site listed by search engines? Here's a tip: avoid using CGI query strings to link your pages. Most search engine spiders refuse to follow these links.
CGI query strings are a method used by your browser to pass extra information to a Web server. You're most likely to see query strings when using an HTML form. When you submit the form, the query string is embedded in the URL shown on your browser's address bar.
To see an example, visit Yahoo!, enter a search, and then look at the URL in your browser's address field. You should see something like this:
http://search.yahoo.com/bin/search?p=apples
|
The query string is the part of the URL following the question mark. In this example the query string passes a variable called "p" to the Web server and loads it with the value "apples." Yahoo's server then uses this information to search for pages about apples.
Query strings aren't just used by HTML forms; they can also be embedded directly into links inside a page. Many "active page" Web sites use query strings in just this way.
With Web sites becoming increasingly sophisticated, more Webmasters are using active page technologies like Cold Fusion, Active Server Pages, or PHP3 to build their sites. These technologies allow you to build your Web pages "on the fly."
Active page sites often link their pages together by embedding a CGI query string in each link. This allows them to do a number of things, such as track visitors from page to page.
For example, Barnes & Noble uses Active Server Pages and query strings to track the paths visitors take through their site. When you visit the Barnes & Noble site, the Web server assigns you a unique visit ID and then embeds that ID in every link on the home page. As you move from page to page, each page's links are rewritten to include your visit ID.
This is a powerful technique, but it has a significant downside: it can keep your site from getting listed by search engines. Most search engine spiders refuse to follow links containing CGI query strings. Why? Because in the past query strings have caused problems for the search engines' spiders.
Search engines use spiders to index your Web site. When you submit your site, the search engine adds your home page to a list of URLs awaiting indexing. The search engine periodically sends out a series of spiders that work their way down this URL list, fetching each home page in turn. When your home page is fetched, the spider follows its links to crawl through the rest of your site.
To conserve computer resources, the search engine sends out a limited number of spiders. If something happens to tie one of those spiders up, the engine's ability to index the Web suffers.
That's why most search engine spiders avoid links with query strings. Query strings sometimes act as "spider traps." Badly written CGI programs or programs that expect the spider to pass additional information, such as a cookie, often trap the spider in an infinite loop where it requests the same page again and again.
To avoid any chance of this happening, most search engine spiders simply avoid any links containing a query string. If your site embeds query strings in links between pages, then there's a good chance the spider will index your home page, but skip all other pages in the site. That can dramatically reduce your visibility in the search engine.
The ideal solution to this would be for the search engines to improve their spiders. Spiders can be written to be smart enough to detect spider traps. Our own NetMechanic spiders will follow CGI query strings without problems.
However, the search engines have shown little inclination to keep their spiders in synch with changing Web technologies. Many search engine spiders don't even understand the FRAME tag.
So if your Web site makes heavy use of query strings, you have two solutions. Either change your site, or submit important pages individually without their associated query strings.
|
