Promotion Tip:
Deep Submit Your Dynamic Pages

by Larisa Thomason,
Senior Web Analyst,
NetMechanic, Inc.

Just think how easily you could promote your dynamic Web site if those pesky search engine spiders could just index the content of your dynamic pages. Well, the exciting news is that some spiders can do just that! Other search engines have taken a smaller step. They allow webmasters to submit important dynamic pages directly - one page at a time.

Problems With Dynamic Sites

A dynamic site has content delivered from a database in response to user queries. It's usually easy to spot a dynamically generated page: just look for a file extension like .asp, .php, or .cfm.

The problem isn't with the file extension, but with the special characters that indicate a query string - a question mark, equal sign, or ampersand (?, = , &). The presence of the special characters indicates a non-static HTML page, or one that only exists in response to a particular query from a visitor.

In the past, search engine spiders avoided those pages entirely because they often became "spider traps." Spiders would get caught on the page, unable to continue through the site or backtrack out of it. Sometimes, the problem would be so severe that the Web site's server would crash - locking out all visitors!

Google's Webmaster Guidelines page notes that:

"If you decide to use dynamic pages (i.e., the URL contains a '?' character), be aware that not every search engine spider crawls dynamic pages as well as static pages. It helps to keep the parameters short and the number of them small."

Our February 2001 Webmaster Tip discusses several ways to rewrite the URLs of dynamic pages to remove the problem characters. However, that doesn't always help search engines crawl through your dynamic pages.

Opening Doors To Dynamic Sites

Google has been quietly allowing its spider (called Googlebot) to crawl dynamically generated pages and index the content. This experiment has actually been going on for over a year and Google admits to approaching the project "slowly and cautiously."

Google's Webmaster FAQs page warns that:

"We are able to index dynamically generated pages. However, because our web crawler can easily overwhelm and crash sites serving dynamic content, we limit the amount of dynamic pages we index."

Inktomi is also slowly adding dynamic capability to its spider too (but remember that Inktomi doesn't allow any free site submissions anymore). Inktomi's Web site notes that its spider (called Slurp) can index dynamic pages, but will not "crawl them by default."

"If your Web site is based on dynamic links and you want your site to appear in our search engine, one approach is to have some static pages which have links to your dynamic pages."

Indeed, that's easiest way to get both Googlebot and Slurp to index your dynamic pages. Because the value of the initial query is included in the link text, the spiders aren't expected to create the query, only index the results.

For instance, if your ecommerce site has all its product information inside a database, it's a good idea to create a products page that contains the links that retrieve important information about your major product categories. Googlebot and Slurp reliably follow these dynamic links.

So the dynamic links you want these spiders to index might look like this:

<a href="products.cfm?type=InkJetPrinters" 
  title="See a list of ink jet printers in 
    stock!">Ink Jet Printers</a>



<a href="products.cfm?type=LaserPrinters" 
  title="See a list of laser printers in 
    stock!">Laser Printers</a>

Both Googlebot and Slurp follow the links on the static HTML page and index the content of the dynamically generated pages. But they stop there. Neither spider will reliably go deeper into your dynamic pages unless you create more static pages that link to that additional information.

AltaVista indexes dynamic pages submitted using its Trusted Feed premium inclusion program. This service gives Alta Vista partners more control over how their pages are indexed and ranked. The service description specifically notes one of the benefits: "accepts pages that are traditionally difficult for crawlers to index, such as framed pages or pages with dynamic content."

The paid inclusion is a reliable way for large Web sites to get their entire database content spidered, but the cost is often way too much for smaller sites to even consider.

Deep Submit Helps With Some Engines

If you can't justify the expense of a premium inclusion program and you don't want the bother of creating static HTML pages with dynamic links, you do have another option: deep submit your dynamic pages directly to search engines.

When you deep submit pages, you're ensuring that the search engine indexes all the important pages in your site by submitting individual pages directly to the search engine.

This way, you can tell the search engines exactly what query results you want indexed. Unfortunately, even this can cost you money if individual engines require you to pay to submit. For instance:

AltaVista still offers a free submit option, but you'll do a lot more typing using the free submit than with the Trusted Feed program. That's because the Basic Submit limits submissions to one URL at a time. You end up with a lot of repetitive typing and the chances of making a mistake increase.

This is actually a big problem with any type of deep submit strategy because every search engine wants the URL, your email address, and more information about your site. You could spend hours - or days - submitting the same data over and over to various search engines.

Or you could use NetMechanic's easy deep submission tool Search Engine Starter to submit multiple pages of your Web site to top search engines like Google, AllTheWeb, and HotBot. Search Engine Starter will help you get all your important pages indexed - dynamic and static alike.