Promotion Tip:
Is Your Web Host Spider Friendly?

by Larisa Thomason,
Senior Web Analyst,
NetMechanic, Inc.

	March 2002 (Part 2) Vol. 5, No. 6  • Promotion Tip  • Accessibility Tip  • HTML Tip

You may spend every waking hour optimizing your Web site and writing dazzling content to entice visitors, but have you given much thought to the role your Web host plays in your promotion efforts? Some Web hosting companies can sabotage your hard work quicker than you can say "banned from Google."

The High Cost Of Free Hosting

You can pay a whole lot for Web hosting or you can pay next to nothing - sometimes even nothing at all. Some online services offer free Web site hosting and Web hosts often provide their ISP customers with a free personal Web site.

That sounds like a great way to save money, but it can reduce your credibility with visitors and visibility to search engines:

Virtual Hosting Basics

Unless you're shelling out big bucks for a dedicated server, you're paying for virtual hosting. That means you're sharing server space with hundreds - maybe even thousands - of other sites.

Let's take a minute to understand how virtual hosting works. It's easy to get confused when somebody's throwing around terms like IP addresses, routers, and DNS servers, so let's use an example familiar to many people.

A shared server is like a college dormitory where a bunch of students live in a single building.

When the pizza delivery guy shows up with a delivery for Bob, he has to know both the street address and Bob's room number. Although the students all live at the same street address, visitors still need the individual room numbers to find the correct student's room.

A server's IP address is like the dorm's street address. All the sites on the server share the same IP address, so spiders have to ask for the "room number" when they come crawling by looking for a particular site.

Beware Of Your Neighbors

Virtual hosting is usually a good, cost-effective hosting solution for medium and small sites - unless you move into a bad neighborhood. Remember that you're sharing the same IP address with many other sites. If your neighboring sites have a history of causing trouble for search engine spiders, then the search engines may prevent their spiders from visiting any site with the same IP address.

Think about our college dorm: if the students dropped water balloons on delivery drivers or called in bogus orders, then the pizza company might just refuse to deliver to the dorm at all. For the same reason, search engines may ban an IP address if that address has a history of spamming search engines. You could be penalized even though you're innocent of such nefarious schemes.

Or what if you're sharing space with a site that's getting so many hits each day that the server can't respond to other requests? When the The Statue of Liberty-Ellis Island Foundation's Ellis Island records site debuted, the resulting stampede of traffic in the first few days locked out many eager visitors. While that site runs on several dedicated servers, consider the effects if it shared space: every site residing on that server would have been down.

It pays to know who your neighbors are and what they're up to!

Site Outages Are Expensive

It also pays to know what kind of provider you're relying on. When you're considering a Web host, don't make a snap decision or automatically select the cheapest. Put together a checklist of requirements and select your host carefully.

Pay close attention to the download limits imposed by the host. Otherwise you could be the victim of your own success. Some hosts impose monetary penalties on sites that exceed download limits. If you don't promptly pay the extra cost, some hosts will take your site down and hold it hostage until you cough up the ransom money.

Now that many search engines have added paid submission programs, a site outage can cost you in ways other than lost visitor traffic. As we calculated in our Paid Inclusion article, you could spend over $1800 to get 25 pages of your site spidered by Inktomi and AltaVista! With that much on the line, you better be sure that the site is up and running when spiders and visitors come by.

About the worst thing that can happen is for a spider to come crawling by and find your Web site down.

Don't Rely On Promises

Find out what kind of uptime guarantee your host has. Many claim to offer 99% uptime. That sounds comforting, but it's always best to verify those claims for yourself.

Or what if your site is up, but responds so slowly that visitors give up and leave before the page loads? If the server request times out, the spider has no content to index.

Note that this problem has nothing to do with the page load time measured by HTML Toolbox. Even though you've optimized your images with GIFBot, stripped your code down to the bare essentials, and created a page that loads in 5 seconds, a poor-performing server can still sabotage your site.

NetMechanic's Server Check tool will monitor server performance and alert you by email, phone, or pager when your site goes down. You can alert the host and get action in a few minutes instead of hours - or even days.

Web hosts make a lot of great claims about their speed and service. Just make sure they're following through on them. If not, it's no big deal to change Web hosts.

Your Web host is critical to your site's success. Make sure it's an ally and not a saboteur.