promotion Tip:
Where Did My Site Go?

by Larisa Thomason,
Senior Web Analyst,
NetMechanic, Inc.

It's every webmaster's nightmare. One day your site is steadily climbing in the search engine rankings and building solid link popularity. But the next week, you find that it has either vanished entirely from the search index or the rank has dropped so low that nobody will ever find it. Here are some hints to help you find, fix, and hopefully avoid this traumatic situation.

Check Your Server

First, try to access your site online and make sure the server is up. This may seem really obvious, but just ask anyone who works in a tech support center how many calls they field from confused customers whose computers, scanners, etc. don't work. Not because the products are defective, but because they need to be plugged in first!

Most Web hosting companies boast at least a reliable "99% uptime," but perhaps the search engine spider cruised by during that crucial 1% and found nothing. Quite a few human visitors may have had the same experience. Once a page is dropped, it stays gone until the spider revisits your site. Search Engine Starter lets you easily deep submit dropped pages, but remember that it may be months before the spider crawls by again!

Since you can't possibly sit and monitor your server 24 hours a day, NetMechanic's Server Check Pro tool will do it for you and alert you by pager or email if your server isn't responding. A server problem is always the first thing to check because correcting it is usually as easy as making a phone call to your hosting company. That's a lot easier than redesigning your site.

Review Your Robots.txt File

The robots.txt file tells spiders which pages they're allowed to visit and lists those that are off limits. Have you made any changes to the file recently? If so, you may have inadvertently closed the site to one or more search engine spiders. Or, you may have placed it in the wrong directory and the spider couldn't find it.

Review this excellent discussion of the robots.txt file to learn more about its important function.

Review Your Web Host's Terms Of Service

Perhaps your site has moved into a "bad neighborhood." That's search engine-speak for a server that hosts either adult sites or sites with a reputation for spamming search engines. Search companies don't bother to ask why your site is hanging out with these unsavory elements: they just ban the server's IP address entirely.

So if you're having a problem getting indexed, check your server logs to see if any search engine spiders have visited. The online Search Engine Dictionary maintains a list of spider bot names so you can check to see who has been by. If you've submitted and waited months without a single visit, then perhaps your site is hosted in the wrong place.

Check your Web hosting company's terms of service (TOS) agreement to see if they allow adult or gambling sites. Both have a reputation for being inveterate search engine spammers (even though not all are). It could be that the company began allowing those sites after you signed up and the search engines have caught on and banned the IP address.

In this case, your only real recourse is to move your site to a new hosting company. Refer to our Seven Questions To Ask Your Web Host article for tips on evaluating hosting companies. It is possible to appeal your ban with a particular engine, but that process could take months and cost you a lot of visitors.

Have You Paid Your Bills?

If you're using a pay for placement (like Overture) or pay for indexing (like Inktomi), or pay for inclusion (like Yahoo) service, make sure your payments are up to date. Check your email for notifications from the company.

Many webmasters maintain a separate email account for domain renewal, search engine and directory submission, and other related site issues. It's easy - but dangerous - to forget to check that address regularly. Most free email providers either close or delete the mail from inactive accounts and accounts may be termed "inactive" if you fail to check them at least once every 30 days.

Fortunately, this is the easiest problem to fix. Login to your account, make your payment, and your site is back in the index!

Have All The Pages Disappeared?

Well, now we're getting to the hard stuff. The first four suggestions pointed to outside sources that could cause problems. But it's just as likely that the problem is with your Web site content itself.

Before you change a thing though, check to see if every single page of your site has been deleted from the index. Most likely there's a problem with just a few pages. That points to an algorithm update at the search engine.

The "algorithm" is the super-secret formula each search engine uses to rank and/or penalize Web sites. Search engines change them constantly to thwart spammers and to provide more relevant results to searchers. Each time the algorithm changes, your page rank may change too.

You'll have to look at your page content and HTML code to determine which design or coding technique is causing the problem. Compare the dropped pages (or those whose rank dropped precipitously) with your high-ranking pages. Maybe you'll see that the dropped pages lack header tags or ALT text on images. Perhaps the pages have very little text content. Look the dropped pages over carefully and try to find a common element.

Evaluate Your Site Content

Let NetMechanic's HTML Toolbox tool scan the problem pages before you spend hours combing through your code. It will identify simple coding errors that may not affect page display in a browser, but may keep the search engine spider from indexing all of the page's content.

Once you have an error-free page, check for other issues that can also affect search engine rank.

  1. Before and After: If you kept a backup copy of the site, compare the pages before the change and after. That terrific new design you implemented may be shutting visitors out instead of drawing them in.

  2. Server Redirects: Try to avoid META refresh tags. Some spam sites lure traffic by purchasing expired domain names that have a decent search engine rank and backlinks. Then, they use a META refresh to direct the unsuspecting visitor to their spam site. Search engines are wise to this tactic and may penalize sites that use META refresh tags.

    It's much safer to use an HTTP header 301 (permanent redirect) when you need to change a page location. Check with your Web host to see if you have access to HTTP header information.

  3. Keyword Stuffing: Yes, it's important to select the best keywords and keyword phrases for your site and use them inside header text, ALT tags, and page content. But it's easy to get too much of a good thing. Search engine algorithms penalize sites that over-use keyword phrases.

    Unfortunately, it's hard to know if you've used a phrase too often or not often enough. NetMechanic's Page Primer tool will calculate your page's keyword density score and alert you if there's a problem.

  4. Link Farming? Remember the importance of keeping your site out of a "bad neighborhood?" Well, it's just as critical to avoid linking to sites in bad neighborhoods or to sites that are totally unrelated to your site's topic.

Read more about the dangers of link farming in our April, 2002 Webmaster Tip.

A great way to avoid these types of problems is to use NetMechanic's handy, affordable coding and search engine tools. Even if you've made a simple change, always use HTML Toolbox to check the changed page for errors.

Then use NetMechanic's Page Primer tool. It will scan your pages for problems and provide engine-specific tips to improve page rank. You'll know if you've used your keywords too much (or too little), if there's enough content, or if you've inadvertently used some design technique (like hidden text) that got you banned.

 

Webmaster Tutorials

 Webmaster
 Resource Center

Browse our Webmaster Tips library organized by topic. Learn about search engines, JavaScript, HTML, &  more!


Quick Search

Search our newsletter archives by keyword phrase.

 
                   

Volume 7 (2004)

 November
 September
 July
 June
 May
 April
 March
 February (Part 2)
 February
 January (Part 2)
 January

Volume 6 (2003)

 December
 November (Part 2)
 November
 September
 August (Part 2)
 August
 July (Part 2)
 July
 June (Part 2)
 June
 May (Part 2)
 May
 April (Part 2)
 April
 March (Part 2)
 March
 February (Part 2)
 February
 January (Part 2)
 January

Volume 5 (2002)

 December (Part 2)
 December
 November (Part 2)
 November
 October (Part 2)
 October
 September (Part 2
 September
 August (Part 2)
 August
 July (Part 2)
 July
 June (Part2)
 June
 May (Part 2)
 May
 April (Part 2)
 April
 March (Part 2)
 March
 February (Part 2)
 February
 January (Part 2)
 January