|
JavaScript Tip:
Spiders May Choke On Your Cookies
by Larisa Thomason,
Senior Web Analyst,
NetMechanic, Inc.
Even though many people can't resist the offer of a fresh baked cookie, search engine spiders have more willpower. They get offers of JavaScript cookies all the time, but always turn them down. If visitors have to accept cookies before they can view your site's content, you may be hiding your best content from search engines!
Spiders Don't Eat Cookies
Have you ever had cookies turned off in your browser and visited a site that required them? You get a warning message that says something like "your browser must accept cookies for this site to function properly." Other times, the message has a more unfriendly tone: "this site requires cookies and your browser doesn't support them."
A human visitor has three options:
- Enable cookies in the browser.
- Change browsers.
- Visit another Web site.
|
A search engine spider just leaves. None are configured to accept cookies because they are just basic text browsers. All a spider is interested in is content and if that content is hidden behind a cookie, that's just too bad for the webmaster.
Why Set Cookies At All?
A JavaScript cookie can be extremely useful both for you and your site visitors. A cookie is a small text file saved on a visitor's computer. It identifies a particular machine and browser. Our May, 2000 Beginner Tip "What Exactly Are Cookies?" explains cookies more fully and also describes how to create simple cookies for your own site.
Cookies help you customize your visitors' browsing experience. Without them, your shopping cart system won't work; you'll have trouble tracking repeat visitors, and will lose a lot of valuable information. Just make sure that you're setting cookies because you really need them - and that you're setting them at the correct point in the browsing process.
You'll only have a problem if you offer a cookie too early in the browsing experience - and make it mandatory!
Using Cookies Effectively
Some sites protect their content by forcing visitors to sign in and accept a cookie before going farther into the site. This strategy protects your paid content, but shuts spiders out of all your content. How will prospective subscribers or customers ever find you if the site can't be indexed?
Carefully consider just when it's appropriate to force a cookie on your visitors. Here are strategies some site use:
- Top Levels Open: Leave your home page and several important pages open and set the cookie a few layers down. This allows you to have several entry pages for search engines. Then, use a mandatory cookie to protect content that's deeper inside the site.
- Use Teaser Content: The online news and opinion site Salon.com uses this technique. Most of their content is available only to subscribers, but they allow non-subscribers access to the first few paragraphs of each article - the teaser. After reading those paragraphs, visitors must either login or become a subscriber.
When visitors login, a cookie that gives full site access is set on that particular machine and browser. On their next visit, subscribers automatically have access to the full content.
This technique opens up every page in the site to at least partial indexing.
- Spider Detection and Redirection: Create a JavaScript function that checks to see if the browser is really a search engine spider. If so, it delivers content without requiring a cookie. The spider indexes it and goes away happy.
|
Don't confuse this with a shady technique called cloaking! With cloaking, you're creating pages that are designed to fool search engine spiders about your site's real content. But in this case, you're trying to make sure that spiders actually read your content!
In most cases, browser detection isn't tricky to code. Our Browser Detection And Redirection article covers it in detail.
However, detecting for search engine spiders can get quite complex because there are so many of them. The RobotsTxt.org maintains a list of spider names and other information about Web robots.
The list is pretty long. Imagine trying to write a function that checked for every single one and keep it updated!
For most sites, the simplest solution is usually the best one:
- Create content-rich pages optimized for search engine spiders and human visitors.
- Make some of that content available to all users.
- Use that content to draw a human visitor deeper into the site and encourage her to become a subscriber or fill her online shopping cart to the brim!
|
But remember that cookies aren't the only problem affecting site promotion. Many sites have problems attracting and keeping visitors because they've hidden their best content inside Flash files, images, or used other design techniques that make content inaccessible to search engines.
Optimize your page with Page Primer, part of Search Engine Power Pack's suite of search engine submission and optimization tools. Page Primer scans your page and suggests techniques that make your page more attractive to particular search engine spiders.
|
