WebCrawler News
The List
In This Issue
ODP History
ODP Submit
Guest Editor

WebCrawler Crawls Along

WebCrawler uses Excite's technology to index web pages. Although they mainly index index pages of a site, WebCrawler will also crawl deeper into some sites, particularly those with high "link popularity" which means other sites which have links to yours.

WebCrawler indexes each word on each page up to 1 megabyte of text.

The description on their results page will be the keywords under which your page is found.

WebCrawler suggests that you use a title which truly describes your page. Their relevance alogrithm gives more weight to titles than to body text. If you use "Welcome to My Site" or "So-and-So's Homepage" as a title, it will get buried with the other sites with "Welcome to" and "Homepage" in the title.

Describe as fully as possible what the site is about. Use as many of your keywords as possible as close to the top of the document as possible.

Embedded user IDs in the URL such as the ones generally used in many shopping cart sites are not indexed, nor does the spider recognize CGI directories, so pages generated on the fly are not indexed.

Your best bet for WebCrawler is to set up a static HTML page as the front door of your site and submit that one.


L10 Web Stats Reporter 3.15 L10 Hit Counter - Free Web Counters
LevelTen Professional Web Design Company - Flash, Web Designers