[Home]Web crawler

HomePage | Recent Changes | Preferences

Difference (from prior major revision) (no other diffs)

Changed: 1c1
A web crawler (also known as web spider) is a program which browses the World Wide Web in a methodical, automated manner. Web crawlers typically keep a copy of all the visited pages for later processing - for example by a Search Engine.
A web crawler (also known as web spider) is a program which browses the World Wide Web in a methodical, automated manner. Web crawlers typically keep a copy of all the visited pages for later processing - for example by a Search Engine.

Changed: 3c3
In general, the web crawler starts with a list of URL to visit. As it visits these URLS, it identifies all the links in the page and adds them to the list of URLS to visit. The process is either ended manually, of after a certain number of links have been followed.
In general, the web crawler starts with a list of URL to visit. As it visits these URLs, it identifies all the links in the page and adds them to the list of URLs to visit. The process is either ended manually, of after a certain number of links have been followed.

A web crawler (also known as web spider) is a program which browses the World Wide Web in a methodical, automated manner. Web crawlers typically keep a copy of all the visited pages for later processing - for example by a Search Engine.

In general, the web crawler starts with a list of URL to visit. As it visits these URLs, it identifies all the links in the page and adds them to the list of URLs to visit. The process is either ended manually, of after a certain number of links have been followed.

Web crawlers typically take great care to spread their visits to a particular site over a period of time, because they access many more pages than the normal (human) user and therefore can make the site appear slow to the other users if they access the same site repeatedly.

For similar reasons, web crawlers must obey the [robots.txt protocol]?, with which web site owners can indicate which pages should not be spidered.


HomePage | Recent Changes | Preferences
This page is read-only | View other revisions
Last edited April 27, 2001 3:22 pm by WojPob (diff)
Search: