http://www.websitehits.com/taxonomy/term/20/0 A program or automated script which browses the Web in a methodical, automated manner. en http://www.websitehits.com/web-crawler <p>A web <a class="glossary-term" href="/seo-terms-and-definitions/spider"><acronym title="spider: A program or automated script which browses the Web in a methodical, automated manner. ">crawler</acronym></a> (also known as a Web <a class="glossary-term" href="/seo-terms-and-definitions/spider"><acronym title="spider: A program or automated script which browses the Web in a methodical, automated manner. ">spider</acronym></a> or Web <a class="glossary-term" href="/seo-terms-and-definitions/spider"><acronym title="spider: A program or automated script which browses the Web in a methodical, automated manner. ">robot</acronym></a>) is a program or automated script which browses the World Wide Web in a methodical, automated manner.</p> <p>This process is called Web crawling or spidering. Many legitimate sites, in particular <a class="glossary-term" href="/seo-terms-and-definitions/search-engine"><acronym title="search engine: A system designed to help find information stored on the Web.">search engines</acronym></a>, use spidering as a means of providing up-to-date data. Web <a class="glossary-term" href="/seo-terms-and-definitions/spider"><acronym title="spider: A program or automated script which browses the Web in a methodical, automated manner. ">crawlers</acronym></a> are mainly used to create a copy of all the visited pages for later processing by a <a class="glossary-term" href="/seo-terms-and-definitions/search-engine"><acronym title="search engine: A system designed to help find information stored on the Web.">search engine</acronym></a>, that will index the downloaded pages to provide fast searches. Crawlers can also be used for automating maintenance tasks on a Web site, such as checking links or validating <a class="glossary-term" href="/seo-terms-and-definitions/html"><acronym title="html: The language used in creating web pages.">HTML</acronym></a> code. Also, crawlers can be used to gather specific types of information from Web pages, such as harvesting e-mail addresses (usually for spam).</p> <p><a href="http://www.websitehits.com/web-crawler">read more</a></p> spider Thu, 9 Nov 2006 15:30:33 -0500 joseph 20 at http://www.websitehits.com