Australia Travel Industry News
The index of a crawler-based search engine is built through the
use of robots (spiders, web crawlers) which operate on a fixed set
of instructions. The robot selects a page to visit from a list of
links (a "queue") gathered from web pages that were previously
searched. It fetches the web page, collects certain information
(such as visible text, meta tags, links, etc.) and sends it to an
indexing program. The information is then entered into a database,
ready for searching inquiries, then the newly gathered links are
entered into the queue for a future visit and the process begins
again.
|
Click here for information on our
services
Click here
for an obligation free Search Ready Report on your site |