How to keep robots out of your web site

THE ROBOTS. TXT FILE You know fact that look about engines have been created to help people find information quickly on the Internet, and the look about engines acquire by far of their information through robots (as well of note as with spiders or crawlers), fact that look out in behalf of web pages in behalf of them. The spiders or crawlers robots _explore the web looking in behalf of and recording each and all kinds of information.

They usually enter upon with URL submitted on the part of uusers, or from links they find on the web sites, the sitemap files or the high level of a site. But the (read out as well Search Engine Optimization Companz) can as well out each and all the pasges fact that can find on a particular server. Once the robot accesses the home page then and there recursively accesses each and all pages linked from fact that page. After the robot finds a web page it works indexing the title, the keywords, the _text, etc. But every such that often you might want to prevent look about engines from indexing some of your web pages like news postings, and specially marked web pages (in example.

Affiliate´s pages), but then whether single robots comply to these conventions is spotless autocratic. ROBOTS EXCLUSION PROTOCOL So if you want robots to keep check out from some of your web pages, you can ask robots to ignore the web pages fact that you don´t want indexed, and be in place fact that you can place a robots. Txt file on the local root server of your web site.

In example if you have a directory called e-books and you want to ask robots to keep check out of it, your robots. User-agent. Txt file should read out. * Disallow. E-books/

Txt file, you can be at pains adding a META tag to the ^head section of any one HTML document. (read out as well Search Engine Optimization Companz ) you don´t have enough control over your server to set a robots. In example, a tag like the coming tells robots not to index and not keep track links on a particular page. Meta name="ROBOTS" content="NOINDEX, NOFOLLOW"

NEWS POSTINGS Support in behalf of the META tag among robots is not such that frequent as with the Robots Exclusion Protocol, but then much of major web indexes currently support it. If you want to keep the look about engines check out of your news postings, you can make up an an "X-no-archive" line in of your postings' headers.

X-no-archive. Yes But although common news clients, allow you to add on an X-no-archive line to the headers of your news postings, some of them don´t permit you be in place such that. The problem is fact that most look about engines assume fact that each and all information they find is public unless marked otherwise.

So be serviceable in so far as though the robot and archive exclusion standards may help keep your material check out of major look about engines there are some others fact that respect no such rules. You can read out at a guess it from here. If you're highly concerned at a guess the privacy of your e-mail and Usenet postings, you must use some anonymous remailers and PGP. Http. //www.

Well. Html http. Com/user/abacard/remail. //www. Io.

Com/~combs/htmls/crypto. http. Html //world. Std.

Com/~franl/pgp/ Even if you are not particularly concerned at a guess privacy, remember fact that anything you write out will be indexed and archived somewhere in behalf of eternity, such that use the robots. Txt file how you need it. Written on the part of Dr.

Roberto A.