Thorp and Sailor's Grave Board

attn, people w/ websites

SHAKO KEN - 6-15-2006 at 08:06 AM

can't you disable google indexing? i only have file hosting but i could choose if i wanted to be indexed or not.

Everyday

ksonic26 - 6-15-2006 at 08:18 AM

Just create a robot.txt file to stop it.

BDx13 - 6-15-2006 at 12:00 PM

yeah, there's no guarantee it will work, but a robots.txt file is a great place to start.

technically speaking, bots, spiders and the like are supposed to read the robots.txt file before working their way thought the content of your site and indexing your everything. you can use this file to tell these programs where they can and can not go. bots from reputable companies will follow these restrictions, but bots from spammers definitely will not.

if your goal is to minimize traffic, this will probably work for ya, cause bots from google and the other search engines make up the majority of the bot traffic.

you can read more here:
http://www.robotstxt.org/wc/robots.html
http://www.searchtools.com/robots/robots-txt.html