digg

Join DiggAbout

Discover the best of the web!
Learn more about Digg by taking the tour.

How to write robot.txt to control search engine spider

techiecorner.com — What is the use of robot.txt or robots.txt? robot.txt or robots.txt (plural) is just a simple text file tat use to control how search engine spider or crawler should go thru your site and which spider is not allow to visit your site. Example of a robot.txt ...

Bury
show profanity settings
expand all
  •  

    Login or register to add a comment

    Create a new account or login to join in the conversation on Digg. You'll also be able to Digg stories to help promote things you like.


© Digg Inc. 2008 — Content posted by Digg users is dedicated to the public domain.
DIGG, DIGG IT, DUGG, DIGG THIS, Digg graphics, logos, designs, page headers, button icons, scripts, and other service names are the trademarks of Digg Inc.