Any one who is new in SEO field will find this post very useful because it will clear many of your doubts regarding many small but useful concepts let us have a look at robots.txt.These file is available in the / folder and any one can access it so please do not put any sensitive information in it.
  • What is a robot or bot?
They are such a programs that traverse the web automatically.Search engines like google use them to index the web contents.As each coin has two sides these bots are also used for bad purposes viz the spammers use them to scan email addresses and they may have many other purposes.
  • Robots are also known as crawlers/Web-wanderers/Spiders 
Website owners use the robots.txt file to give instructions about  their website to  the web robots.This is called as The Robots Exclusion Protocol.
  • Can we stop bad robots?
We can block bad robots by using firewalls that will block the IP address which makes more connection with the website.