robots.txt is NOT a security control
...txt file was a good security control. Not only is it not a security control, it also doesnt add much value beyond helping control which search engines get to index your Web sites and what they get to index. In other words, they have no value when attempting to stop an attacker from retrieving information during target reconnaissance activities
