Rapid7 Vulnerability & Exploit Database

HTTP Robots.txt Content Scanner

Back to Search

HTTP Robots.txt Content Scanner



Detect robots.txt files and analyze its content


  • et <et@metasploit.com>


Module Options

To display the available options, load the module within the Metasploit console and run the commands 'show options' or 'show advanced':

msf > use auxiliary/scanner/http/robots_txt
msf auxiliary(robots_txt) > show actions
msf auxiliary(robots_txt) > set ACTION < action-name >
msf auxiliary(robots_txt) > show options
    ...show and set options...
msf auxiliary(robots_txt) > run 

Time is precious, so I don’t want to do something manually that I can automate. Leveraging the Metasploit Framework when automating any task keeps us from having to re-create the wheel as we can use the existing libraries and focus our efforts where it matters.

– Jim O’Gorman | President, Offensive Security