Detect robots.txt files and analize its content
To display the available options, load the module within the Metasploit console and run the commands 'show options' or 'show advanced':
Time is precious, so I don’t want to do something manually that I can automate. Leveraging the Metasploit Framework when automating any task keeps us from having to re-create the wheel as we can use the existing libraries and focus our efforts where it matters.
– Jim O’Gorman | President, Offensive Security