Robots.txt Generator

Generate crawler rules and download robots.txt.