DigitalOcean interview question

How does the robots.txt file do?