Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.
txt Generator. Search Engines are exploitation robots (or therefore known as User-Agents) to crawl your pages. The robots. txt file could be a document that defines that elements of a website will be crawled by a robot.