What is a Robots.txt File?

The robots.txt file is an HTML document that contains instructions for search engine bots and crawlers. It’s uploaded to your server so web crawlers and other web “robots” know how to interact with your site. Using this file, you can block certain pages from getting indexed and you can also block certain crawlers or search engines from accessing your content. It does make sense to block some pages as you don’t want private pages, secure pages, or other pages not meant for public access to take up the crawl budget for your website or to get indexed in search results.