Online Robots Text Creator
Robots Text File (robots.txt), is a standard text file used by websites to communicate with search engines and search engine bots such as (Googlebot & Bingbot). This simple text file instructs the search engines which areas of the website may or may not be scanned or analyzed. Some of the major search engines follow this standard:
Having a robots.txt file on your website will function as a navigation to specified search engine robots to ignore or follow specified files & directories when crawling your site. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operates on certain data. Links to pages listed in robots.txt can still appear in search results if they are linked to from a page that is crawled.