Robots txt Generator

Search Engine Optimization

Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

Online Robots Text Creator

Robots Text File (robots.txt), is a standard text file used by websites to communicate with search engines and search engine bots such as (Googlebot & Bingbot). This simple text file instructs the search engines which areas of the website may or may not be scanned or analyzed. Some of the major search engines follow this standard:

  • Ask 
  • AOL 
  • Baidu 
  • Bing
  • DuckDuckGo 
  • Google 
  • Yahoo!
  • Yandex

Having a robots.txt file on your website will function as a navigation to specified search engine robots to ignore or follow specified files & directories when crawling your site. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operates on certain data. Links to pages listed in robots.txt can still appear in search results if they are linked to from a page that is crawled.