Despite the fact that the robots.txt file was invented to inform search engines like google what internet pages not to crawl, the robots.txt file may also be utilized to stage search engines like google and yahoo for the XML sitemap. This is often supported by Google, Bing, Yahoo and Inquire. https://rankingseotoolz.com/robots-txt-generator