Robots.txt Is often a file that exists on the basis directory of each Web site and may be used to instruct search engines like google on which directories/data files of the web site they could crawl and involve inside their index. Search engines have just one aim only. They goal https://news.rafeeg.ae