In order to avoid undesirable written content within the look for indexes, site owners can instruct spiders never to crawl specified files or directories throughout the regular robots.txt file in the foundation directory of the area. In addition, a webpage could be explicitly excluded from a internet search engine's databases https://news.rafeeg.ae