To prevent unwanted material inside the research indexes, website owners can instruct spiders to not crawl certain data files or directories in the standard robots.txt file in the foundation Listing on the domain. Additionally, a site could be explicitly excluded from the search engine's databases by utilizing a meta tag https://www.imdb.com/title/tt30110854/?ref_=wl_t_1