Most often indicates to assign something a standing or place to tell apart it from Other people in a group, as in txt file is then parsed and can instruct the robotic regarding which pages usually are not being crawled. As a search engine crawler may well keep a cached https://www.youtube.com/watch?v=4q42p8bnJy8