Initially, website owners submitted the tackle of a website page, or URL to the various engines like google, which would mail an online crawler to crawl that page, extract links to other web pages from it, and return facts observed within the page being indexed.[5] txt file is then parsed https://www.youtube.com/watch?v=0USI3vTZ0aw