Dignity nobility high quality condition highness standing course caste position superiority priority style difference dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and may instruct the robotic as to which pages are certainly not to become crawled. For a internet search engine crawler https://www.youtube.com/watch?v=HmdZ6Ymv04I