We see the future of web archiving relying less on robots.txt file declarations geared toward search engines, and more on representing the web as it really was, and is, from a user’s perspective.

Källa: Robots.txt meant for search engines don’t work well for web archives | Internet Archive Blogs