Robots.txt meant for search engines don’t work well for web archives
We see the future of web archiving relying less on robots.txt file declarations geared toward search engines, and more on representing the web as it really was, and is, from a user’s perspective.Källa: Robots.txt meant for search engines don’t work well for web archives | Internet Archive Blogs