We at the National Library are about to begin our 2018 New Zealand domain crawl. As in
previous years, the crawl is being run by Internet Archive, and some of you may have
already seen the crawlers as IA go through their initial setup with some test crawls.
The user agent string their crawlers will be using is: NLNZ_IAHarvester2018
While we do our best to crawl respectfully, obey robots.txt, and break out of any crawler
traps before grinding too much bandwidth, if you encounter any issues from us then please
get in contact.
Ben O’Brien | Digital Preservation Web Engineer | Preservation, Research and Consultancy
National Library of New Zealand | Te Puna Mātauranga o Aotearoa
PO Box 1467, Wellington 6140, New Zealand |