Our crawler
If you look after a museum website and have seen EnfiladeBot in your logs — that is us. This page says who we are, what we take and how to shut us out.
EnfiladeBot is the crawler of a museum catalogue. It gives its full name and leaves the address of this page; a robot with no return address any administrator is entitled to block without discussion.
Opening hours, prices, address, telephone, the ticket link and the list of exhibitions — the same things a visitor reads on your pages. We take no images, keep no whole pages and crawl nothing beyond those sections. Everything taken is credited to you with the date it was read, and from the entry a link leads back to your own page.
No more than one request every two seconds to a single host, and at night, when nobody is at the museum. The request is conditional (ETag and If-Modified-Since), so that almost always we receive “not modified” and take nothing. Each class of fact is revisited on its own schedule: exhibitions weekly, hours quarterly, the address once a year.
A line in robots.txt — we parse it and obey it, and we settle doubtful cases against ourselves: a disallow outweighs an allow of equal length. A rule naming us replaces the general one entirely.
Mark up your opening hours and prices with schema.org — OpeningHoursSpecification, Offer, ExhibitionEvent. Then we shall take what you have declared yourselves, and your hours will stop breaking with every redesign. Of sixty-one museum sites, exactly one has done this.
Write to us through the contact form — a request to stop crawling is honoured the same day and without argument.