AI bots and crawlers are flooding institutional repositories at an unprecedented scale. A 2025 survey of 66 research institutions found that over 90% face aggressive bot traffic at least weekly, causing service disruptions and tying up resources (Shearer & Walk, 2025). This isn’t hypothetical — in May 2026, the University of Zurich’s repository ZORA had to restrict access to its internal network just to stay online (Gerber, 2026). Beyond the technical strain, there’s an ethical tension: researchers happy to share data for science rarely agreed to it training AI models.
Current countermeasures — robots.txt, IP-blocking, rate-limiting — are reactive, temporary, and quickly bypassed: a constant cat-and-mouse game (Shearer & Walk, 2025; Weinberg, 2025). Worse, without monitoring, institutions often don’t even know how distorted their own usage numbers are — one institution’s reported traffic dropped from 1 million to 125,000 visits once bot activity was filtered out (Weinberg, 2025).
That’s the gap this challenge addresses: a prototype Crawler Monitor that makes bot activity visible and understandable — a foundation for data-informed decisions instead of reactive firefighting.
Full description (PDF)
Attachments:
Previous
GLAMhack 2026
Next project