Crawler Monitor for Institutional Repositories

When does online traffic become an attack?

Fachhochschule Nordwestschweiz FHNW

↓  Open

AI bots and crawlers are flooding institutional repositories at an unprecedented scale. A 2025 survey of 66 research institutions found that over 90% face aggressive bot traffic at least weekly, causing service disruptions and tying up resources (Shearer & Walk, 2025). This isn’t hypothetical — in May 2026, the University of Zurich’s repository ZORA had to restrict access to its internal network just to stay online (Gerber, 2026). Beyond the technical strain, there’s an ethical tension: researchers happy to share data for science rarely agreed to it training AI models.

Current countermeasures — robots.txt, IP-blocking, rate-limiting — are reactive, temporary, and quickly bypassed: a constant cat-and-mouse game (Shearer & Walk, 2025; Weinberg, 2025). Worse, without monitoring, institutions often don’t even know how distorted their own usage numbers are — one institution’s reported traffic dropped from 1 million to 125,000 visits once bot activity was filtered out (Weinberg, 2025).

That’s the gap this challenge addresses: a prototype Crawler Monitor that makes bot activity visible and understandable — a foundation for data-informed decisions instead of reactive firefighting.

Full description (PDF)

Attachments:

All attendees, sponsors, partners, volunteers, and staff are required to abide by the Hack Code of Conduct. The organisers are committed to enforcing it throughout the event and expect everyone to contribute to creating a safe, respectful, and welcoming environment for all.

The contents of this website, unless otherwise stated, are licensed under a Creative Commons Attribution 4.0 International License. The application that powers this site is available under the MIT license.

This dataset was generated using artificial intelligence and is not subject to copyright protection. It is available in the public domain for unrestricted reuse, modification, and distribution. https://creativecommons.org/public-domain/