Scholar Data
Sign in
Back to Announcements

A note on recent uptime

Scholar Data has seen a sharp rise in traffic from AI scrapers, and our rate limiting hasn't always kept pace. Here's what's happening and what we're doing about it.

Some of you may have noticed occasional slowdowns or errors on Scholar Data over the past few weeks. We want to explain what's going on.

What's happening

We've seen a sharp increase in traffic from AI scraping bots crawling the site, including OpenAI's GPTBot, Anthropic's ClaudeBot, and Meta's external agent. These bots often arrive in bursts and don't always respect standard crawling etiquette, which puts unexpected load on our servers. We rate limit known bots automatically, but new scrapers and user agents show up faster than we can add rules for them, so there are windows where we're behind the curve and real users can feel the impact.

What we're doing

We're actively tightening our bot detection and rate limiting, and monitoring traffic closely to respond faster as new scrapers appear. Check /status any time for a live read on system health.

What to expect

You may still see brief periods of slower response times while we tune this. Reliability for real users is our priority, and we'll keep improving our defenses in the background. Thanks for your patience.