Four things that break a scraper in production

A scraper that works once is a script. A scraper that works every morning at 6am for a year is infrastructure, and the gap between the two is where most projects quietly die.

None of what follows is exotic. These are the four failure modes that account for nearly every scraper that stopped working without anyone noticing.

1. Silent partial failure#

This is the one that actually costs money.

Your scraper doesn't crash. It returns 200 OK, parses the page, and writes rows to the database. It's just that the selector for the price field stopped matching last Tuesday, so every row since then has null where the price should be.

HTTP status monitoring will never catch this. Neither will a try/catch. The request succeeded — it's the meaning that broke.

What to do instead: assert on the shape of the result, not the success of the request.

function assertShape(rows: Row[]) {
  if (rows.length === 0) {
    throw new Error('scrape returned zero rows — selector likely broke');
  }

  // A field that is *sometimes* null is normal. A field that is
  // *always* null across a full page means the selector is dead.
  const priced = rows.filter((r) => r.price !== null).length;
  if (priced / rows.length < 0.5) {
    throw new Error(
      `only ${priced}/${rows.length} rows have a price — check the parser`,
    );
  }
}

Two cheap checks with outsized value: row count against yesterday's (a 60% drop is a broken selector, not a quiet news day) and null rate per field. Both catch partial breakage in one run instead of one quarter.

2. The IP that used to work#

Datacenter IPs are cheap and get blocked. Residential IPs are expensive and get blocked more slowly. Neither is permanent.

What makes this hard to debug is that blocking is rarely binary. You'll see:

• Soft blocks — a 200 OK serving a page with none of your target content
• Redirect walls — every request bouncing to a challenge page
• Selective degradation — the site returns real data for the first few pages and a stripped-down version afterwards

The last one is the worst, because your first page of test results looks perfect.

What to do instead: treat "content missing" as a distinct outcome from "request failed". If a page returns 200 but none of your required selectors match, log it as blocked rather than parse_error. Once you can count blocks separately, you can actually see whether a proxy pool is degrading.

A block rate that's climbing week over week is a scraper with weeks to live. You just can't see it if blocks are being logged as parse errors.

3. Timing assumptions that were never true#

The classic: your scraper works fine at 3am when you're testing and falls over at 9am when the site is under real load.

Several flavours of this:

• Timeouts tuned to a fast night. A 5-second timeout that's generous at 3am is aggressive at peak.
• Concurrency that's polite on one target. Ten parallel requests is nothing for a large site and a denial-of-service for a small one.
• Retries without backoff. A retry storm turns one slow response into a self-inflicted block.
• No jitter. Every scraper on a cron hits at exactly :00. Adding a random 0–120s offset costs nothing and makes your traffic look less like a bot.

What to do instead: exponential backoff with jitter, and a concurrency limit you set deliberately rather than inherit from whatever your HTTP library defaults to.

async function withBackoff<T>(fn: () => Promise<T>, attempts = 4): Promise<T> {
  for (let attempt = 0; attempt < attempts; attempt++) {
    try {
      return await fn();
    } catch (error) {
      if (attempt === attempts - 1) throw error;
      // 1s, 2s, 4s … plus up to 1s of jitter so parallel workers
      // don't all retry on the same tick.
      const delay = 2 ** attempt * 1000 + Math.random() * 1000;
      await new Promise((resolve) => setTimeout(resolve, delay));
    }
  }
  throw new Error('unreachable');
}

4. Nobody owns it#

The least technical failure and the most common one.

A scraper gets written for a project, works, and becomes load-bearing. The person who wrote it moves on. Six months later it breaks and the people depending on the data don't know it exists, let alone how to fix it.

What to do instead: three things, none of which take an afternoon.

• A README next to the code — what it scrapes, what breaks it, what the output feeds.
• An alert that goes to a channel, not a person. Alerts routed to an individual expire when they change teams.
• A comment on every non-obvious selector explaining what it's anchored to. .a-price-whole means nothing in six months; // price, anchored to the whole-number span — Amazon splits cents into a sibling means everything.

The short version#

Three of these four are detected by monitoring the shape of your output rather than the status of your requests. If you only do one thing on this list, do that one.

FAQ#

Why does my scraper return empty results without erroring?#

Almost always a soft block: the site returns a 200 OK with a challenge page or a stripped-down version instead of the content. Because the request technically succeeded, error handling doesn't fire. Check whether your required selectors matched, and treat "no match" as its own failure category.

How do I know if my IP is blocked?#

Look for 200 responses whose content is missing your target selectors, redirects to challenge or verification pages, and a success rate that degrades as a run progresses. Count those separately from parse errors so a rising block rate is visible before it becomes total.

What's a safe request rate?#

There's no universal number — it depends entirely on the target's size. Start conservative, add exponential backoff with jitter, set an explicit concurrency limit rather than accepting your HTTP client's default, and respect robots.txt and any published rate limits.

Should I use headless browsers for scraping?#

Only when you need them. Headless browsers are far slower and more resource-hungry than plain HTTP requests. Try fetching the HTML directly first — a surprising number of sites that look JavaScript-rendered will serve complete markup to a simple request.

How do I stop a scraper from breaking silently?#

Assert on output shape every run: compare row count against the previous run, and check the null rate per field. A selector that breaks produces a sharp change in both, which is detectable in one run rather than one quarter.