I wired Claude into my CI pipeline — here's what actually broke

The setup took maybe 20 minutes. The cleanup took the rest of the weekend.

I wanted automated PR reviews on a monorepo — not AI slop suggestions, but real structural feedback: "this function is doing three things," "this new dependency is already handled by X," "you're not handling the error branch here." So I wired Claude into a GitHub Actions workflow triggered on `pull_request`.

The auth part is fine

You create an API key, put it in your repo secrets, pull it as an env var in the workflow YAML. Anthropic's API is well-documented; there's nothing tricky about the auth itself. If you've ever called any external API from Actions you already know how this goes.

- name: Review PR with Claude
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: python scripts/review_pr.py

That works. The problems start after you fire the first real request.

Rate limits hit faster than expected

If multiple PRs land within a few minutes — normal during a team's active hours — you'll hit rate limits fast. The default tier has a requests-per-minute cap that sounds generous until you're running reviews on five PRs simultaneously and each one fires two or three requests (I split the review into a diff analysis pass and a dependency check pass).

The fix is simple but non-obvious: wrap every API call with exponential backoff and jitter, cap your concurrency at the workflow level with `max-parallel`, and cache the model's response for the same diff hash so re-runs don't re-bill. I use a Redis instance for the cache; a file-based cache keyed on the diff SHA works too if you're cost-sensitive.

Context window overflow at 2 AM

This is the one nobody writes about until it bites them. A large PR — say, 800 lines changed across 20 files — blows past claude-3-5-sonnet's 200k context window when you also include the full file context for each changed file, plus the system prompt, plus the schema definition, plus whatever else you've packed in there and keep forgetting to account for. The API returns a clean error, your workflow marks the step failed, and you get a Slack alert at 2 AM that says "PR review failed" with no useful detail in the logs.

My first instinct was to increase the context limit. There isn't a knob to turn — 200k is the ceiling. So I had to actually fix the problem.

Two things worked: first, I stopped including full file context and instead included only the diff plus a shallow type/function signature summary of the surrounding file (generated with a `tree-sitter` parse). Second, I chunked large PRs by directory — each directory group becomes its own Claude call, and the results get stitched back together. You lose some cross-directory reasoning, but the reviews stay coherent.

For context on an alternative — I looked at running a local model in CI to sidestep the context and rate-limit issues entirely, but the quality gap on structural reasoning is large enough that I kept the API approach.

Output format drifts unless you enforce it

Claude does not consistently return structured review comments if you just prompt it in prose. Some runs return markdown lists. Some return numbered items. Some return prose paragraphs. If you're parsing the output to post inline GitHub review comments via the API, you need the format to be stable.

Fix: use the API's structured output / JSON mode. Define a schema with `required_fields`: `comments` (array of `{file, line, severity, body}`), and the model actually stays in it. Took me way too long to stop prompting for "please return JSON" and just use the schema enforcement. That's the trap — prompting for a format and enforcing a format are not the same thing.

Two weeks later

Schema enforcement should have been day one. Two days fighting inconsistent output before I stopped prompting for JSON and started using the schema parameter — recoverable, annoying. The chunking shape is still not stable. Directory-based chunking loses the thread on PRs where the important change is a one-line edit in a shared utility that ripples across three directories. I added a fallback that chunks by dependency graph instead. It took a production miss to see why directory chunking wasn't enough.

The auth setup is boilerplate. Review quality depends on chunking strategy and prompt design, not on anything Claude-specific. The edge-case reasoning is where the real work lives — that's the context behind each decision below.