Running Ollama in GitHub Actions CI: What Actually Works

The pitch is obvious: run a local LLM in CI to test your prompts without paying per-token, without hitting rate limits, without stubbing the model and pretending the tests mean something. What nobody writes in the pitch is the six things that fail first.

I spent two weeks getting Ollama running reliably in GitHub Actions. What follows is the full picture — the setup that works and the things that burned time getting there.

What Broke First

**Model download time.** `llama3.2:3b` is 2GB. `mistral-nemo` is 7GB. GitHub-hosted runners don't cache between jobs unless you explicitly use `actions/cache`. First run: 8 minutes just downloading the model. If you're running on every PR, that's a problem.

**Runner memory.** The default `ubuntu-latest` runner has 7GB RAM. `llama3.2:3b` needs about 3.5GB model memory plus the runner overhead. `llama3.2:8b` OOM-kills the runner. I found this out by trying `llama3.2:8b` first, because the 8b model gives better results in my local evals and I figured the runner could handle it. It cannot. The job dies silently — no OOM error in the logs, just a cancelled run with exit code 137. I spent an hour thinking the Ollama install was broken before I looked at the runner memory dashboard and saw a flat line at 100% just before the kill. Then I spent another 45 minutes convinced the problem was a systemd memory limit on the runner — I was tweaking ulimits in the workflow YAML, rerunning, getting the same exit 137, convinced there was some container-level cap I hadn't found yet. There wasn't. It was just OOM. I should have read the exit code and moved on in five minutes.

**Ollama startup race.** The standard advice is `ollama serve &` then immediately run `ollama pull`. This fails intermittently — the pull starts before the server is ready. You need to poll the health endpoint.

**Timeout costs.** GitHub Actions charges by the minute on larger runners. A flaky test that hangs waiting for a model response that never comes costs real money.

The Working Setup

- name: Start Ollama
  run: |
    curl -fsSL https://ollama.com/install.sh | sh
    ollama serve &
    # Wait for Ollama's REST API to be ready
    for i in $(seq 1 30); do
      curl -s http://localhost:11434/api/tags && break
      sleep 2
    done

• name: Pull model (cached)
• uses: actions/cache@v4
• with:
• path: ~/.ollama/models
• key: ollama-llama3.2-3b-${{ runner.os }}

• name: Ensure model is present
• run: ollama pull llama3.2:3b

• name: Run LLM integration tests
• timeout-minutes: 10
• env:
• OLLAMA_HOST: http://localhost:11434
• LLM_MODEL: llama3.2:3b
• run: npm test -- --grep "llm-integration"
• ```

The cache step is load-bearing. After the first run, `~/.ollama/models` is cached by key and the download step becomes a no-op. Cache hit time: ~35 seconds to restore 2GB. Without cache: 8-12 minutes.

The `timeout-minutes: 10` on the test step is also non-negotiable. Ollama can hang silently on certain prompts, and without a hard timeout a hung test will eat your entire job runtime, billing you for the privilege.

What the Tests Actually Look Like

I'm not testing whether the LLM produces the "right" answer. Correctness under varying inputs is an eval problem, not a CI problem. What the CI tests are actually checking:

• Does the prompt render correctly with real data substituted in?
• Does the structured output (JSON mode) parse without errors?
• Does the response length stay under our context budget?
• Does error handling work when the model times out?

test('classification prompt returns valid JSON', async () => {
  const response = await ollamaClient.generate({
    model: 'llama3.2:3b',
    prompt: buildClassificationPrompt(FIXTURE_TICKET),
    format: 'json',
    options: { temperature: 0, num_predict: 200 }
  });
  const parsed = JSON.parse(response.response);
  assert.ok(parsed.intent, 'response must have intent field');
  assert.ok(VALID_INTENTS.includes(parsed.intent), 'intent must be in schema');
});

This catches real bugs: prompt template errors, schema mismatches, context window overflows. It does not catch "the model is giving wrong answers" — that's a separate eval harness.

Cost and Timing on GitHub-Hosted Runners

On `ubuntu-latest` (2-core, 7GB): ~4 minutes per run after cache is warm. llama3.2:3b is the largest model that fits comfortably.

On `ubuntu-latest-4-core` (4-core, 16GB): ~2.5 minutes, and you can run `llama3.2:8b` without OOM. Costs 2× per-minute. Usually not worth it for prompt-format tests.

Self-hosted runner with an M-series Mac: 90 seconds cold, 45 seconds warm. This is what I'd use if CI runs heavily enough that the minutes add up.

The Security Angle

This setup only runs local inference — no data leaves the runner. That's useful when test fixtures contain customer-adjacent data. We keep a small set of real (anonymized) support tickets in the test fixtures, and we're not comfortable sending those to an API endpoint on every PR.

It also means we caught the pattern where prompt injection broke our customer-facing chatbot before it had to be debugged in production — the injection test cases run in CI against the local model, which is exactly where that kind of discovery should happen.

I had initially set up the injection tests to only run on a manual trigger, thinking they were too slow for every PR. That was the wrong call — once I put them in the standard PR check with `llama3.2:3b`, the total runtime stayed under 4 minutes and we caught a regression in the input classifier within the first week.

Lessons kept short

The two steps that are actually load-bearing are the health-check poll (prevents the startup race) and the model cache (cuts 8-12 minutes to 35 seconds after the first run). Everything else — the timeout, the model size constraint, the test scope — is stuff you'd figure out from the error messages, though you'd rather not.

One thing I'm still not happy with: the `llama3.2:3b` quality ceiling. The 3b model is good enough for prompt-format and JSON-schema tests, but there are structural reasoning checks I've had to move to manual review because the local model doesn't catch them reliably. Using a 4-core runner with the 8b model would fix some of that, but at 2× the per-minute cost for tests that run on every PR it stops making economic sense. There's probably a smarter split — lighter model for the format checks, heavier model on a scheduled run for the reasoning checks — that I haven't built yet.