I Benchmarked Three Embedding Models on 50k Support Tickets
text-embedding-3-small, nomic-embed-text, and bge-m3 — same 50k tickets, same retrieval eval. The winner surprised me.
**Three models. 50,342 tickets. Same retriever, same chunking, same eval set. Here's the table.**
| Model | Recall@5 | MRR@5 | Ingestion (50k docs) | Cost (50k docs) |
|---|---|---|---|---|
| text-embedding-3-small | 0.71 | 0.58 | 4m 12s (API) | $0.038 |
| nomic-embed-text v1.5 | 0.68 | 0.54 | 11m 08s (local, M2) | $0 |
| bge-m3 | **0.79** | **0.65** | 18m 41s (local, M2) | $0 |
bge-m3 won by 8 Recall@5 points over OpenAI's model. On the Spanish/Portuguese subset (n=47), the margin blew out further: bge-m3 at 0.82, text-embedding-3-small at 0.61, nomic at 0.55. If your corpus has any multilingual content, skip straight to bge-m3 — the other two aren't competitive. Even on English-only, the recall gap is wide enough to matter before you touch anything else in your pipeline.
Pin the commit hash. I'll explain why at the end.
**Corpus:** 50,342 support tickets, SaaS product, mostly English with Portuguese and Spanish from Latin American customers. **Eval:** 200 labeled query-ticket pairs as ground truth. **Task:** retrieve top-5 most relevant tickets per query.
The Three Models
**text-embedding-3-small** (OpenAI, 1536 dims, $0.02/1M tokens). Default choice for most teams. Fast to spin up, no infra to manage.
**nomic-embed-text v1.5** (Nomic, 768 dims, Apache-2.0). Local via Ollama, 137M parameters. Their benchmark numbers looked aggressive on paper.
**bge-m3** (BAAI, 1024 dims, MIT). Designed for multilingual. bge-m3 handles multilingual input natively — no separate model for Spanish, no preprocessing hacks.
Setup
Qdrant v1.9.2, HNSW index (`m=16`, `ef_construct=100`). Same chunking for all three: 512 tokens, 64-token overlap, tiktoken cl100k. Batch size 256. Cosine similarity for all three — no per-model tuning, to keep the comparison clean.
Recall@5 and MRR@5 as primary metrics.
Full Results
The numbers from the top of this post, broken out further by language subset:
**Spanish/Portuguese subset (n=47):** bge-m3 0.82, text-embedding-3-small 0.61, nomic 0.55. The multilingual gap is larger than the overall gap. Corpus with any non-English content: the decision is settled before you look at cost.
Caveats and Limits
**nomic-embed-text context window.** Advertised as 8192 tokens. Anything over 512 tokens returned vectors that looked numerically reasonable but scored poorly in retrieval. Took two hours to trace as a model behavior rather than an indexing artifact. Check your actual vector similarity scores on known-similar pairs before you blame the retriever.
**bge-m3 is slow.** 18 minutes on an M2 Pro for a one-time 50k index build is acceptable. For incremental updates at volume, you'll want a GPU endpoint or a hosted option — the weights are 2.3GB on disk. I haven't run the incremental-update test yet; if the numbers change, I'll update this.
**Scope.** I tested retrieval quality only. Downstream generation quality with each embedding set is a separate question — one I'm tracing in the context of fine-tuning vs prompting for intent classification, which uses an overlapping labeled dataset.
**Not tested:** `text-embedding-3-large` (3072 dims, $0.13/1M tokens vs $0.02/1M). OpenAI's MTEB evals suggest it might narrow the gap with bge-m3. At 6.5× the cost, I want evidence before I run that experiment.
The Versioning Issue
Pin the commit hash for `BAAI/bge-m3` on Hugging Face. The repository has silent revisions — three weeks after my initial index build, a re-index returned different scores with no config change on my side. No changelog. No version tag. The scores shifted by enough to matter in a production comparison. Pin the commit, note it in your infra config, and you skip that problem entirely.