Recall@5 Comparison: bge-m3 vs nomic-embed-text v1.5 vs text-embedding-3-small

# I Benchmarked Three Embedding Models on 50k Support Tickets > text-embedding-3-small, nomic-embed-text, and bge-m3 — same 50k tickets, same retrieval eval. The winner surprised me. **Source:** Broke It (https://brokeit.dev/posts/i-benchmarked-three-embedding-models-on-50k-support-tickets/) **Published:** 2026-05-12 **Original source:** https://huggingface.co/BAAI/bge-m3 **Topics:** embeddings, rag, vector-db, benchmarks --- **Three models. 50,342 tickets. Same retriever, same chunking, same eval set. Here's the table.** | Model | Recall@5 | MRR@5 | Ingestion (50k docs) | Cost (50k docs) | |---|---|---|---|---| | text-embedding-3-small | 0.71 | 0.58 | 4m 12s (API) | $0.038 | | nomic-embed-text v1.5 | 0.68 | 0.54 | 11m 08s (local, M2) | $0 | | bge-m3 | **0.79** | **0.65** | 18m 41s (local, M2) | $0 | bge-m3 won by 8 Recall@5 points over OpenAI's model. On the Spanish/Portuguese subset (n=47), the margin blew out further: bge-m3 at 0.82, text-embedding-3-small at 0.61, nomic at 0.55. If your corpus has any multilingual content, skip straight to bge-m3 — the other two aren't competitive. Even on English-only, the recall gap is wide enough to matter before you touch anything else in your pipeline. Pin the commit hash. I'll explain why at the end. --- **Corpus:** 50,342 support tickets, SaaS product, mostly English with Portuguese and Spanish from Latin American customers. **Eval:** 200 labeled query-ticket pairs as ground truth. **Task:** retrieve top-5 most relevant tickets per query. ## The Three Models **text-embedding-3-small** (OpenAI, 1536 dims, $0.02/1M tokens). Default choice for most teams. Fast to spin up, no infra to manage. **nomic-embed-text v1.5** (Nomic, 768 dims, Apache-2.0). Local via Ollama, 137M parameters. Their benchmark numbers looked aggressive on paper. **bge-m3** (BAAI, 1024 dims, MIT). Designed for multilingual. bge-m3 handles multilingual input natively — no separate model for Spanish, no preprocessing hacks. ## Setup Qdrant v1.9.2, HNSW index (`m=16`, `ef_construct=100`). Same chunking for all three: 512 tokens, 64-token overlap, tiktoken cl100k. Batch size 256. Cosine similarity for all three — no per-model tuning, to keep the comparison clean. Recall@5 and MRR@5 as primary metrics. ## Full Results The numbers from the top of this post, broken out further by language subset:

Recall@5 across three embedding models on the same 50k-ticket corpus. Higher is better. bge-m3 outperforms by 8 points over OpenAI's model.

**Spanish/Portuguese subset (n=47):** bge-m3 0.82, text-embedding-3-small 0.61, nomic 0.55. The multilingual gap is larger than the overall gap. Corpus with any non-English content: the decision is settled before you look at cost. ## Caveats and Limits **nomic-embed-text context window.** Advertised as 8192 tokens. Anything over 512 tokens returned vectors that looked numerically reasonable but scored poorly in retrieval. Took two hours to trace as a model behavior rather than an indexing artifact. Check your actual vector similarity scores on known-similar pairs before you blame the retriever. **bge-m3 is slow.** 18 minutes on an M2 Pro for a one-time 50k index build is acceptable. For incremental updates at volume, you'll want a GPU endpoint or a hosted option — the weights are 2.3GB on disk. I haven't run the incremental-update test yet; if the numbers change, I'll update this. **Scope.** I tested retrieval quality only. Downstream generation quality with each embedding set is a separate question — one I'm tracing in the context of fine-tuning vs prompting for intent classification, which uses an overlapping labeled dataset. **Not tested:** `text-embedding-3-large` (3072 dims, $0.13/1M tokens vs $0.02/1M). OpenAI's MTEB evals suggest it might narrow the gap with bge-m3. At 6.5× the cost, I want evidence before I run that experiment. ## The Versioning Issue Pin the commit hash for `BAAI/bge-m3` on Hugging Face. The repository has silent revisions — three weeks after my initial index build, a re-index returned different scores with no config change on my side. No changelog. No version tag. The scores shifted by enough to matter in a production comparison. Pin the commit, note it in your infra config, and you skip that problem entirely.