Skip to content
StackSignal
RAG and search • 9 min read

Open-Source vs. Proprietary Embedding Models: MTEB Leaderboard, Latency, and GPU Math

Benchmark OpenAI text-embedding-3 against open-source alternatives (BGE, BAAI, Cohere, Nomic, Mistral): self-hosted GPU costs vs API pricing at scale.

By the StackSignal Editorial Team

Selecting an embedding model is one of the most consequential decisions when architecting an enterprise vector search or RAG infrastructure. While proprietary cloud APIs—such as OpenAI’s text-embedding-3-small and text-embedding-3-large—offer frictionless pay-as-you-go integration, open-source embedding models have closed the quality gap on the Massive Text Embedding Benchmark (MTEB).

Understanding the trade-offs between API convenience, inference latency, dimensionality pruning (Matryoshka representation learning), and GPU infrastructure hosting economics determines long-term software viability.

MTEB Performance Comparison: Top Models

The table below benchmarks prominent proprietary and open-source embedding models evaluated on the MTEB English Retrieval sub-benchmark:

Model NameProvider / LicenseDimensionsContext WindowMTEB Retrieval (NDCG@10)Pricing / Token Cost
text-embedding-3-smallOpenAI (Proprietary)1,536 (Prunable)8,191 Tokens55.6$0.02 / 1M Tokens
text-embedding-3-largeOpenAI (Proprietary)3,072 (Prunable)8,191 Tokens64.6$0.13 / 1M Tokens
embed-english-v3.0Cohere (Proprietary)1,024512 Tokens64.5$0.10 / 1M Tokens
bge-large-en-v1.5BAAI (Apache 2.0)1,024512 Tokens64.1Free OSS (Self-hosted)
bge-m3BAAI (Apache 2.0)1,024 (Multi-ling)8,192 Tokens65.2Free OSS (Self-hosted)
nomic-embed-text-v1.5Nomic AI (Apache 2.0)768 (Matryoshka)8,192 Tokens62.4Free OSS (or $0.08/1M)
gte-Qwen2-7B-instructAlibaba (Apache 2.0)3,58432,768 Tokens70.2Free OSS (Heavy GPU)

Matryoshka Representation Learning (MRL): Pruning Storage Costs

Vector databases bill and allocate RAM directly proportional to vector dimensions ($D \times 4 \text{ bytes per float32}$). A 3,072-dimensional vector requires 12.3 KB of memory per chunk, creating massive RAM expenses across 10 million vectors.

Modern models (including OpenAI text-embedding-3 and Nomic embed-text) utilize Matryoshka Representation Learning (MRL). MRL trains the model so that information is concentrated into the earliest dimensions. You can slice the vector from 1,536 dimensions down to 512 dimensions via basic array truncation:

  • Storage / RAM Reduction: 66% decrease in vector database memory footprint.
  • Search Speedup: 3x faster HNSW vector distance calculations.
  • Accuracy Loss: Less than 1.5% decrease in MTEB retrieval performance.

Financial Break-Even: Cloud API vs. Dedicated Self-Hosted GPU

Should an enterprise host its own embedding inference service (using vLLM or Hugging Face TGI on AWS) or pay commercial API providers?

Scenario: 500 Million Tokens Ingested per Month

  1. OpenAI text-embedding-3-large:
    • 500M tokens * $0.13 / 1M = $65.00 / month
    • Maintenance overhead: 0 hours
    • Data privacy: Data sent to external commercial API.
  2. Self-Hosted bge-m3 on AWS (g5.xlarge with NVIDIA A10G 24GB):
    • On-Demand EC2 cost: $1.006 / hour * 730 hours = $734.38 / month
    • Reserved 1-Year Instance: ~$450.00 / month
    • Break-even token threshold: Self-hosting is economically disadvantageous for embeddings until ingest volume reaches 3 to 5 billion tokens per month, unless regulatory compliance (HIPAA, ITAR, GDPR) strictly prohibits third-party API transmission.

Sources and references

Frequently asked questions

What is the Massive Text Embedding Benchmark (MTEB)?
MTEB is the industry-standard benchmark that evaluates embedding models across 8 tasks: retrieval, clustering, classification, pair classification, reranking, summarization, semantic textual similarity (STS), and bitext mining.
Can open-source embedding models outperform OpenAI embeddings?
Yes. Top open-source models on MTEB (such as BAAI/bge-en-icl, SFR-Embedding, and nomic-embed-text) consistently outscore OpenAI text-embedding-3-large on retrieval tasks while supporting local data privacy.
At what document volume does self-hosting embeddings on GPUs become cheaper than commercial APIs?
At continuous ingestion workloads exceeding 200 million tokens per month (or strict air-gapped on-premise compliance environments), hosting an open-source model on a dedicated AWS EC2 g5.xlarge GPU is more economical than managed API calls.

About the author

Written by the StackSignal Editorial Team, which researches, writes, and tests every guide on this site. Read more about our editorial standards or contact us with suggestions.

Keep reading