Latest / Logosofia no Dia a Dia / Embedding Models vs LLMs: What Actually Connects?
Transcript
- [Prompt] custom: One of the few aspects of AI that is really affordable is embedding models. These are generally very | Hosts: corn, herman [Response] Corn: Daniel sent us this one — and it's the kind of question that sits in the back of your mind while you're building something, then suddenly you realize you've been ignoring it for six months. He's got two questions about embedding models in RAG pipelines. First: does your choice of embedding model constrain which LLM you can use? We're running DeepSeek v4 Pro for the script and Pinecone for retrieval, with whatever auto-embedding model Pinecone picked — and Daniel's honest enough to admit he's forgotten which one that is, which I respect. Second question: if embeddings are already cheap and reliable and basically feel like a solved problem, why are new embedding models still dropping every month? What's actually improving? Herman: Both questions cut to something real. The first one is a portability anxiety — are we accidentally building a pipeline that only works with one specific stack? The second one is the quiet suspicion that maybe the embedding model churn is just benchmarks chasing benchmarks, and none of it actually matters for retrieval quality. Corn: And the tension is that embeddings are simultaneously the most boring part of a RAG pipeline and the part where silent failures cause the most damage. A bad embedding model doesn't crash — it just returns documents that are slightly wrong, and your users slowly lose trust without knowing why. Herman: That's the thing Daniel's really asking, even if he didn't phrase it exactly this way. When we say an embedding model is "better," what does better actually mean for someone running a production RAG system? Not on a leaderboard — in the actual retrieval results. Corn: Let's break this into two distinct questions. First, the compatibility question — does DeepSeek v4 Pro care what embedding model we use? Then the innovation question — what's actually driving the new model releases, and when should you care? Herman: The compatibility question is the cleaner one to settle, so let's start there. Embedding models and LLMs are fundamentally different neural networks trained for fundamentally different objectives. An embedding model is optimized for one thing — mapping text into a vector space where semantically similar texts land close together. It's trained with contrastive learning, pulling related sentences near each other and pushing unrelated ones apart. An LLM is optimized for next-token prediction — a completely different loss function, different training data, different architecture. Corn: They're not just different models — they're different species. One's a mapmaker, one's a storyteller. Herman: Exactly the right image. And because they operate in different representational spaces — embeddings produce fixed-length vectors, LLMs consume variable-length token sequences — there's no architectural coupling between them. Your embedding model could be from OpenAI, your LLM from DeepSeek, your vector database from Pinecone. None of them need to know about each other. Corn: Which means Daniel's instinct that he should know which embedding model Pinecone is using is correct — but not because DeepSeek v4 Pro cares. It doesn't. The concern is portability between the embedding model and Pinecone itself. Herman: The only hard constraint is dimensionality. If your Pinecone index expects fifteen thirty-six dimensions and you switch to a model that outputs ten twenty-four, you're re-indexing. That's a storage and operations problem, not an LLM compatibility problem. Corn: That's question one — they're decoupled. But that makes question two sharper. If any embedding model works with any LLM, and embeddings already feel commoditized, what are all these new models actually for? Herman: That's where it gets interesting. The innovation isn't about making embeddings universally "better" — it's about fixing specific failure modes that only show up when you push retrieval hard. Domain specialization, multilingual alignment, efficiency through dimensionality reduction. The new models aren't chasing a general benchmark — they're solving problems that general-purpose embeddings quietly fail at. Corn: The arc is: settle the decoupling question definitively, then dig into what "better" actually means when you look past the leaderboard. Herman: Let's nail down the architectural separation, because it's more fundamental than most people realize. An embedding model and an LLM don't just have different training objectives — they don't share weights, they don't share vocabularies, they don't even share training data. The embedding model learned to map "what is the capital of France" near "Paris is the capital of France" in vector space. The LLM learned that after the token sequence "the capital of France is" the next token is probably "Paris." Completely different games. Corn: One's playing semantic proximity, the other's playing token prediction. They could be trained by different companies on different continents with different datasets and they'd still interoperate fine, because the only thing that passes between them is text. Herman: The embedding model takes text in, spits a vector out. That vector gets stored in Pinecone. When a query comes in, the same embedding model encodes the query, Pinecone finds the nearest vectors, and returns the corresponding text chunks. The LLM never sees a vector — it only sees the retrieved text. So from the LLM's perspective, the embedding model is completely invisible. Corn: Which means the idea that you need OpenAI embeddings to work with an OpenAI LLM is pure superstition. You could use Cohere embeddings with DeepSeek, or Voyage embeddings with Claude, or a completely homegrown embedding model with any of them. Herman: This is where Daniel's setup is instructive. He mentioned Pinecone's auto-embedding — where Pinecone handles the embedding generation transparently. Looking at Pinecone's hosted models, they offer pinecone-text-embedding-v2 alongside integrations with OpenAI, Cohere, and Voyage AI. Whatever model is running under the hood, DeepSeek v4 Pro doesn't know and doesn't care. It just receives text. Corn: The compatibility question has a clean answer: no constraint. But you raised dimensionality earlier, and that's where the real portability concern lives. If Pinecone's auto-embedding model outputs, say, seven sixty-eight dimensions and we decide to switch to something that outputs fifteen thirty-six, we're re-indexing the entire vector store. Herman: And Pinecone's create_for_model API — as of the twenty twenty-five ten API version — handles this elegantly. You specify the model name, and it automatically configures the index with the correct dimensions. OpenAI's text-embedding-three-small at fifteen thirty-six dimensions, Cohere's embed-english-v-three at ten twenty-four — the API knows the mapping. But if you switch models, you create a new index. The old vectors don't fit the new index's dimension expectations. Corn: It's a storage migration problem, not a compatibility problem. The kind of thing that's annoying but well-understood — like switching database schemas. Herman: There's one subtlety worth flagging though. While any embedding works with any LLM, retrieval quality isn't uniform. If your embedding model was trained on general web text and your LLM is specialized for code generation, the embeddings might not capture the semantic distinctions that matter for code retrieval. Two code snippets that do different things might look similar to a general embedding model, and the LLM receives irrelevant context. Corn: The failure pattern isn't that it breaks — it's that it silently retrieves the wrong documents. The LLM still generates fluent text, but it's working with bad source material. Which is almost worse than an outright crash, because at least a crash gets your attention. Herman: And that's not a compatibility constraint — it's a quality constraint. The embedding model and LLM are technically decoupled, but the retrieval quality depends on whether the embedding model understands the domain well enough to surface what the LLM actually needs. If you're doing legal RAG with a general embedding model, it might not distinguish between "the court upheld the ruling" and "the court overturned the ruling" — those sentences are semantically similar in everyday English, but legally they're opposites. Corn: Think about how that plays out in practice. The LLM gets both documents in its context window — one saying upheld, one saying overturned. It doesn't know which one is the correct precedent because the retrieval step already failed to discriminate. So it either picks one at random or tries to synthesize both, producing an analysis that's internally coherent but legally nonsense. Herman: That's the nightmare scenario. And it's why I keep coming back to this point — the decoupling is real, but it doesn't absolve you of thinking about the embedding model. It just changes why you're thinking about it. You're not asking "will this work with my LLM?" You're asking "will this surface the right documents for my LLM? Corn: Which brings us back to Daniel's pipeline. The fact that Pinecone's auto-embedding abstracts the model choice is convenient, but it also means the model could change without anyone noticing. If Pinecone upgrades from one hosted model to another, retrieval behavior shifts — and nobody documented which model was in use originally. Herman: That's the silent drift problem. And it's exactly why Daniel's instinct to check is right, even though the LLM compatibility concern was a red herring. The documentation isn't for DeepSeek — it's for future us, when something changes and we need to know what changed. Corn: I want to pause on that for a second, because "silent drift" is one of those phrases that sounds abstract until you've lived through it. Can you give me a concrete scenario where this actually burns someone? Herman: Imagine you're running a customer support RAG system for a SaaS company. Your embedding model has been quietly mapping "I can't log in" to documentation about password resets, which is correct. Then Pinecone upgrades the auto-embedding model to something newer. The new model has better multilingual support — great for some users — but it subtly shifts the semantic neighborhood around authentication queries. Now "I can't log in" retrieves documentation about account creation instead of password resets. The LLM still generates helpful-sounding text, but it's telling users to create new accounts when they just need to reset their passwords. Corn: You don't notice until your support tickets spike and someone digs through six weeks of logs trying to figure out what changed. Herman: And if you never documented which model was running originally, you can't even confirm that the embedding model change was the culprit. You're just staring at a degradation with no obvious cause. Corn: The compatibility question is settled — embeddings and LLMs are decoupled, and the real portability concern is dimensionality migration. But that clean answer makes Daniel's second question more urgent. If embeddings are cheap and they work and you can plug any of them into any LLM, why are we seeing a steady stream of new models? Corn: It's the "solved problem" illusion. Embeddings are reliable enough that they fade into infrastructure — like electricity, you stop thinking about them until the lights flicker. But reliable doesn't mean optimal, and the flickering happens in specific ways that only show up at scale. Herman: Three failure pattern in particular. The first is out-of-domain generalization. Early embedding models — think OpenAI's ada-oh-oh-two — were trained overwhelmingly on general web text. That's fine for broad semantic search. But drop them into a specialized domain and they stumble in ways that aren't obvious until you audit the retrieval results. Corn: Give me a concrete example. Herman: Legal retrieval is the classic case. A general embedding model sees "the court upheld the lower ruling" and "the court overturned the lower ruling" as nearly identical — they're structurally similar sentences about court decisions. But legally, they're opposites. If your RAG system retrieves case law to support a legal argument and the embedding model can't distinguish upheld from overturned, you're building arguments on precisely the wrong precedents. Corn: The failure is silent. The LLM gets text that looks relevant, generates fluent analysis, and nobody catches the error unless a lawyer reads the source documents. Herman: Newer models address this through domain-specific fine-tuning. BGE-M-three and E-five-mistral use multi-stage training that preserves general performance while improving retrieval on specialized corpora — scientific papers, code repositories, medical literature. Cohere offers legal-specific fine-tunes that explicitly learn to distinguish terms of art that general models blur together. Corn: The innovation isn't "this model is better at everything" — it's "this model doesn't confuse overturned with upheld. Herman: That's the pattern. And it's worth noting that this isn't just about legal text. The same thing happens in medicine, where "the patient responded to treatment" and "the patient did not respond to treatment" are structurally identical sentences that a general embedding model will place right next to each other. Or in finance, where "the stock outperformed the market" and "the stock underperformed the market" get collapsed into the same semantic neighborhood. The negation word is a tiny edit distance but a massive semantic difference, and general models are often blind to it. Corn: That's a great way to frame it — the model is optimizing for structural similarity, but the domain expert cares about a single word that flips the entire meaning. And the general embedding model doesn't know which words are load-bearing in which contexts. Herman: failure pattern two is multilingual and cross-lingual retrieval. If you're searching documents in Hebrew and querying in English, older models often fail because they embed each language in a separate region of vector space. A Hebrew document about Jerusalem and an English query about Jerusalem land far apart, even though they're semantically identical. Corn: Which matters for Daniel's setup specifically — our lore books probably mix Hebrew and English. Herman: Cohere's embed-multilingual-v-three and BGE-M-three explicitly optimize for cross-lingual alignment. They're trained to pull semantically equivalent sentences in different languages close together in the same vector space. A Hebrew query and an English document about the same topic land in the same neighborhood. Corn: The second failure pattern is language boundaries, and the fix is collapsing them. What's the third? Herman: The third failure pattern is the one I find most technically elegant — fine-grained retrieval. Basic semantic search works great when documents are broadly different. But when you need to distinguish between "Python three point eleven documentation" and "Python three point twelve documentation," the semantic difference is tiny. Both are about Python documentation. General embeddings struggle to separate them. Corn: That's where the Matryoshka thing comes in. Herman: Matryoshka Representation Learning. The idea is that you train an embedding model to produce vectors where you can truncate dimensions and still preserve relative ranking. OpenAI's text-embedding-three-small lets you specify a dimensions parameter — you can request two fifty-six dimensions instead of the full fifteen thirty-six. The short vector keeps the same nearest-neighbor ordering as the full vector, just with slightly coarser resolution. Corn: You do a fast coarse search at low dimensions, then re-rank the top candidates at full dimensions. Herman: And the practical gain is substantial — retrieval quality at two fifty-six dimensions is nearly identical to fifteen thirty-six for most queries, but latency and storage costs drop by about six times. That's not a benchmark improvement — that's real infrastructure savings at scale. Corn: I want to make sure I understand the mechanism here. The Matryoshka training process explicitly optimizes the model so that the first two fifty-six dimensions carry the most important semantic signal, and each additional block of dimensions adds finer-grained detail. So truncating doesn't randomly lose information — it loses the least important information first. Herman: It's not like taking a high-resolution photo and cropping it arbitrarily. It's more like a progressive JPEG, where the coarse structure loads first and the fine details fill in afterward. The training objective enforces that ordering, so the model learns to pack the most discriminative features into the early dimensions. Corn: Which brings up the benchmark problem itself. Most embedding model comparisons use MTEB — the Massive Text Embedding Benchmark — which averages performance across fifty-plus tasks. Classification, clustering, semantic textual similarity, summarization, retrieval. Herman: A model can top MTEB by crushing tasks you don't care about while being mediocre at the one task your pipeline actually runs. The model that's best at clustering product reviews might be worse at retrieving legal documents than a domain-specific model that scores lower overall. Corn: Chasing the MTEB leaderboard is a trap. Herman: The smarter approach is evaluating on your own retrieval task. Sample a hundred queries from your domain, retrieve with candidate models, and measure precision at k. The best model for your data might rank fifteenth on MTEB. Corn: To pull this together — the innovation in embedding models isn't about making them universally better. It's about making them more specialized, more efficient through dimensionality reduction, and more robust to distribution shift. If your RAG pipeline works well today, upgrading just because a new model dropped probably won't help. But if you're hitting one of these specific failure pattern — bad domain retrieval, poor multilingual support, high latency from large dimensions — the new models directly address those. Herman: That's the answer to Daniel's second question. The progress is real, but it's narrow. The new models aren't replacing the old ones across the board — they're filling gaps that only became visible once embeddings were deployed at scale in production systems. Herman: The innovation is real but targeted. Let's turn that into practical guidance — four things a practitioner can actually do this week. Corn: First one is almost embarrassingly simple, but it's exactly what Daniel caught himself not doing. Document your embedding model. Model name, dimension, provider. If you're using Pinecone's auto-embedding like we are, go check which hosted model is actually running under the hood — Pinecone's docs list the supported models explicitly. Write it down somewhere your team can find it. Herman: Because the failure pattern here isn't catastrophic — it's silent drift. Pinecone upgrades the auto-embedding model, retrieval behavior shifts subtly, and six months later you're wondering why certain queries return slightly different results. If you never documented the original model, you can't even diagnose what changed. Corn: I'd add: put that documentation somewhere that survives team churn. Not in a Slack message from six months ago that nobody can find. Put it in the repo README, or a runbook, or wherever your team keeps operational knowledge. The half-life of a Slack message is about three days. Herman: Second: when you're choosing an embedding model, ignore the MTEB leaderboard. It averages fifty-plus tasks, most of which have nothing to do with your retrieval use case. Instead, sample a hundred queries from your actual domain, run them through two or three candidate models, and measure precision at k on the retrieved results. Corn: A hundred queries is manageable — you can do this in an afternoon. And the model that wins on your data might be fifteenth on MTEB. The benchmark leader is optimizing for an average across classification and clustering and semantic similarity tasks that your pipeline never runs. Herman: Let me add some texture to that. When you're building your evaluation set, don't just grab a hundred random queries from your logs. Make sure you've got examples of the hard cases — the queries where you know the retrieval has historically struggled, the edge cases where semantic similarity breaks down. If your evaluation set is all easy queries where any model would succeed, you're not actually measuring anything useful. Corn: An evaluation set that only contains "what is the capital of France" isn't going to discriminate between models. You need the queries where the answer lives in a document that's semantically adjacent to a wrong answer, and the model has to make a fine-grained distinction. Herman: Third insight is about future-proofing. If you think you might switch embedding models down the line, pick one that supports variable dimensions — OpenAI's text-embedding-three with the dimensions parameter is the obvious example — or standardize on a common dimension like seven sixty-eight or ten twenty-four that multiple models support. Re-indexing a vector database is tedious, and dimension flexibility gives you an escape hatch. Corn: The fourth one is the one that saves the most time: if your RAG pipeline is working well, don't touch it. A new embedding model dropping is not a reason to upgrade. The improvements are real but narrow — better multilingual alignment, better domain specificity, better efficiency at low dimensions. If you're not hitting those specific failure pattern, the upgrade is a migration cost with no retrieval gain. Herman: The hardest discipline in this space is knowing when not to move. Corn: Which raises a bigger question. As embedding models become more specialized — legal embeddings, medical embeddings, code embeddings — are we heading toward a marketplace of per-domain models? Or does the general-purpose model eventually get good enough across all of them that specialization becomes unnecessary? Herman: I think the marketplace is already forming. Cohere's legal fine-tunes, Voyage's code embeddings — these aren't general models that happen to work on legal text. They're trained specifically for those domains. And the economic logic is the same as any specialized tool. A general embedding model is a Swiss Army knife. It does everything adequately. But if you're a law firm building RAG for case law retrieval, you don't want adequate — you want the model that never confuses overturned with upheld. Corn: The counterargument is that models like BGE-M-three are already handling a hundred-plus languages and multiple retrieval modes in a single model. The trajectory might not be toward fragmentation — it might be toward models that are general enough to handle any domain without fine-tuning. Herman: I think both trajectories are happening simultaneously. The general models are getting broader, and the specialized models are getting deeper. The general model handles eighty percent of use cases well enough, and the specialized models capture the twenty percent where "well enough" isn't good enough. That's a healthy ecosystem, not a contradiction. Corn: The real wildcard is something you mentioned earlier in passing. Models like Gemini and GPT-four-o already produce embeddings internally as part of their architecture. If future LLMs expose those internal representations directly, the entire embedding model market collapses into the LLM. Herman: Because the embedding is the LLM. You query the model, it returns both the generated text and the vector representation of that text — all from the same weights, same training, same understanding. At that point, the compatibility question Daniel started with becomes moot in the most elegant possible way. There's no pairing decision because there's only one model. Corn: Retrieval quality stops being about which embedding model you chose and starts being about how good your LLM is at understanding text in the first place. Which is a much more intuitive axis to optimize on. Herman: Though I'll push back slightly on the utopian framing. If your embedding model is your LLM, then every time you upgrade your LLM, you're implicitly changing your embedding model too. You'd need to re-index your entire vector store every time a new model version drops, unless the provider guarantees embedding stability across versions — which nobody currently does. Corn: That's the dark side of the consolidation argument. Right now, the decoupling we talked about earlier is actually a feature — you can upgrade your LLM without touching your embeddings, and vice versa. If they collapse into a single model, you lose that independence. Every LLM upgrade becomes a full pipeline migration. Herman: Maybe the future isn't total consolidation. Maybe it's a world where LLMs expose stable embedding endpoints that are versioned separately from the generative capabilities. You get the benefit of shared representations without the migration cost. Corn: That's a few years out, but the pieces are already visible. For now, the advice stands: document what you're using, evaluate on your own data, and don't upgrade just because something new shipped. Herman: If you take nothing else from this episode, take this: the embedding model and the LLM don't know about each other, don't care about each other, and don't need to. The only thing that matters is whether the embedding model retrieves documents that help the LLM do its job. Everything else is infrastructure. Corn: If you enjoyed this deep dive, leave a review — it helps other practitioners find the show. And if you have a weird prompt or a question about your own RAG pipeline, send it in. Email the show at show at my weird prompts dot com. Herman: Now: Hilbert's daily fun fact. Hilbert: In the early Renaissance, a pigment known as "mosaic gold" was synthesized by alchemists in the Aral Sea basin by fusing tin with sal ammoniac and sulfur — producing tin sulfide, a golden crystalline compound used to imitate gold leaf in illuminated manuscripts. Corn:...huh. Fake gold from the Aral Sea. I have questions, but I'm not sure I want the answers. Herman: This has been My Weird Prompts. I'm Herman Poppleberry. Corn: I'm Corn. We'll catch you next time.