Latest / Logosofia no Dia a Dia / Custom GPTs: When to Build, When to Skip
Transcript
- [Prompt] custom: I got into AI in a big way via OpenAI's custom GPT feature. This is one of those products that I kee | Hosts: corn, herman [Response] Corn: Daniel sent us this one — he built a custom GPT today for his car's user manual. Uploaded the PDF, added the VIN-specific codes, gave it a role, and now he's got a mechanic in his pocket that knows his exact trim level. But here's the thing. He's also built, by his own count, forty-seven other custom GPTs he never touched again after the first week. And that tension is exactly what we're going to sit with today. Herman: Forty-seven is a very specific number. I respect the candor. Corn: It's the kind of honesty you only get from someone who's stared into the abyss of the "Create a GPT" button and clicked it forty-eight times. The question he's asking is where the dividing line actually is — when does a custom GPT earn its keep versus when are you just cosplaying as a prompt engineer for an audience of nobody? Herman: This question matters right now more than it did a year ago, because the GPT store is still here. Mid two thousand twenty-six, roughly three million custom GPTs, and every six months someone writes a thinkpiece about how OpenAI's about to sunset the whole thing. Hasn't happened. The platform's stubbornly alive, which means these tools have real staying power. But the actual sweet spot — the set of problems where a custom GPT is clearly better than just opening a fresh chat — is way narrower than three million GPTs would suggest. Corn: So we're not here to do a product review. This is a decision framework. When do you build one, when do you skip it, and what's actually happening under the hood when you upload that PDF and click publish. Herman: I want to name the tension upfront, because Daniel nailed it without quite saying it this way. Custom GPTs are simultaneously underrated and overused. They do something powerful that most people don't understand — but they also get built for tasks where the base model would do the job just as well, faster, with zero maintenance. Corn: The car manual is the platonic ideal. We'll come back to why. The Jerusalem restaurant recommender is the cautionary tale. The base model already knows most of those restaurants, the hours change faster than you'll update the knowledge base, and you've just volunteered for a maintenance job nobody asked you to do. Herman: Let's get into it. What are the actual levers a custom GPT gives you, what do they cost you, and how do you know which side of the line you're on? Herman: The three levers. And I want to be precise here because most people throw around "custom GPT" like it's one thing, when it's actually three very different tools wearing the same trench coat. Corn: The trench coat is the "Create a GPT" button. Herman: Lever one is the system prompt. You're injecting instructions that get prepended to every single conversation. Define the role, set the tone, constrain the behavior. This is the one everyone uses and the one that's easiest to misuse. Corn: Because you can just tell the base model "you are a helpful automotive expert" in a fresh chat and get roughly the same thing. Herman: Right — for a single session. The system prompt's value isn't the role definition in isolation. It's persistence. Every time you open that GPT, the same constraints are there, no copy-paste, no re-explaining. That matters when you're sharing it with family members who don't want to type a paragraph of instructions every time they have a check-engine light. Corn: My leaf medicine practice has a very similar intake process. Herman: I'm going to pretend you didn't say that. Lever two is knowledge base upload. You drop in PDFs, spreadsheets, text files, and OpenAI chunks them, embeds them using text-embedding-three-large, and retrieves relevant chunks at inference time via cosine similarity. This is retrieval-augmented generation — RAG — but without you having to stand up a vector database or write a single line of infrastructure. Corn: Which is remarkable and also the source of most of the disappointment people feel with custom GPTs. They upload a document and assume the model learned it. It didn't. It's searching it. Herman: That's the single biggest misconception. Custom GPTs are not fine-tuned. There are no weight updates happening. You're giving the model a reference library it can flip through, not a textbook it's studied. And retrieval quality depends entirely on how well your documents chunk and whether the embedding model pulls the right passages for a given query. Upload a poorly formatted PDF with mixed fonts and scanned tables, and your "expert GPT" is basically guessing. Corn: Which brings us to lever three — custom actions. This is where you define an OpenAPI three-point-oh spec and the model can actually call external APIs. Query real-time data, pull your car's diagnostic codes from a live database, check the NHTSA recall registry. This is function calling for consumers, and it's the only lever that does something the base model cannot do on its own. Herman: It's also the lever that breaks most dramatically. Auth tokens expire. The model hallucinates parameters that look plausible but don't match the actual schema. I've built custom GPTs with actions that worked beautifully for three weeks and then silently started returning garbage because the endpoint added a required field. Corn: You've got persistence, reference retrieval, and external function calling. Three useful capabilities. And every single one of them comes with a maintenance bill. The system prompt drifts as your use case evolves. The knowledge goes stale. The actions break. And the base model, meanwhile, keeps getting better — GPT-four-o's context window is a hundred twenty-eight thousand tokens now, and there are rumors of GPT-five pushing past a million. Herman: Which means the gap between "build a custom GPT" and "just paste the document into a chat" is shrinking. Not gone — we'll get into where it still matters — but shrinking. And that's why this question Daniel's asking is more urgent now than when the GPT store launched in January twenty twenty-four. Three years in, the novelty has worn off, and what's left is a cold calculus: does the specificity of my use case justify the overhead? Corn: That's the thesis I want to plant right here before we go deeper. The dividing line isn't about capability. It's about specificity of reference. Custom GPTs win when your use case requires a fixed, bounded set of facts the model cannot know — your car's exact VIN-specific codes, your company's internal style guide, the proprietary framework your team built that has no public documentation. They lose when the task is general reasoning, common knowledge, or anything the base model already ingested during training. Herman: The car manual is bounded knowledge. One PDF, one vehicle, codes that don't change. The Jerusalem restaurant recommender is unbounded — the model already knows the restaurants, the hours shift weekly, and you're not going to update that knowledge base every time a hummus place changes ownership. Corn: Which, in Jerusalem, is approximately every forty-five minutes. Herman: That's not in the training data but it should be. So the framework we're building here — and this is what the rest of the episode is about — is how to look at your idea and know, before you click that button, whether you're building the car manual or the restaurant recommender. Herman: Let's get under the hood of the system prompt first, because this is where I see people burn the most tokens without realizing it. The mechanism is deceptively simple. Every time you send a message to a custom GPT, OpenAI takes your system prompt — all of it — and prepends it to the conversation before anything else. Corn: If your system prompt is two thousand tokens, you're paying that tax on message one, and again on message two, and again on message twelve. Herman: Two thousand tokens is roughly fifteen hundred words. That's about one point five percent of GPT-four-o's hundred twenty-eight thousand token context window, burned before the model even sees your actual question. Do that across a thirty-message conversation and you've spent forty-five thousand tokens just on system instructions. That's over a third of the context window, gone, on something you wrote once and probably haven't looked at since. Corn: Most people's system prompts are bloated. They write paragraphs about personality, add "always do this, never do that," include example responses, define edge cases that never come up. I've seen system prompts longer than the documents they're supposed to be referencing. Herman: The hidden tax isn't just the token cost. It's that every token your system prompt consumes is a token the model can't use for reasoning about your actual problem. Long system prompts shrink the effective working memory. And the worst part — most of that instruction is redundant with what the base model already knows how to do. You don't need three paragraphs telling GPT-four-o to be helpful and thorough. It does that by default. Corn: The system prompt lever is powerful but the discipline is: write the shortest prompt that captures only what the base model cannot infer. For the car manual GPT, that's basically "you are referencing the owner's manual for this specific vehicle, identified by this VIN." Everything else is waste. Herman: Now let's talk about the knowledge retrieval, because this is where the magic happens and also where the magic quietly fails. When you upload that eight hundred forty-seven page PDF of your RAV4 manual, here's what actually occurs. The document gets split into chunks — typically a few hundred tokens each, with some overlap — and each chunk gets passed through text-embedding-three-large, which converts it into a vector. A list of numbers representing semantic meaning. Corn: The embedding model is the translator. It turns "what does this warning light mean" and "dashboard indicator illumination" into vectors that sit near each other in mathematical space, even though the words are different. Herman: When you ask a question, your query gets embedded too, and the system runs cosine similarity to find which chunks in your knowledge base are closest. Those chunks get injected into the context window alongside your question and the system prompt. The model never sees your whole PDF. It sees maybe the top five or ten most relevant snippets. Corn: Which is why retrieval quality is everything. If your manual has a section on the hybrid battery cooling system and your chunk boundary splits "cooling system requires" into one chunk and "inspection every thirty thousand miles" into the next, the model might retrieve the first chunk but miss the actual maintenance interval. Herman: You'd never know. The model will confidently tell you something plausible-sounding about hybrid battery maintenance because it doesn't know what it didn't retrieve. This is the failure mode that makes knowledge-base GPTs dangerous in ways people don't anticipate. It's not that the model is wrong about what's in your document. It's that the retrieval step is silent and invisible. You can't see what it missed. Corn: The car manual works here because the knowledge is structured. Section headers, diagnostic codes in consistent formats, clear fault-tree logic. The embedding model can map "P-zero-one-seven-one" to the right diagnostic page reliably. But try uploading two hundred restaurant reviews as unstructured text and asking "what's the best hummus in the German Colony" — the retrieval is going to grab fragments that mention hummus, fragments that mention the German Colony, and hope they overlap. Sometimes they will, sometimes you'll get a recommendation for a place that closed in twenty twenty-three. Herman: We've seen how the levers work technically. But here's the thing nobody talks about — the maintenance cost. Let's talk about knowledge debt. Corn: I like that phrase. It's the promise you make to yourself when you click publish — that you'll keep this thing current — and it accrues interest while you sleep. Herman: Every custom GPT you build is a commitment to update its knowledge base when the underlying facts change. The car manual is a free ride because Toyota already printed the manual. It's static. The diagnostic codes for a twenty-twenty-six RAV4 Hybrid are not going to mutate in August. But now imagine you build a company policy GPT for your startup. You upload the employee handbook, the benefits guide, the expense policy. Then HR updates the PTO accrual formula in September. Did you remember to re-upload? Did you even notice? Corn: The failure pattern here is worse than the base model being wrong. The base model, when it doesn't know something, will often hedge. A stale custom GPT will cite your own document back at you with total confidence, because it retrieved the chunk, it just retrieved the wrong version of the chunk. Herman: I saw this play out with a friend who built a React component reviewer GPT. Three thousand token system prompt, twelve uploaded blog posts about best practices. Six months later, three of those posts were recommending patterns that had been deprecated. The GPT was confidently telling him to use APIs that no longer existed. The base model, with a fresh system prompt and no stale documents, would have given him the current answer. Corn: The knowledge debt compounds, and the interest payment is that your custom GPT becomes actively worse than not having one. You'd have been better off just asking ChatGPT directly. Herman: This brings us to something that's shifting the calculus in real time — the context window substitution effect. GPT-four-o gives you a hundred twenty-eight thousand tokens to work with. Rumors of two hundred fifty-six thousand are circulating, and GPT-five whispers are pushing past a million. At a certain point, you can just paste the entire car manual into a single prompt. All eight hundred forty-seven pages. No chunking, no retrieval step, no silent misses. Corn: The tradeoff is cost. Longer prompts cost more per token, and you're paying that premium every session instead of amortizing the knowledge upload once. But the accuracy gain is real. No retrieval means no retrieval failures. Herman: And for a one-off diagnostic question, pasting the relevant chapter into a fresh chat might actually be faster and more reliable than opening your custom GPT and hoping the embedding model grabbed the right chunks. The custom GPT wins on convenience — it's always there, your spouse can use it without knowing what a PDF is — but it loses on flexibility. You can't easily say "actually, ignore the RAV4 manual, I'm driving a rental Camry today. Corn: Which leads to the discovery problem. There are roughly three million custom GPTs in the store right now. Somewhere in there is probably a beautifully crafted GPT for a twenty-twenty-six Toyota RAV4 Hybrid XSE. Good luck finding it. The store search is notoriously bad — it's basically a keyword match against titles and descriptions. The network effects everyone assumed would materialize never did. Herman: Because the premise of the store was that builders would create, users would discover, and quality would float to the top. But discovery requires search, and search requires metadata that most builders don't bother with. So you end up with three million GPTs and no way to find the one you need. It's faster to build your own than to find someone else's, which completely undermines the marketplace model. Corn: Here's where the dividing line actually lands, after all of this. Build a custom GPT when three things are true. One, the knowledge is bounded and static — it won't change, or it changes on a schedule you control. Two, you need consistent persona or role enforcement across many sessions — you're sharing this with non-technical users who won't write their own system prompts. Three, you want persistence — the tool is always there, configured the same way, no copy-paste required. Herman: Skip it when the task is one-off, when the knowledge is already in the model's training data, or when you're essentially just adding a system prompt for a single conversation. If you can get the same result by opening a fresh chat and typing two sentences of instruction, you don't need a custom GPT. You need a text file with your two favorite sentences. Corn: The car manual hits all three build criteria. Bounded knowledge, consistent role, shared with family. The Jerusalem restaurant recommender hits none of them. The React component reviewer hit two out of three until the blog posts went stale, and then it became a liability. Herman: That's the framework. It's not about how clever your system prompt is or how many documents you can upload. It's about whether the problem you're solving has edges. The car manual has edges. The manual ends. The VIN is specific. The recall database is finite. Most things people build custom GPTs for don't have edges — they're open-ended questions the base model already handles well, dressed up with a system prompt that adds cost without adding accuracy. Corn: Let's make this operational. Before you ever click "Create a GPT" again, I want you to ask yourself three questions. And Daniel, if you're listening, this one's for you specifically — you've already built forty-eight. Number forty-nine needs to earn its keep. Herman: Question one: is this knowledge bounded and slow-changing? Does it have edges? The car manual passes — eight hundred forty-seven pages, and page eight hundred forty-eight does not exist. A restaurant guide to Tel Aviv fails — menus rotate, chefs leave, the hot new place in Florentin is old news by the time you upload it. Corn: Question two: will I use this more than five times? Be honest with yourself. Not "could I imagine a scenario where this would be useful." Will you actually open it next Tuesday. Most custom GPTs are built for a problem that happens once. Herman: Question three: would a single well-crafted prompt in a fresh chat do the same job? If you can paste two paragraphs of instruction and get identical results, you don't need a custom GPT. You need a Notes app entry you can copy and paste. Corn: If you answer yes to all three, build it. If you answer no to any of them, skip it. That's the test. It's not complicated, but almost nobody applies it before they start uploading PDFs. Herman: Here's the piece that makes this framework actually stick — the maintenance cadence. Set a calendar reminder. Every ninety days, open each custom GPT you've built, look at the knowledge base, and ask: is anything in here obsolete? Because stale knowledge is worse than no knowledge. A blank-slate model will hedge when it's unsure. A custom GPT with outdated documents will cite your own material back at you with total confidence and be completely wrong. Corn: The React component reviewer you mentioned is the perfect cautionary tale. Three deprecated blog posts, six months of drift, and suddenly your "expert reviewer" is telling you to import functions that don't exist. You'd have been better off deleting it. Herman: That leads to what I think is the most practical approach for most people — the hybrid model. Use custom GPTs as starter templates. Build them for the system prompt and the actions — those are the things that are painful to reconfigure every session. But keep the knowledge minimal. Rely on the base model's training data for general facts. Only upload the specific, narrow, slow-changing reference material that the model cannot know. Corn: For the car GPT — upload the manual and the VIN codes, set a tight system prompt, maybe wire up the NHTSA recall action. That's it. Don't upload a generic "automotive repair encyclopedia" the model already ingested during training. You're just paying the token tax for knowledge it already has. Herman: Here's the open question I keep coming back to. What happens if GPT-five lands with a million-plus token context window? At that point, you could paste the entire car manual, the entire NHTSA recall database, and a small library of repair guides into a single prompt and still have room for the conversation. Corn: The knowledge advantage evaporates. The only thing a custom GPT would give you is persistence — not having to paste it every time. And at some point, OpenAI's going to realize they could just offer persistent system prompts in the base product and call it a day. Herman: They might not even call it anything. It could just become a feature — "save this prompt for future sessions" — and suddenly the GPT store is a vestigial organ. Three million custom GPTs, all of them technically still functional, but the reason they existed as a separate product just quietly disappeared. Corn: Which is why the framework we just laid out isn't just about deciding what to build today. It's about building things that actually earn their keep, regardless of what the platform does tomorrow. The car manual GPT works because the problem has edges. It doesn't matter whether those edges are enforced by a separate product or by a persistent prompt in the main chat. The specificity is what makes it valuable. Herman: That's the final thought I want to leave people with. The best custom GPT is the one you build for a problem that's just specific enough that the base model can't solve it, but just narrow enough that you can maintain it. The car manual is the platonic ideal. Everything else is a prototype. Corn: If your idea doesn't look like the car manual — bounded, static, specific, shared — you're probably building the restaurant recommender. And the restaurant recommender is a trap dressed as a project. Herman: Now: Hilbert's daily fun fact. Hilbert: In nineteen twelve, Danish explorer Knud Rasmussen reported observing a transient lunar phenomenon near the crater Aristarchus while crossing the Greenland ice cap. His notes described it as a reddish glow lasting several minutes, but the observation was dismissed for decades because no other expedition in the region confirmed it. The event was eventually reclassified as credible in nineteen sixty-nine when Apollo-era lunar monitoring programs identified similar transient events in the same crater. Corn: The moon was doing something weird over Greenland, and everyone just agreed to ignore it for fifty-seven years. Herman: That's very on-brand for nineteen twelve astronomy. Corn: This has been My Weird Prompts. Thanks to our producer Hilbert Flumingtop. If you enjoyed this episode, we'd love it if you left us a review wherever you listen — it helps other people find the show. I'm Corn. Herman: I'm Herman Poppleberry. See you next time.