Your business can fit in a smaller model

AI has stopped being something only big companies do. Using it seriously once meant hiring your own researchers and spending millions, but today a hotel chain uses AI to handle bookings, a clinic to transcribe appointments, and an insurance broker to read claims. The small and specialized businesses that never had an engineer dedicated to AI, and never expected to need one, now use it every day to run their businesses better.

What made this possible are companies like OpenAI and Anthropic, who built AI so capable and so easy to plug in that adding it to a business went from a research project to a line on the monthly bill. For most teams that's a remarkable deal, and the right place to start.

But it's worth understanding what you're buying. These are extremely large models that can do a thousand things your business will never ask for, and that excess shows up in the price and in the time your customers wait for an answer, since a model built to do everything is far bigger than the one your task needs. Newer and more powerful models usually cost more, while the extra quality isn't guaranteed for your case, as we saw when we measured seven frontier models on the same real task and the supposed best one delivered its quality gain at four and a half times the cost. Competition helps, but prices for some models remain high, and for a growing number of companies the AI bill has quietly become a structural cost.

That's why more and more companies are opening the door to building their own AI, which means training a specialized model of their own on what the business already generates, the past conversations, the documents, the way they talk to their customers. The result can be dramatically cheaper and faster to respond, while keeping the quality of the big models or even improving on it, and the reason is simple: the model isn't good at everything, it's good at your business, your style, your cases.

If yours is one of the businesses starting to ask that question, this article should help you get your bearings. We'll walk through the types of models you can start from, what each is built for and their tradeoffs in plain terms, and share what has worked when we've made this same choice for the businesses we work with.

The types of models you can start from

The models you've been using so far, GPT, Claude, Gemini, are what the industry calls closed-weight models. The weights are the model itself, billions of numbers learned during training that encode everything it knows. Closed means you never see them, so the model lives on the provider's computers and you pay for every answer sent over the internet. It's a service, like electricity.

Open-weight models are the alternative that makes everything in this article possible. Labs like Meta, Google, Alibaba, Mistral or DeepSeek publish the actual model files for anyone to download, and the best of them sit surprisingly close behind the frontier. Owning the files changes what you're allowed to do. You can run the model on your own machines so customer data never leaves your systems, stop paying per request, and keep training it on your own conversations and documents until it becomes a specialist in your business.

Once you step into that world, choosing a model is no longer picking a name from a provider's list. Several decisions take its place: what type of model, how big, which version of it, under what terms it's published. The type comes first and shapes the rest, because models differ less in how good they are than in what they're built to do, and picking the wrong type means paying for machinery your task will never use.

One family of models is worth a brief mention before we go on. Encoder models, the descendants of BERT, don't write at all; they read text and produce a decision, which keeps them the cheap way to sort and route text at scale. What businesses usually need from AI, though, is generated text: conversations, replies, and summaries, so from here on we'll stay with the models that write, the ones behind ChatGPT and Claude, which the industry calls decoders.

Behind every decoder sits a series of design choices, more than we could cover in one article, so we'll focus on the two that weigh most in practice, and the good news is that both are close to a yes-or-no question, how the model spends its parameters and what long inputs cost it. Take them in order and most model cards on the market become legible.

How it spends its parameters: dense or mixture-of-experts

Every decoder faces the same budget question: when it processes a token, how much of the model does the work? The two answers define the two camps.

Dense models: in these models, all the parameters work on every token. That makes them the most data-efficient way to specialize, because every example teaches the entire model, and a few thousand good conversations can meaningfully specialize a small one. It also makes them the mature option, since dense models have been the industry's main option for years: the tooling is polished enough that adapting an 8B (eight billion parameters) fits on one affordable GPU, serving runs anywhere from a mid-range GPU down to a laptop, and optimizations and quantization support (compressing a model so it runs smaller and cheaper, at little quality cost) arrive here first. The price of all this is speed. The time a model takes to produce each word is roughly proportional to the parameters that activate to produce it, and in a dense model that's all of them, so latency grows directly with size. The saving grace is that specialization lets you stay small enough to generate tokens at an acceptable pace even on modest GPUs. This is where the classic names live: the small Qwen3.5 sizes (0.8B to 9B, Qwen3.5-4B among them), Llama 3.2 (1B and 3B), Gemma 4 (E2B and E4B), and Qwen3.8-27B at the capable end.

Mixture-of-experts models: in these models, only a fraction of the parameters work on each token, so the model knows as much as a huge one but thinks at the cost of a small one. Inside, the parameters are grouped into many small specialists, the experts, and each token is handled by the few that suit it. The "A" number on the label tells the story. Qwen3.6-35B-A3B, for example, stores thirty-five billion parameters and uses three billion per token, so a model that knows as much as a 35B answers with the speed of a 3B. That speed comes with tradeoffs. In training, each example only teaches the experts that handled it, so your data spreads across dozens of specialists instead of concentrating on one model, and a mixture-of-experts needs noticeably more data to learn your business than a comparable dense one. In serving, answering at the speed of a 3B doesn't mean paying for a 3B, because every expert has to be loaded in GPU memory whether it's working or not, so the hardware is sized by everything the model knows, not by how fast it thinks. The names here include Qwen3.6-35B-A3B, Gemma 4 26B-A4B, gpt-oss (20B and 120B), and at the frontier end Inkling (975B stored, 41B active).

Choosing between them comes down to your data and your latency target. Dense models learn more from fewer examples and ask the least to get started, so unless your product demands instant replies, a small dense model is the cheapest way into ownership. If a small dense model can't reach the quality you need, or its replies feel slow to your customers, mixture-of-experts is the natural step up, provided your dataset is large enough to feed its experts.

The same request arrives, token by token

Dense

every token wakes every weight

I→need→two→nights→

27B of 27B parameters active per token

Mixture-of-experts

each token wakes a few weights

I→need→two→nights→

3B of 35B parameters active per token

The cost of context: standard or hybrid attention

The second choice is independent of the first, and it only shows itself when inputs get long or many customers talk to the model at the same time. Attention is the mechanism a model uses to relate each word to everything that came before it, and the more text in play, the more computation and memory it needs. The two designs on offer differ in how fast those needs grow.

Standard attention: in this design, every token can consult every other token and nothing gets lost in between, which makes it the most complete form of reading and the most battle-tested, since the whole ecosystem of training methods, serving software and optimizations was built assuming it. The trouble starts when conversations get long, because the price of reading climbs steeply with the number of tokens to read (double the tokens and the computation roughly quadruples), and every open conversation pays its own way, keeping a working memory that grows with every token exchanged, so the same GPU that comfortably serves fifty short chats can choke on ten long ones, or even on a single conversation that piles up enough tokens. It used to be worse, since that memory originally climbed as steeply as the computation (that is, quadratically), but modern serving software trades a little extra computation to keep its growth linear, in step with the tokens. Models that read this way include Qwen3-4B, Llama-3.2-3B, and gpt-oss-120B.

Hybrid attention: this design changes the mechanism itself, combining a lighter, faster form of attention with a few layers of the standard kind so that those needs grow gently with length instead of steeply. The working memory grows several times more slowly as the model reads, which lets it take in enormous documents, up to a million tokens in this line, and because each conversation occupies so much less memory, one GPU can hold many more of them at once, lowering your cost per request as traffic grows. The trade is some accuracy loss in the reading itself (up to 6% in measured tests). The lighter attention doesn't hold on to every detail; it carries a compressed memory of the text forward, so an exact figure or the wording of a single clause from two hundred pages back can fade in a way standard attention never allows. The few standard layers mixed into the design recover much of that, but for exact recall, a fully standard model remains the reference. There's a training catch too. To get the model good at your long documents, you have to train it on long documents, and those are expensive to process and scarce; most teams have thousands of short conversations and only hundreds of long files. Models that read this way include Qwen3-Next-80B-A3B (the reference of this line), the small Qwen3.5 models like Qwen3.5-4B, which carry the same design down to sizes you can fine-tune cheaply, Qwen3.8-27B, the dense model from the previous section, and IBM Granite 4.0, which mixes attention with a different lightweight mechanism to the same end.

Choosing between them comes down to the length of your inputs. When they're short, the steep part of the curve never arrives, either design serves you equally well, and the choice can simply ride along with the model you pick. Hybrid earns its accuracy trade in two situations. One is conversations or documents that stretch into tens or hundreds of thousands of tokens, where splitting them would cost you accuracy. The other is many long conversations open at once, where each one takes much less memory, so the same GPU can serve many more customers.

One conversation's working memory as it grows

attention memory per conversation, bf16: Qwen3-4B ~144 KB/token · Qwen3.5-4B ~32 KB/token

The piece that matters more than the model

We wanted this article to be a first map for businesses starting to explore the idea of owning their AI, and what we've covered, the type of model, is only the first axis of that choice. A real decision would also weigh how big the model should be, which of its versions to start from, the terms it's published under, the languages it covers and the hardware it needs. But there's one piece we can't leave out because it will shape the result more than any axis we could add: the data. What transforms a general model into yours is the training, and the training is only as good as the examples you feed it. A well-chosen model trained on messy, unrepresentative examples becomes a messy, unrepresentative specialist; a modest model trained on clean ones can quietly outperform it.

That's why most of the real work of building your own AI doesn't look like reading model cards. It looks like working with your own information, gathering the conversations and documents your business already produces, filtering out the noise, the duplicates and the cases you'd never want imitated, and curating what remains into a dataset that reflects how your business actually operates, the awkward cases included. Choosing the model is an afternoon of reading; building the dataset that makes it yours is where the craft lives, and it deserves an article of its own.

Conclusions

The two questions we've walked through, how the model spends its parameters and what long inputs cost it, cover most of the choice, and your task's requirements answer each one. Many businesses end up with more than one answer under the same roof, an encoder routing requests next to a small dense model holding conversations, and that's a sign the process worked; the task decided, not the catalog. Whatever combination you land on, the data you train it with will shape the result more than the choice itself.

Nothing about your product needs to change after reading this. GPT or Claude will keep answering your customers tomorrow, and doing it well. What changes is quieter: the next time the AI line appears on the invoice, you'll read it with more context, and you'll know there's an alternative if you ever need one.