Should you always default to an open source model for cost reasons?

No. Open source only becomes cheaper than a proprietary API past a sustained usage threshold, once hosting and maintenance costs are counted. Below that threshold, the proprietary API is usually still the cheaper option in total cost.

How long does a serious model comparison take?

Budget one to two weeks to build a set of real cases, test three to four candidate models against it, and price the cost at projected volume. That is the timeframe we apply before every production model choice.

Is a model that scores higher on general benchmarks always the better choice?

Not necessarily. The gap between the best proprietary and open models is now under two points on reference benchmarks, down from more than fifteen points in 2023. A high score on your real use cases matters more than a general leaderboard rank.

Can you switch models after the product has been built on top of one?

Yes, if portability was checked from the start: prompt format, no dependency on vendor-specific proprietary functions. That is why the portability clause is one of the five steps, before signing anything.

Should a single product use a different model for each AI feature?

Often yes. A support assistant and a document summarizer do not share the same latency requirements or cost profile. Testing each use case separately avoids overpaying for a powerful model on a simple task.

Choosing an AI Model for Your Business: The Criteria That Matter

The wrong question: hunting for "the best model"

Almost every client opens with the same question: what is the best AI model right now? It is a fair instinct, but it rarely leads to a good decision. Model leaderboards shift every six to eight weeks, and the model that tops a general benchmark is not necessarily the one that fits your use case, your budget, or your data constraints.

The question that actually matters is different: among the models available today, which one best satisfies the six criteria that decide whether an AI project survives contact with production, not just a demo? That is the question we work through with every client before writing a single line of code, and it almost always changes the initial answer, sometimes radically.

The six criteria that actually decide the choice

None of these criteria is decisive on its own. What settles the choice is their combination, weighted for your specific project. A support assistant billed per ticket does not carry the same hierarchy of criteria as an internal legal summarization engine.

Criterion	What actually matters	Common trap
Total cost	Price per token at real projected volume, not a one-off test rate	Comparing an API price to a hosting cost without counting maintenance
Latency and reliability	Response time under load, not in an isolated test, and whether an SLA exists	Ignoring latency variance from third-party inference providers
Data confidentiality	Where data transits and is stored, contractually	Assuming an open model guarantees confidentiality by itself
Customization	Real fine-tuning available, or only prompting and RAG	Fine-tuning a closed proprietary model with no portability clause
Vendor lock-in	How easily you could switch models if pricing or terms change	Building the entire product layer around a single proprietary API
Raw quality gap	Score on your real use cases, not a general leaderboard	Choosing by reputation instead of testing with your own data

The performance gap collapsed, the rest did not change

This was not always true. Two years ago, picking the strongest model on a leaderboard was close enough to picking the right model, because the performance gap between vendors was wide enough to dominate every other factor. That is no longer the case, which is precisely why the other criteria in this article deserve as much attention as the model's raw score.

On reference benchmarks, the gap between the best proprietary model and the best open model dropped from more than fifteen points in 2023 to under two points in early 2026. In practice, raw performance is no longer enough to decide, except for a small minority of extreme reasoning tasks, such as formal mathematical proofs or very long horizon planning. The other five criteria above now carry more weight than they did two years ago, and that is good news: they are the criteria that are easiest to measure objectively, unlike a vague impression of quality.

A useful cost reference point: an open model deployed on dedicated infrastructure typically costs five to ten times less than a proprietary model consumed through an API, at comparable volume, but that calculation reverses below a certain daily usage threshold. Below it, fixed hosting, monitoring and incident-handling costs outweigh the API bill. The crossover usually sits somewhere between a few hundred and a few thousand francs of monthly API spend, depending on how much operational load the team can absorb internally without hiring a dedicated person.

Confidentiality is a contract question, not only a technical one

A self-hosted open model guarantees nothing by construction. What protects data is where it is stored and the contract that governs it, not the "open source" label. If your clients or your industry require certain data to stay in Switzerland or the European Union, that constraint narrows the list of candidate vendors before you even start comparing performance scores. Three concrete questions to ask on the first call with a vendor: where is data physically stored, how long is it kept after the call, and is it used to retrain the model by default or only on explicit opt-in. Better to check these three points before signing a yearly contract than after.

Reading a public benchmark without being misled

Public model leaderboards are useful for a first impression, not for a final decision. Three biases show up repeatedly. First, contamination: some test questions may have leaked into a model's training data, artificially inflating its score without reflecting real generalization ability. Second, prompt selection: a vendor publishing its own score usually picks the prompt most favorable to its model, rarely a neutral one. Third, task shape: a model excellent at code generation can be mediocre at structured information extraction, and a general leaderboard averages both.

A common trap illustrates this: a model posts an impressive score on a general coding test, then turns out noticeably less reliable once confronted with an internal codebase that has its own conventions and history. The public leaderboard measured a generic skill, not the real task. That is exactly what testing on your own cases, described below, catches before you sign a contract rather than after three months in production.

The practical consequence: a public leaderboard is useful to shortlist three or four plausible candidates, never to pick the final model directly. That final choice should come from the test described below, on your own cases.

A five-step method to decide

Write the use case as one testable sentence. Not "a smarter chatbot" but "answer a billing question in under three seconds with the correct answer nine times out of ten."
Build a set of twenty to fifty real cases, pulled from actual conversations or documents, not cases invented for the test. A hand-crafted test set consistently produces more flattering scores than reality.
Test three to four candidate models on that same set, with the same prompt and the same evaluation method, to isolate the variable that matters. Mixing prompts across models invalidates the comparison.
Price the cost at your real projected volume, not the test volume, including hosting, monitoring and a margin for error if self-hosting is on the table.
Check the data portability clause before signing anything, not after six months of building on top of it. A proprietary prompt format or a vendor-specific function is enough to block a future migration.

This method takes one to two weeks. That is little, compared with the cost of redoing the integration six months later because the first choice was made on a vendor's reputation rather than on a test with your own data.

When this question does not deserve an answer yet

Before a product has found its market, spending two weeks comparing models is usually the wrong trade. A prototype meant to validate a hypothesis in fifteen days can run on a single generic proprietary model, picked by default, with no formal comparison. The risk at that stage is not picking the wrong model, it is losing two weeks on a decision the prototype might make irrelevant anyway.

The right moment to apply the five-step method above is when you move from demo to production, when the cost of a wrong choice is measured in months of delay rather than hours of testing. Similarly, if your usage stays under a few dozen requests a day, self-hosting an open model almost always costs more in team time than it saves in API billing. It only becomes a good calculation once usage is sustained and predictable, not sporadic.

Consider two contrasting situations. A two-person team validating whether anyone will pay for an AI-powered scheduling assistant should ship on a single default model in the first week, not spend that week benchmarking alternatives. A ten-person team migrating a support desk handling a few thousand tickets a month to an AI-assisted workflow is in the opposite situation: a wrong model choice there means weeks of rework and a support team that stops trusting the tool. The five-step method belongs to the second situation, not the first.

What the model choice does not solve

A good model choice does not make up for missing evaluation. Forward deployed engineering is precisely about testing a system on real data before the demo, whatever model sits underneath: the evaluation grid, the test set and the correction loop usually matter more than the initial choice between two models with similar scores. Two teams starting from the same model get very different production results depending on how rigorous their testing method is, not on which vendor they picked.

This point comes up often at the end of an engagement: a team that picked a solid model but never formalized its evaluation grid discovers regressions in production through customer complaints, rather than before shipping. A team that formalized that grid from the start catches the same regression in minutes, on its test set, before a single user ever sees it.

See our packages and pricing for support on this kind of decision, or check the frequently asked questions if you are just starting this work. You can also browse all articles on the blog for related topics, including the evaluation method to run before a demo.