Tesseract SA

Local AI in your business: a model on your premises, for whom?

Running an AI model on your own premises: when it makes sense, what hardware to plan, what quality to expect, and when an online service remains the better choice.

Small white 3D clay house on a grey base with a red padlock at its door, symbolising an AI model kept on your own premises.

What is local AI in a business?

Local AI means a language model that runs on hardware you control (a server in your office, a dedicated machine, sometimes a single workstation), so documents and prompts never leave your network. The model is an open-weights model (Mistral, Llama, Qwen, Gemma and others) that you download and run yourself, instead of calling a vendor’s API.

It is not the same as “sovereign” AI, which can be hosted by a provider in Switzerland or the EU. Local means on your hardware. Sovereign means under a jurisdiction you control. The two overlap without being identical.

The useful question for an SME owner is not “is it possible?” (it has been for a while) but “for which task, and under which conditions is it worth the effort?”. That is what this article settles.

Who local AI makes sense for

Three situations genuinely justify bringing the model in-house.

  • The data must not leave. Client files at a fiduciary firm, documents of an independent wealth manager, contracts at a broker: if your internal rule or mandate forbids sending them to a third party, a local model removes the question instead of managing it by contract.
  • Volume is high and steady. A local model has no per-use bill. Providers who calculate the break-even against cloud put it between 6 and 18 months depending on usage, but those simulations come from companies selling hardware or integration. Treat them as an order of magnitude to redo with your own volumes.
  • The task is narrow and repetitive. Sorting documents, extracting fields from an invoice, summarising an exchange, drafting a reply. For these, a mid-sized model is often enough.

If none of these describes you, the honest answer is further down: probably not yet.

Local, sovereign cloud or single workstation: three options not to confuse

OptionWhere the model runsMain strengthMain limit
On-site serverIn your premisesData never leaves the networkYou operate and update the machine
Sovereign cloudAt a host in Switzerland or EuropeNo hardware to buy, easy scalingData leaves your network, so the contract must be solid
Single workstationOn one employee’s computerFast start, low marginal costNo sharing, little power, hard to supervise

For a company of ten to fifty people, the real choice is often between a modest on-site server and a sovereign cloud. The single workstation is mostly for testing.

Which hardware for which model

Graphics memory (VRAM) is the limiting factor: a model must fit entirely in it to answer at a usable speed. If the model spills into regular memory, several technical guides estimate a slowdown of 10 to 50 times, and CPU-only inference drops to a few words per second.

Orders of magnitude commonly published for models quantised to 4 bits (check at purchase time, they change with every model generation):

Model sizeApproximate GPU memoryTypical use
7 to 8 billion parameters6 to 12 GBSorting, extraction, short summaries
About 24 billion16 to 24 GBDraft writing, questions over documents
About 70 billion40 to 48 GB and moreFiner reasoning, load shared across users

Hardware budgets quoted in guides range from a few hundred euros for a mini PC running a small model to several thousand for a GPU server. The range is wide because it mixes very different uses: do not keep a figure, keep the method. Start from the task, pick the smallest model that passes it, then size the machine.

What you lose in quality

A local model of 7 to 24 billion parameters does not match the best online models on long reasoning, complex instructions or less common languages. Nor does it know recent facts unless a search tool is plugged in beside it.

On a bounded task the gap shrinks a lot. On an open task (“analyse this file and recommend”) it stays visible. That is why a local model is mostly a good performer of precise tasks, and rarely a good replacement for a general assistant.

The method several guides suggest is the right one: take a model of 7 to 24 billion parameters and test it on 30 to 50 of your real documents before buying anything. Note the errors. If the error rate is acceptable with human validation, you have your answer. If not, you just saved yourself a server. Our article on the criteria for choosing a model details the comparison grid.

The hidden costs of an on-site server

The price of the graphics card is only the visible part. An honest budget counts five items: purchase, electricity, maintenance, backup and someone’s time for all of it.

  1. Updates. A new open model appears every few months. Someone decides when to migrate and checks nothing regresses on your cases.
  2. Monitoring. A server that goes down on Monday morning gets noticed. You need an alert, and a person who receives it.
  3. Security. A local model does not protect anything on its own: access control, logs, disk encryption and backups remain your job.
  4. Continuous evaluation. After a model or document update, results change. A fixed test set lets you see it.
  5. Availability. A cloud host aims for very high availability. Your server in a technical room has no redundant power and no on-call rota unless you build them.

Ollama, LM Studio or vLLM: which tool runs the model

Three tools come up in almost every guide. LM Studio offers a graphical interface: it suits trying a model on a workstation without touching the command line. Ollama downloads and serves a model with one command, with an API your tools can call: it is the usual choice for a first pilot. vLLM is built to serve many users in parallel with better throughput, at the price of a more technical setup.

Simple rule: Ollama to validate the idea, vLLM only if several dozen people use it at the same time. In most SMEs you will never reach that second case.

A worked example: sorting 400 invoices a month

Take a calculation assumption, not a measurement: a fiduciary firm receives 400 supplier invoices a month and spends 3 minutes extracting fields and filing each one. That is 20 hours a month. If a local model does the extraction and a person validates in 30 seconds, the time drops to about 3 hours 20 minutes, close to 17 hours recovered.

At a fully loaded cost of 80 Swiss francs an hour, that is about 1,300 francs a month. That figure, to be redone with your own volumes, is what you set against the cost of the machine and its operation. The gain is real, but it does not justify a big server: a small or mid-sized model is enough for this task. To put numbers on your own case, the express diagnostic is a starting point.

When local AI is not the right choice

Local AI is not the right choice when you have no strong confidentiality constraint, no steady volume, and nobody to run the machine. In that case a well-contracted online service costs less and works better.

  • Occasional use, a few requests a week: the subscription is cheaper than the purchase.
  • You need the best reasoning quality: the most advanced online models are still ahead.
  • No in-house skills or provider for monitoring: the saving on the bill is paid for in incidents.
  • A poorly understood regulatory constraint: a local model helps with data residency but does not exempt you from secure infrastructure and documentation. For the Swiss framework, see what to check before sending data to an LLM. This text is not legal advice.

The most common setup in practice is hybrid: a local model for sensitive documents, an online service for the rest.

What it looks like in practice: a five-step path

  1. Pick one narrow use case with a verifiable output (for example, extracting the amount, date and supplier from an invoice).
  2. Build a set of 30 to 50 real documents, anonymised if needed, with the expected answer for each.
  3. Test a model of 7 to 24 billion parameters on an existing machine or one rented by the week, with a common runtime such as Ollama.
  4. Measure error rate and speed, then decide where human validation goes.
  5. Size the machine only afterwards, from the smallest model that passes.

At Tesseract Studio this path sits within the method: a foundation, then modules, where local AI is an option of the foundation when confidentiality requires it.

Summary: the decision rule

Ask three questions in this order. Is the data forbidden to leave? Does the volume justify a fixed investment? Can someone operate the machine? Three yes: local AI is a good choice. Only a yes on confidentiality: consider a sovereign cloud. Three no: stay on an online service and revisit in a year, open models improve fast.

Frequently asked questions

What is local AI in a business?

It is an open-weights language model run on hardware the company controls, so documents and prompts never leave its network. It differs from a sovereign cloud, which is hosted by a provider.

Is a local model as good as ChatGPT or Claude?

Not on long reasoning and complex instructions, where the best online models keep a lead. On a narrow, well-defined task (extraction, sorting, summarising), a model of 7 to 24 billion parameters is often enough.

What hardware does a local LLM need?

It depends on model size. With 4-bit quantisation, commonly cited figures are about 6 to 12 GB of GPU memory for 7 to 8 billion parameters, 16 to 24 GB for about 24 billion, and 40 to 48 GB or more for about 70 billion.

How do I know if local AI is worth it for my company?

Test a mid-sized model on 30 to 50 of your real documents before buying hardware. If the error rate is acceptable with human validation, size the machine from that model.

Is local AI enough to be compliant with data protection rules?

No. It helps with data residency but does not exempt you from secure infrastructure and documentation. This article is not legal advice: have your case reviewed.

Sources
Read next