The phrase "private LLM" is used loosely, and vendors often mean different things by it. This guide is about the concept, not the product: what an LLM is, what makes one private, where the open-weight models people mention fit in, and how a model comes to know about your business. By the end you should be able to tell whether the phrase describes something you need.
What is an LLM, in plain terms?
An LLM, or large language model, is software trained on a very large amount of text to predict what text should come next. That one skill is enough to draft, summarize, classify, translate and answer questions in ordinary language. It does not look facts up the way a database does. It produces the most plausible continuation, which is why it can sound confident and still be wrong.
Two parts are worth separating. The model is the trained file, often called its weights. The service around it is everything that receives your text, runs the model and returns an answer. Most confusion about "private" comes from mixing those two up.
What does "private" actually mean?
There is no standard definition, so ask what is meant. In practice it usually describes some mix of these choices:
- Where the model runs: on your own hardware, in a cloud account you control, or on a vendor's shared service
- Where your data goes: whether prompts and documents leave your boundary, and who can see them
- What is kept: whether prompts and answers are stored, for how long, and who may use them
- Who can use it: whether access is limited to named people and logged
So a private LLM is best read as a deployment and data-handling choice: a model set up so that your information is designed to stay inside a boundary you have defined, unless you approve otherwise. It is not a special kind of model. The same model can be run privately or publicly. Where it runs is a separate question, covered in local AI vs cloud AI.
What is an open-weight model?
An open-weight model is one whose trained weights are published, so you can download them and run them on your own machine or in your own cloud account. That is what makes self-hosting possible. A hosted model, by contrast, can usually be reached only through the vendor's service.
Open-weight is not the same as open source. The weights may be available while the training data and training code are not, and the license attached to the weights can restrict how you use them. Licenses differ from model to model and from version to version. Read the license for the exact model and version you plan to use, keep a note of which one you deployed, and have someone qualified review it if the use is commercial or sensitive. Do not assume that a model you can download is a model you may use for any purpose.
Choosing between a hosted vendor and an open-weight model is a separate decision, covered in OpenAI API or an open-weight model.
How does a model learn about your business?
Out of the box, a model knows only what was in its training text. It has never seen your contracts, your price list or last week's email. There are two common ways to close that gap, and they do different jobs.
| Question | Retrieval over your documents | Fine-tuning |
|---|---|---|
| What does it do? | Finds relevant passages in your documents and gives them to the model with the question | Adjusts the model itself with extra examples, so it changes how it behaves |
| Best for | Answering from facts that change: policies, product notes, past correspondence | Teaching a consistent style, format or narrow task |
| Updating it | Change the document and the answer can change | Needs a new training run |
| Showing the source | Can point to the passage it used | Generally cannot say where an answer came from |
| Data exposure | Your documents stay in a store you control; only the retrieved passages reach the model | Your examples become part of a new set of weights, which then need protecting too |
| Effort | Needs clean, organized documents and sensible permissions | Needs good example data, testing and repeat work for each new model version |
Read the table as a guide to order. Most businesses should try retrieval first. The questions they have are usually about their own documents, those documents change, and being able to check the source matters. Fine-tuning is worth considering later, when you have a stable task, plenty of good examples and a clear reason a better prompt will not do.
Is a private LLM automatically more secure?
No. Private describes where your data goes, not how well it is protected. A model on your own hardware is protected only as well as that hardware, its updates, its access controls and its backups. A well-run hosted service with clear contract terms can be a stronger setup than a neglected machine in a cupboard.
A private LLM also does not make the answers correct. It can still invent details, and a retrieval system can pass along a document a particular user should not see unless permissions are built in. Hostile text hidden in an email or file can still try to steer it. Private reduces who receives your data. It does not remove the need for access control, logging, testing and review. The NIST AI Risk Management Framework is a voluntary, neutral reference for thinking those risks through.
Key terms in plain English
- LLM: a large language model, software trained on text to produce language
- Weights: the trained numbers that make up a model; the file you download or the thing a vendor keeps
- Open-weight: a model whose weights are published so you can run it yourself, subject to its license
- Hosted model: a model you reach through a vendor's service rather than running it yourself
- Prompt: the text you send to a model, including any instructions and documents
- Retrieval: finding relevant passages in your own documents and supplying them with the question
- Fine-tuning: further training of a model on your examples to change how it behaves
- Hallucination: a confident answer that is wrong or invented
- Self-hosting: running a model on infrastructure you operate or control
Who actually needs a private LLM?
Many businesses do not, and that is a perfectly good answer. The cases where it is worth a closer look tend to share a few features:
- The work involves information you are required, or have promised, to keep inside a defined boundary
- A contract, client or regulator asks where data is processed and who can see it
- You need to know exactly which model version produced a result, and keep it fixed
- The documents are the value, and you want answers drawn from them with the source shown
If none of that applies, a well-chosen hosted service with sensible settings and terms you have read may serve you better, and be less work to keep running. If several apply, the next step is to map your data, decide what must stay inside, and match the setup to that. Our private AI page describes how we scope that work.
The same restraint applies to our own private email assistant, an internal project that uses one local model for one job: drafting replies that a person reviews before anything is sent.
What should you do next?
List the data your first use case would touch, and mark what must not leave your boundary. Then ask three questions of any option you consider: where does it run, what is stored and for how long, and who can reach it. If you would like a second opinion, read about our private AI for business work and get in touch from there.