# Local AI Setup

> Turn on DocGraph's AI with a model on this computer — the built-in runtime, Apple's on-device model, Ollama, an AI agent or a provider — and manage the models you download.

Source: https://docs.docgraph.app/ai/setup/

DocGraph's AI runs **on your machine** by default. You don't need to install anything else: DocGraph ships a **built-in runtime** that runs a model file on your computer, and on a Mac with Apple Intelligence it can use **Apple's on-device model** with no download at all. [Ollama](https://ollama.com) still works, and is optional. AI is **off by default** — you opt in when you're ready, and enabling it is what starts the local indexing of your vault.

## Set up AI

Open the **Set up AI** wizard from either place:

- The chat panel's **Set up AI** button, shown while AI is off. (If you had already chosen a model and only turned AI off, it simply turns AI back on.)
- **Settings → AI → Models & providers → Set up AI…** (under **Not sure which to use?**). The wizard can be re-entered here at any time to switch paths.

The master switch is the **AI features** toggle at the top of **Settings → AI → Models & providers** (off by default). Clicking **Use** on Apple's model or a downloaded model turns it on for you.

The wizard has four steps, listed on its left: **Your computer**, **Choose**, **Set up** and **Try it**. Nothing is downloaded until you click a download button.

### 1. Your computer

The wizard reads your chip, memory, free disk space and system version, and looks for AI that is already on the machine: Apple's model, Ollama and its models, models DocGraph has downloaded, and AI agents such as Claude Code. A sentence sums up what fits, for example *"With 16 GB of memory, this computer can run a balanced AI model comfortably."* If the memory figure is wrong, edit the **Memory** number and the suggestions update. The same hardware check is what [MMF](https://modelfit.docgraph.app), DocGraph's free tool for finding the best model for a computer, does in full.

### 2. Choose how to turn AI on

The ways are grouped by where the AI runs — **On this computer** and **Through a service you choose** — each tile with a chip that says what this computer has (for example **Suggested: Balanced**, **3 models** or **Not running**). The wizard remembers your pick for this session until you finish setup.

| Tile | What it does |
|---|---|
| **Use Apple's AI** | Apple's on-device model; nothing to download. Shown only on a Mac that has it or is getting it (see [below](#apples-on-device-model)). |
| **Download a model** | DocGraph downloads a model once and runs it with its built-in runtime. Nothing else to install. |
| **Use Ollama** | Pick one of the models your Ollama already has (each marked **Can edit notes** or **Questions only**), or start or install Ollama. |
| **Use an AI agent** | Lists command-line agents found on this computer (Claude Code, Codex and others). Turn one on in **Settings → Connections → Agents & MCP**, then pick it in the chat's model menu. |
| **Use an AI provider** | **Add an API key** for OpenAI, Anthropic and similar. Your questions go to that company — see [Cloud AI providers](https://docs.docgraph.app/ai/cloud-providers/). |

**Download a model** offers the sizes [MMF](https://modelfit.docgraph.app) picks for this computer — **Small** (fast, good for short questions), **Balanced** (recommended for most people) and **Large** (best answers, needs a powerful computer) — plus a **Best fit** card for the model DocGraph has tested on computers like yours, when MMF didn't pick it itself. One is tagged **Suggested for this computer**; a model that would not fit on the disk is greyed out, and one already downloaded reads **Use it**. Each card shows a rough speed and says whether the model **Can search, open and edit your notes** ([Agent Mode](https://docs.docgraph.app/ai/chat/#let-it-take-actions)) or **Answers questions only**. If MMF has no suggestion, two fixed cards appear instead (**Small model** and **Larger model**); if nothing fits comfortably, the wizard says so and points you to a provider.

The fold **Advanced: a model from Hugging Face or a link** takes either a Hugging Face file (`org/repo/file.gguf`) or an `https://` link to a `.gguf` file. DocGraph does not check models from elsewhere — see [Model downloads](https://docs.docgraph.app/privacy/ai/#model-downloads).

### 3. Set up

A chosen model downloads with a progress bar and time estimate. You can close the window and the download keeps going; an interrupted one resumes where it stopped, and DocGraph checks the file when it arrives. **Cancel** stops it.

If your vault has scanned PDFs, the wizard asks once how to read them: **Fast** (built into the Mac, no download) or **Best quality** (about 1.7 GB, downloaded in the background), then **Ask me per file** or **Read all scans automatically**. **Not now** skips it; the choice lives in **Settings → AI → Scanned documents**.

If you haven't chosen a [search language](#search-language) yet, the wizard asks next.

### 4. Try it

DocGraph sends a first message ("Are you ready?") to the model and shows the reply. **It works — continue** finishes setup; if the model doesn't answer, the error stays on screen with **Try again**, **Choose something else** and **Use an AI provider instead**. The last screen reads **AI is on.** with a few questions to try and an **Open chat** button.

Everything the wizard chooses can be changed later in **Settings → AI**.

## Apple's on-device model

On a Mac with Apple Silicon running macOS 26 or later, Apple ships a small language model. DocGraph can use it with nothing to download and nothing sent off the machine. The wizard and **Settings → AI → Models & providers → Apple's on-device model** show one of these states:

| State | What to do |
|---|---|
| **Ready** | Click **Use** (**Use it** in the wizard). |
| **Apple Intelligence is off** | **Turn on…** (**Open System Settings** in the wizard), switch Apple Intelligence on, then come back. |
| **macOS is still downloading it** | Wait, then **Check again**. |

If the Mac can't run it, the section doesn't appear.

Its limits are real. Its window is about **4,000 tokens**, which is enough to answer a question from the passages DocGraph finds in your notes and not enough for the tool list Agent Mode needs. So with Apple's model selected:

- **Agent Mode is unavailable.** The **Agent** button in the chat is greyed out and explains that Apple's model can't create or edit notes. To let the AI act, pick a downloaded model.
- Chat answers questions about your notes from retrieved passages, without the tool loop. Long messages are trimmed to fit, and replies are kept short.
- It reads only some languages. Settings lists them. If an answer used notes in a language it doesn't read, the chat adds a warning under the answer.

## Manage models on this computer

**Settings → AI → Models & providers** holds everything about models that live on this computer. Its **Built-in AI** section covers the bundled runtime. The runtime status reads **Ready · starts on your first question**, **In memory**, or **Stopped after repeated errors** (with **Try again**). The runtime starts on the first question that needs it, never at launch, and listens only on this computer.

**Downloaded models** lists every model on this computer with its size and the total disk space used. A model tagged **✓ checked by MMF** is in MMF's curated list; one tagged **your file** is a file you added.

### Switch the model chat uses

Click **Use** on a downloaded model; the one in use reads **In use**. You can also pick any model in the chat's [model picker](#picking-a-model-in-chat).

### Download another model

Click **Download a model…**. It opens the setup wizard at the model choice, with the same size cards and the **Advanced: a model from Hugging Face or a link** fold described [above](#2-choose-how-to-turn-ai-on). Downloads resume after an interruption and are checked when they arrive. A download DocGraph needs on its own, such as a model missing when the app opens, first asks in a dialog that names its size and where it comes from (**Download** or **Not now**); the answer is remembered per model.

### Add a model file you already have

Click **Add a model file you have…** and pick a `.gguf` file on disk. It is listed as **your file**. **Remove** makes DocGraph forget it and leaves the file where it is.

### Free memory: unload the model

While a model is loaded, the status reads **In memory** with an **Unload** button. Click it to free the memory; the next question loads the model again.

### Check for new models

Click **Check for new models** to fetch MMF's latest list of models that fit your computer. It reports *up to date* or that it couldn't reach the list and is using the one built into DocGraph. Nothing about you is sent.

### Delete a model

Click **Delete** on a downloaded model and confirm. This frees its disk space; you can download it again later.

### Use Ollama (optional)

Ollama is not required. If you already use it, the **Ollama** section below Built-in AI shows whether it is running (**Connected · _n_ models**, **Running · no models pulled** or **Not running**) and lists its models:

- **Pull a model**: type a name (for example `gemma4:e2b-mlx`, the Ollama model we recommend on a Mac) and click **Pull**. Progress shows with **Cancel**.
- **unload** appears on a model that is loaded in memory.
- Hover a row for its trash icon to delete that model from Ollama, after you confirm.
- **MLX?** appears on a model running on the default llama.cpp engine. Click it to ask Ollama's registry (`registry.ollama.ai`) whether an Apple MLX build exists (this sends only the model name, and only when you click). If one does, its name appears; click it to put it in the pull box. Otherwise the row reads **no MLX build**. A model that already runs on MLX is tagged **MLX**.

DocGraph talks to Ollama at `http://localhost:11434`; change it under **Settings → AI → Chat & index → Ollama base URL**. An Ollama on another machine is refused unless you turn on **Allow remote Ollama** there and confirm.

## Picking a model in chat

The model picker at the top of the chat panel lists every model you can use: your **Pinned** ones (click the star on a row), the last five you used, then one section per provider — local first, then any [cloud provider](https://docs.docgraph.app/ai/cloud-providers/) you have set up, then external agents. Type to search, or filter with the chips (**Local**, **Cloud**, a provider, **Agents**, and **Tools** / **Vision** / **Reasoning** / **Free**). Each row shows the model's size, context window and an **MLX** tag where it applies; hover the picker for its quantization, memory use and the speed DocGraph measured on this computer. When both an MLX and a llama.cpp build of the same model are installed, **⇄ Mac-native version** (or **⇄ Standard version**) under the chat's header switches to the other build.

Only a local pick (built-in, Apple or Ollama) becomes your default chat model; a cloud model or an agent is used by that conversation only. **Settings → AI → Chat & index** holds the rest: the default chat model, a separate **Utility model** for small background jobs, the **Context window**, **Keep the model in memory** (30 minutes by default) and **Temperature**.

DocGraph learns how each model behaves on your computer — how fast it answers, whether it can handle the small helper jobs (rewriting follow-up questions, expanding searches), whether warming it up in advance pays off. A helper that proves too slow is skipped for 30 days. **Settings → AI → Chat & index → Last answer** shows what your most recent wait was made of (loading the model, searching your notes, or writing) and the measured speed per model.

## Models we've measured

We test models on our own Mac (Apple M4 Pro, 24 GB of memory) with a fixed set of questions about a sample vault. These numbers come from that one machine. Yours will differ with its chip and memory, and the wizard's suggestion is based on your computer, not on this table.

| If you want… | Model | On our test Mac |
|---|---|---|
| Speed, with Agent Mode | `gemma4:e2b-mlx` (Ollama) | Answers in about 3 seconds, about 6 in Agent Mode. Got all but one or two test questions right. Uses about 8 GB of memory. |
| The most accurate answers | `qwen3.5:4b-mlx` (Ollama) | Answers in about 5–9 seconds. Got every test question right. Uses about 6.5 GB. |
| Less memory, no Ollama needed | Ministral 3 3B Reasoning (**Download a model**; the built-in default) | Answers in about 6 seconds. Got all but one test question right. Uses about 5 GB. |
| Nothing to download | Apple's on-device model | Answers in about 4–5 seconds. No Agent Mode (see [above](#apples-on-device-model)). |

Bigger is not always better here. A 12-billion-parameter model gave equally good answers but took over a minute in Agent Mode and used 14 GB of memory. The two Ollama models above run on Apple's MLX engine, which is why they are fast on a Mac (see **MLX** in [Use Ollama](#use-ollama-optional)).

## Search language

The AI finds your notes with a small search helper that runs on your computer — never with the chat model. You choose which one, and **the AI can't search your notes until you do**:

| Choice | Best for | Download |
|---|---|---|
| **English only** | Notes written in English | about 75 MB |
| **Many languages** | Arabic, Hebrew, Chinese, French, Spanish… and English. Finds notes across languages, such as an English question about a note written in Arabic. | about 270 MB |

Nothing is picked for you. Until you choose, nothing is downloaded or indexed for the AI; [⌘K keyword search](https://docs.docgraph.app/getting-around/overview/#search) works either way because it needs no model. The setup wizard asks; with AI on, DocGraph also asks when it starts, and the chat panel asks above its message box, which stays disabled until you choose.

- **One choice for every vault.** Change it in **Settings → AI → Chat & index → Search language**. Each vault is re-indexed with the new helper — the open one right away, the others the next time you open them.
- **Choosing is the download approval.** You have just read the size, so DocGraph does not ask a second time.
- **Each helper comes with a reranker**, a second small model that reads every passage search found together with your question and puts the best first. Its size is included above. Turn it off with **Settings → AI → Chat & index → Rerank results**.
- With **English**, the reranker can also tell when nothing in your notes is about the question; chat then [says so](https://docs.docgraph.app/ai/chat/#when-nothing-matches) instead of answering from unrelated notes. The many-languages reranker only reorders.

## Requirements

DocGraph runs on Macs with Apple Silicon. Local models need enough memory and disk space. The wizard measures your memory and disk and only offers sizes that fit; if none do, use an AI provider. Apple's on-device model needs macOS 26 on Apple Silicon.

## Privacy

- Your notes are embedded and indexed **locally**.
- Unless you pick a cloud model or an agent, chat requests go to a model on this computer: the built-in runtime (which listens only on `127.0.0.1`, protected by a random key), Apple's model, or your local Ollama. DocGraph refuses a non-local Ollama unless you explicitly turn on **Allow remote Ollama**.
- Turning **AI features** off stops the AI's indexing (embeddings) and all model use; nothing is sent to any AI. [⌘K keyword search](https://docs.docgraph.app/getting-around/overview/#search) keeps working.

A few features can reach beyond your machine. All are opt-in and clearly marked:

- **[Cloud AI providers](https://docs.docgraph.app/ai/cloud-providers/)** (OpenAI, Anthropic, OpenRouter, a custom endpoint) are used only once you save a key, allow the provider and pick one of its models.
- **[Web search](https://docs.docgraph.app/ai/web-search/)** is off by default. With it on, the assistant shows you each search before it is sent, and results are treated as untrusted input. **Allow in this chat** lets you skip that card for the rest of one conversation.
- **External CLI agents** (Claude Code, Codex CLI, Gemini CLI, OpenCode) hand a turn to a tool that runs with full vault access and bills your own account with that provider. DocGraph checks the prompt it hands over for secrets first; what the agent sends after that is not inspected, and its changes are not covered by [chat's per-change confirmations](https://docs.docgraph.app/ai/chat/). Enabling one requires ticking an acknowledgement.