Local AI Setup
DocGraph’s AI runs on your machine by default. You don’t need to install anything else: DocGraph ships a built-in runtime that runs a model file on your computer, and on a Mac with Apple Intelligence it can use Apple’s on-device model with no download at all. Ollama still works, and is optional. AI is off by default — you opt in when you’re ready, and enabling it is what starts the local indexing of your vault.
Set up AI
Section titled “Set up AI”Open the Set up AI wizard from either place:
- The chat panel’s Set up AI button, shown while AI is off. (If you had already chosen a model and only turned AI off, it simply turns AI back on.)
- Settings → AI → Models & providers → Set up AI… (under Not sure which to use?). The wizard can be re-entered here at any time to switch paths.
The master switch is the AI features toggle at the top of Settings → AI → Models & providers (off by default). Clicking Use on Apple’s model or a downloaded model turns it on for you.
The wizard has four steps, listed on its left: Your computer, Choose, Set up and Try it. Nothing is downloaded until you click a download button.
1. Your computer
Section titled “1. Your computer”The wizard reads your chip, memory, free disk space and system version, and looks for AI that is already on the machine: Apple’s model, Ollama and its models, models DocGraph has downloaded, and AI agents such as Claude Code. A sentence sums up what fits, for example “With 16 GB of memory, this computer can run a balanced AI model comfortably.” If the memory figure is wrong, edit the Memory number and the suggestions update. The same hardware check is what MMF, DocGraph’s free tool for finding the best model for a computer, does in full.
2. Choose how to turn AI on
Section titled “2. Choose how to turn AI on”The ways are grouped by where the AI runs — On this computer and Through a service you choose — each tile with a chip that says what this computer has (for example Suggested: Balanced, 3 models or Not running). The wizard remembers your pick for this session until you finish setup.
| Tile | What it does |
|---|---|
| Use Apple’s AI | Apple’s on-device model; nothing to download. Shown only on a Mac that has it or is getting it (see below). |
| Download a model | DocGraph downloads a model once and runs it with its built-in runtime. Nothing else to install. |
| Use Ollama | Pick one of the models your Ollama already has (each marked Can edit notes or Questions only), or start or install Ollama. |
| Use an AI agent | Lists command-line agents found on this computer (Claude Code, Codex and others). Turn one on in Settings → Connections → Agents & MCP, then pick it in the chat’s model menu. |
| Use an AI provider | Add an API key for OpenAI, Anthropic and similar. Your questions go to that company — see Cloud AI providers. |
Download a model offers the sizes MMF picks for this computer — Small (fast, good for short questions), Balanced (recommended for most people) and Large (best answers, needs a powerful computer) — plus a Best fit card for the model DocGraph has tested on computers like yours, when MMF didn’t pick it itself. One is tagged Suggested for this computer; a model that would not fit on the disk is greyed out, and one already downloaded reads Use it. Each card shows a rough speed and says whether the model Can search, open and edit your notes (Agent Mode) or Answers questions only. If MMF has no suggestion, two fixed cards appear instead (Small model and Larger model); if nothing fits comfortably, the wizard says so and points you to a provider.
The fold Advanced: a model from Hugging Face or a link takes either a Hugging Face file (org/repo/file.gguf) or an https:// link to a .gguf file. DocGraph does not check models from elsewhere — see Model downloads.
3. Set up
Section titled “3. Set up”A chosen model downloads with a progress bar and time estimate. You can close the window and the download keeps going; an interrupted one resumes where it stopped, and DocGraph checks the file when it arrives. Cancel stops it.
If your vault has scanned PDFs, the wizard asks once how to read them: Fast (built into the Mac, no download) or Best quality (about 1.7 GB, downloaded in the background), then Ask me per file or Read all scans automatically. Not now skips it; the choice lives in Settings → AI → Scanned documents.
If you haven’t chosen a search language yet, the wizard asks next.
4. Try it
Section titled “4. Try it”DocGraph sends a first message (“Are you ready?”) to the model and shows the reply. It works — continue finishes setup; if the model doesn’t answer, the error stays on screen with Try again, Choose something else and Use an AI provider instead. The last screen reads AI is on. with a few questions to try and an Open chat button.
Everything the wizard chooses can be changed later in Settings → AI.
Apple’s on-device model
Section titled “Apple’s on-device model”On a Mac with Apple Silicon running macOS 26 or later, Apple ships a small language model. DocGraph can use it with nothing to download and nothing sent off the machine. The wizard and Settings → AI → Models & providers → Apple’s on-device model show one of these states:
| State | What to do |
|---|---|
| Ready | Click Use (Use it in the wizard). |
| Apple Intelligence is off | Turn on… (Open System Settings in the wizard), switch Apple Intelligence on, then come back. |
| macOS is still downloading it | Wait, then Check again. |
If the Mac can’t run it, the section doesn’t appear.
Its limits are real. Its window is about 4,000 tokens, which is enough to answer a question from the passages DocGraph finds in your notes and not enough for the tool list Agent Mode needs. So with Apple’s model selected:
- Agent Mode is unavailable. The Agent button in the chat is greyed out and explains that Apple’s model can’t create or edit notes. To let the AI act, pick a downloaded model.
- Chat answers questions about your notes from retrieved passages, without the tool loop. Long messages are trimmed to fit, and replies are kept short.
- It reads only some languages. Settings lists them. If an answer used notes in a language it doesn’t read, the chat adds a warning under the answer.
Manage models on this computer
Section titled “Manage models on this computer”Settings → AI → Models & providers holds everything about models that live on this computer. Its Built-in AI section covers the bundled runtime. The runtime status reads Ready · starts on your first question, In memory, or Stopped after repeated errors (with Try again). The runtime starts on the first question that needs it, never at launch, and listens only on this computer.
Downloaded models lists every model on this computer with its size and the total disk space used. A model tagged ✓ checked by MMF is in MMF’s curated list; one tagged your file is a file you added.
Switch the model chat uses
Section titled “Switch the model chat uses”Click Use on a downloaded model; the one in use reads In use. You can also pick any model in the chat’s model picker.
Download another model
Section titled “Download another model”Click Download a model…. It opens the setup wizard at the model choice, with the same size cards and the Advanced: a model from Hugging Face or a link fold described above. Downloads resume after an interruption and are checked when they arrive. A download DocGraph needs on its own, such as a model missing when the app opens, first asks in a dialog that names its size and where it comes from (Download or Not now); the answer is remembered per model.
Add a model file you already have
Section titled “Add a model file you already have”Click Add a model file you have… and pick a .gguf file on disk. It is listed as your file. Remove makes DocGraph forget it and leaves the file where it is.
Free memory: unload the model
Section titled “Free memory: unload the model”While a model is loaded, the status reads In memory with an Unload button. Click it to free the memory; the next question loads the model again.
Check for new models
Section titled “Check for new models”Click Check for new models to fetch MMF’s latest list of models that fit your computer. It reports up to date or that it couldn’t reach the list and is using the one built into DocGraph. Nothing about you is sent.
Delete a model
Section titled “Delete a model”Click Delete on a downloaded model and confirm. This frees its disk space; you can download it again later.
Use Ollama (optional)
Section titled “Use Ollama (optional)”Ollama is not required. If you already use it, the Ollama section below Built-in AI shows whether it is running (Connected · n models, Running · no models pulled or Not running) and lists its models:
- Pull a model: type a name (for example
gemma4:e2b-mlx, the Ollama model we recommend on a Mac) and click Pull. Progress shows with Cancel. - unload appears on a model that is loaded in memory.
- Hover a row for its trash icon to delete that model from Ollama, after you confirm.
- MLX? appears on a model running on the default llama.cpp engine. Click it to ask Ollama’s registry (
registry.ollama.ai) whether an Apple MLX build exists (this sends only the model name, and only when you click). If one does, its name appears; click it to put it in the pull box. Otherwise the row reads no MLX build. A model that already runs on MLX is tagged MLX.
DocGraph talks to Ollama at http://localhost:11434; change it under Settings → AI → Chat & index → Ollama base URL. An Ollama on another machine is refused unless you turn on Allow remote Ollama there and confirm.
Picking a model in chat
Section titled “Picking a model in chat”The model picker at the top of the chat panel lists every model you can use: your Pinned ones (click the star on a row), the last five you used, then one section per provider — local first, then any cloud provider you have set up, then external agents. Type to search, or filter with the chips (Local, Cloud, a provider, Agents, and Tools / Vision / Reasoning / Free). Each row shows the model’s size, context window and an MLX tag where it applies; hover the picker for its quantization, memory use and the speed DocGraph measured on this computer. When both an MLX and a llama.cpp build of the same model are installed, ⇄ Mac-native version (or ⇄ Standard version) under the chat’s header switches to the other build.
Only a local pick (built-in, Apple or Ollama) becomes your default chat model; a cloud model or an agent is used by that conversation only. Settings → AI → Chat & index holds the rest: the default chat model, a separate Utility model for small background jobs, the Context window, Keep the model in memory (30 minutes by default) and Temperature.
DocGraph learns how each model behaves on your computer — how fast it answers, whether it can handle the small helper jobs (rewriting follow-up questions, expanding searches), whether warming it up in advance pays off. A helper that proves too slow is skipped for 30 days. Settings → AI → Chat & index → Last answer shows what your most recent wait was made of (loading the model, searching your notes, or writing) and the measured speed per model.
Models we’ve measured
Section titled “Models we’ve measured”We test models on our own Mac (Apple M4 Pro, 24 GB of memory) with a fixed set of questions about a sample vault. These numbers come from that one machine. Yours will differ with its chip and memory, and the wizard’s suggestion is based on your computer, not on this table.
| If you want… | Model | On our test Mac |
|---|---|---|
| Speed, with Agent Mode | gemma4:e2b-mlx (Ollama) | Answers in about 3 seconds, about 6 in Agent Mode. Got all but one or two test questions right. Uses about 8 GB of memory. |
| The most accurate answers | qwen3.5:4b-mlx (Ollama) | Answers in about 5–9 seconds. Got every test question right. Uses about 6.5 GB. |
| Less memory, no Ollama needed | Ministral 3 3B Reasoning (Download a model; the built-in default) | Answers in about 6 seconds. Got all but one test question right. Uses about 5 GB. |
| Nothing to download | Apple’s on-device model | Answers in about 4–5 seconds. No Agent Mode (see above). |
Bigger is not always better here. A 12-billion-parameter model gave equally good answers but took over a minute in Agent Mode and used 14 GB of memory. The two Ollama models above run on Apple’s MLX engine, which is why they are fast on a Mac (see MLX in Use Ollama).
Search language
Section titled “Search language”The AI finds your notes with a small search helper that runs on your computer — never with the chat model. You choose which one, and the AI can’t search your notes until you do:
| Choice | Best for | Download |
|---|---|---|
| English only | Notes written in English | about 75 MB |
| Many languages | Arabic, Hebrew, Chinese, French, Spanish… and English. Finds notes across languages, such as an English question about a note written in Arabic. | about 270 MB |
Nothing is picked for you. Until you choose, nothing is downloaded or indexed for the AI; ⌘K keyword search works either way because it needs no model. The setup wizard asks; with AI on, DocGraph also asks when it starts, and the chat panel asks above its message box, which stays disabled until you choose.
- One choice for every vault. Change it in Settings → AI → Chat & index → Search language. Each vault is re-indexed with the new helper — the open one right away, the others the next time you open them.
- Choosing is the download approval. You have just read the size, so DocGraph does not ask a second time.
- Each helper comes with a reranker, a second small model that reads every passage search found together with your question and puts the best first. Its size is included above. Turn it off with Settings → AI → Chat & index → Rerank results.
- With English, the reranker can also tell when nothing in your notes is about the question; chat then says so instead of answering from unrelated notes. The many-languages reranker only reorders.
Requirements
Section titled “Requirements”DocGraph runs on Macs with Apple Silicon. Local models need enough memory and disk space. The wizard measures your memory and disk and only offers sizes that fit; if none do, use an AI provider. Apple’s on-device model needs macOS 26 on Apple Silicon.
Privacy
Section titled “Privacy”- Your notes are embedded and indexed locally.
- Unless you pick a cloud model or an agent, chat requests go to a model on this computer: the built-in runtime (which listens only on
127.0.0.1, protected by a random key), Apple’s model, or your local Ollama. DocGraph refuses a non-local Ollama unless you explicitly turn on Allow remote Ollama. - Turning AI features off stops the AI’s indexing (embeddings) and all model use; nothing is sent to any AI. ⌘K keyword search keeps working.
A few features can reach beyond your machine. All are opt-in and clearly marked:
- Cloud AI providers (OpenAI, Anthropic, OpenRouter, a custom endpoint) are used only once you save a key, allow the provider and pick one of its models.
- Web search is off by default. With it on, the assistant shows you each search before it is sent, and results are treated as untrusted input. Allow in this chat lets you skip that card for the rest of one conversation.
- External CLI agents (Claude Code, Codex CLI, Gemini CLI, OpenCode) hand a turn to a tool that runs with full vault access and bills your own account with that provider. DocGraph checks the prompt it hands over for secrets first; what the agent sends after that is not inspected, and its changes are not covered by chat’s per-change confirmations. Enabling one requires ticking an acknowledgement.