litelotus/.nix
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

01-server: deepseek-ocr + embeddings in ollama, whisper.cpp STT server #25

Merged
litelotus merged 1 commit from feat/ollama-models-whisper into main 2026-09-14 14:43:43 +00:00
Conversation 0 Commits 1 Files changed 1 +25 -2
litelotus commented 2026-09-14 13:02:20 +00:00
Owner
Copy link

Model stack expansion on the k3s master node:

ollama (loadModels):

  • deepseek-ocr — 3B OCR vision-language model, official ollama library tag (ollama run deepseek-ocr), converts documents/tables/handwriting to markdown with optional grounding
  • nomic-embed-text — 274MB embeddings model for automation/RAG tooling

whisper.cpp STT server (new systemd service):

  • Ollama has no STT; whisper-cpp 1.8.5 (in pinned nixpkgs) ships both the server and the model downloader
  • OpenAI-compatible /v1/audio/transcriptions on :8000, small model fetched on first start, CPU inference (no GPU contention with ollama), DynamicUser + hardened unit
  • Hermes gateway STT intentionally untouched (hermes has no custom-endpoint STT provider — this serves the automation/notes tooling)

Validated: nix build .#nixosConfigurations.the k3s master node.config.system.build.toplevel exit 0.

Caveat: deepseek-ocr needs llama.cpp VLM support in the ollama 0.32.3 pin (b9509); if ollama run deepseek-ocr fails at runtime with an arch error, we'll pull the GGUF route instead.

Model stack expansion on the k3s master node: **ollama (`loadModels`)**: - `deepseek-ocr` — 3B OCR vision-language model, official ollama library tag (`ollama run deepseek-ocr`), converts documents/tables/handwriting to markdown with optional grounding - `nomic-embed-text` — 274MB embeddings model for automation/RAG tooling **whisper.cpp STT server** (new systemd service): - Ollama has no STT; `whisper-cpp` 1.8.5 (in pinned nixpkgs) ships both the server and the model downloader - OpenAI-compatible `/v1/audio/transcriptions` on :8000, small model fetched on first start, CPU inference (no GPU contention with ollama), DynamicUser + hardened unit - Hermes gateway STT intentionally untouched (hermes has no custom-endpoint STT provider — this serves the automation/notes tooling) Validated: `nix build .#nixosConfigurations.the k3s master node.config.system.build.toplevel` exit 0. Caveat: deepseek-ocr needs llama.cpp VLM support in the ollama 0.32.3 pin (b9509); if `ollama run deepseek-ocr` fails at runtime with an arch error, we'll pull the GGUF route instead.
- services.ollama.loadModels += deepseek-ocr (3B OCR VLM, in the ollama
  library), nomic-embed-text (274MB embeddings for automation/RAG tooling)
- new whisper-cpp systemd service: OpenAI-compatible
  /v1/audio/transcriptions on :8000, small model downloaded on first start
  via whisper-cpp-download-ggml-model (packaged), CPU inference (no GPU
  contention with ollama), DynamicUser + hardened
- firewall: open 8000

Ollama has no STT — whisper.cpp ships server + model downloader in the
pinned nixpkgs (whisper-cpp 1.8.5). Hermes gateway STT stays on its own
providers (no custom-endpoint STT in hermes); this serves automation/notes
tooling.

Validated: nix build toplevel exit 0.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
litelotus/.nix!25
No description provided.