01-server: fix ollama Vulkan JIT sandbox breakage + pin NVIDIA ICD #22

Merged

litelotus merged 1 commit from feat/ollama-vulkan-fixes into main

2026-09-14 12:46:56 +00:00

litelotus commented

2026-09-14 12:21:47 +00:00

Owner

The 'painfully slow' root cause — the nixpkgs ollama unit runs with MemoryDenyWriteExecute=true, which denies the W+X mprotect Vulkan's shader JIT needs. Ollama logs show JIT session error: Permission denied; GPU layers fall back to software rendering, and qwen3:8b Q4 crawls at ~0.5 tok/s despite all 8 M10 dies being reserved.

Two changes in hosts/nixos/the k3s master node/default.nix:

systemd.services.ollama.serviceConfig.MemoryDenyWriteExecute = lib.mkForce false — lets the shader JIT work
VK_ICD_FILENAMES pinned to the NVIDIA ICD — Mesa's lavapipe (software Vulkan) is enumerated on this host and can steal layers

Validated: nix build .#nixosConfigurations.the k3s master node.config.system.build.toplevel exit 0.

Expected: ~0.5 → 10–25 tok/s for 8B Q4 on the M10s. If we still want more after measuring, next step is a custom CUDA 12.8 build (nixpkgs ollama-cuda is CUDA 13 = cannot target sm_50).

**The 'painfully slow' root cause** — the nixpkgs ollama unit runs with `MemoryDenyWriteExecute=true`, which denies the W+X mprotect Vulkan's shader JIT needs. Ollama logs show `JIT session error: Permission denied`; GPU layers fall back to software rendering, and qwen3:8b Q4 crawls at ~0.5 tok/s despite all 8 M10 dies being reserved. Two changes in `hosts/nixos/the k3s master node/default.nix`: 1. `systemd.services.ollama.serviceConfig.MemoryDenyWriteExecute = lib.mkForce false` — lets the shader JIT work 2. `VK_ICD_FILENAMES` pinned to the NVIDIA ICD — Mesa's lavapipe (software Vulkan) is enumerated on this host and can steal layers Validated: `nix build .#nixosConfigurations.the k3s master node.config.system.build.toplevel` exit 0. Expected: ~0.5 → 10–25 tok/s for 8B Q4 on the M10s. If we still want more after measuring, next step is a custom CUDA 12.8 build (nixpkgs ollama-cuda is CUDA 13 = cannot target sm_50).

litelotus added 1 commit

2026-09-14 12:21:47 +00:00

01-server: fix ollama Vulkan JIT sandbox breakage + pin NVIDIA ICD

ci/crow/push/check Pipeline failed

Details

5361601c75

nixpkgs' ollama unit sets MemoryDenyWriteExecute=true, which denies the W+X
mprotect the Vulkan shader JIT needs. Symptom: 'JIT session error: Permission
denied' in ollama logs; GPU layers fall back to software rendering and an 8B
Q4 model crawls at ~0.5 tok/s despite all 8 M10 dies being reserved.

- systemd.services.ollama.serviceConfig.MemoryDenyWriteExecute = mkForce false
- VK_ICD_FILENAMES pinned to the NVIDIA ICD so Mesa's lavapipe (software
  Vulkan, enumerated as a device on this host) can never steal layers

Verified: nix build .#nixosConfigurations.01-server.config.system.build.toplevel
exit 0. Benchmark before/after on qwen3:8b expected: ~0.5 -> 10-25 tok/s.

litelotus referenced this pull request

2026-09-14 12:28:59 +00:00

01-server: ollama native CUDA 12.9 backend (sm_50) instead of Vulkan #23

litelotus merged commit 30d21537e8 into main

2026-09-14 12:46:56 +00:00

litelotus referenced this pull request from a commit

2026-09-14 12:46:58 +00:00

Merge pull request '01-server: fix ollama Vulkan JIT sandbox breakage + pin NVIDIA ICD' (#22) from feat/ollama-vulkan-fixes into main

litelotus referenced this pull request

2026-09-14 12:52:26 +00:00

01-server: ollama native CUDA 12.9 backend (sm_50) — carry-over of #23 into main #24