01-server: ollama native CUDA 12.9 backend (sm_50) instead of Vulkan #23

Merged

litelotus merged 1 commit from feat/ollama-cuda12 into feat/ollama-vulkan-fixes

2026-09-14 12:47:05 +00:00

litelotus commented

2026-09-14 12:28:59 +00:00

Owner

Swap the Vulkan ollama for a native CUDA build on the M10s — real kernels designed for the hardware instead of Vulkan's shader abstractions.

The build (services.ollama.package):

pkgs.ollama.override { acceleration = "cuda"; cudaPackages = pkgs.cudaPackages_12; cudaArches = [ "sm_50" "sm_52" ]; }
CUDA 13 dropped Maxwell, so we use cudaPackages_12 — 12.9.79 in the pinned rev (the intent of '12.8'; still the last line that compiles sm_50)
nixpkgs' 12.x realArches starts at sm_75 — explicit cudaArches is mandatory or the M10 gets zero kernels
llama.cpp b9509 (ollama 0.32.x pin) still ships 50 == Maxwell in its default CUDA arch list and all feature gates are __CUDA_ARCH__ >= X guards — so no version pin needed, we keep modern ollama
sm_52 included for a second Maxwell-class host's GTX 970 if we ever point it here

Stacks on #22 (feat/ollama-vulkan-fixes) — the MDWE sandbox fix stays in (harmless for CUDA, and keeps a one-line fallback to pkgs.ollama-vulkan if the CUDA path underperforms).

A/B benchmark after merge+rebuild: qwen3:8b tok/s, Vulkan vs CUDA.

Swap the Vulkan ollama for a native CUDA build on the M10s — real kernels designed for the hardware instead of Vulkan's shader abstractions. **The build** (`services.ollama.package`): - `pkgs.ollama.override { acceleration = "cuda"; cudaPackages = pkgs.cudaPackages_12; cudaArches = [ "sm_50" "sm_52" ]; }` - CUDA 13 dropped Maxwell, so we use `cudaPackages_12` — **12.9.79** in the pinned rev (the intent of '12.8'; still the last line that compiles sm_50) - nixpkgs' 12.x `realArches` starts at sm_75 — explicit `cudaArches` is mandatory or the M10 gets zero kernels - llama.cpp **b9509** (ollama 0.32.x pin) still ships `50 == Maxwell` in its default CUDA arch list and all feature gates are `__CUDA_ARCH__ >= X` guards — so **no version pin needed**, we keep modern ollama - sm_52 included for a second Maxwell-class host's GTX 970 if we ever point it here **Stacks on #22** (feat/ollama-vulkan-fixes) — the MDWE sandbox fix stays in (harmless for CUDA, and keeps a one-line fallback to `pkgs.ollama-vulkan` if the CUDA path underperforms). A/B benchmark after merge+rebuild: qwen3:8b tok/s, Vulkan vs CUDA.

litelotus added 1 commit

2026-09-14 12:28:59 +00:00

01-server: ollama native CUDA 12.9 backend (sm_50) instead of Vulkan

ci/crow/push/check Pipeline failed

Details

e750cc2dc3

Build current ollama 0.32.3 with the CUDA 12 backend targeting Maxwell:

- acceleration=cuda, cudaPackages=cudaPackages_12 (12.9.79 — last 12.x line
  that still compiles sm_50; CUDA 13 dropped Maxwell entirely)
- cudaArches=[sm_50 sm_52] — nixpkgs' 12.x realArches default starts at
  sm_75, so the M10s would get no kernels without explicit arches
- llama.cpp b9509 (ollama 0.32.x pin) still ships '50 == Maxwell' in its
  default CUDA arch list; all feature gates are __CUDA_ARCH__ >= X guards
  that degrade gracefully on sm_50 — no version pin needed

Stacks on feat/ollama-vulkan-fixes (MDWE sandbox fix stays; harmless for
CUDA and preserves the one-line fallback to pkgs.ollama-vulkan).

Expectation: native quantized GEMM kernels + no shader-JIT dependency vs
the Vulkan backend; A/B benchmark on qwen3:8b.

litelotus merged commit 5c32fdc44e into feat/ollama-vulkan-fixes

2026-09-14 12:47:05 +00:00

litelotus referenced this pull request from a commit

2026-09-14 12:47:05 +00:00

Merge pull request '01-server: ollama native CUDA 12.9 backend (sm_50) instead of Vulkan' (#23) from feat/ollama-cuda12 into feat/ollama-vulkan-fixes

litelotus referenced this pull request

2026-09-14 12:52:26 +00:00

01-server: ollama native CUDA 12.9 backend (sm_50) — carry-over of #23 into main #24

litelotus referenced this pull request from a commit

2026-09-14 12:52:43 +00:00

Merge pull request '01-server: ollama native CUDA 12.9 backend (sm_50) — carry-over of #23 into main' (#24) from feat/ollama-cuda12 into main