01-server: ollama native CUDA 12.9 backend (sm_50) — carry-over of #23 into main #24

Merged

litelotus merged 1 commit from feat/ollama-cuda12 into main 2026-09-14 12:52:41 +00:00

Conversation 0 Commits 1 Files changed 1 +10 -3

litelotus commented 2026-09-14 12:52:26 +00:00

Owner

Copy link

The CUDA swap (commit e750cc2) was merged into its base branch feat/ollama-vulkan-fixes via PR #23, but that base branch had already been merged into main (PR #22) before #23 landed — so main never received the change. Running generation on 01 is still pkgs.ollama-vulkan (journalctl shows OLLAMA_VULKAN:true).

This PR carries the same single-file change into main: package = pkgs.ollama.override { acceleration = "cuda"; cudaPackages = pkgs.cudaPackages_12; cudaArches = [ "sm_50" "sm_52" ]; }.

After merge: pull ~/.nix on the build machine and nixos-rebuild switch 01, then verify with journalctl -u ollama -n 30 | grep -iE "cuda|vulkan" — expect the cuda_v12 runner and no OLLAMA_VULKAN env.

The CUDA swap (commit e750cc2) was merged into its base branch `feat/ollama-vulkan-fixes` via PR #23, but that base branch had **already** been merged into main (PR #22) *before* #23 landed — so main never received the change. Running generation on 01 is still `pkgs.ollama-vulkan` (journalctl shows `OLLAMA_VULKAN:true`). This PR carries the same single-file change into main: `package = pkgs.ollama.override { acceleration = "cuda"; cudaPackages = pkgs.cudaPackages_12; cudaArches = [ "sm_50" "sm_52" ]; }`. After merge: pull `~/.nix` on the build machine and `nixos-rebuild switch` 01, then verify with `journalctl -u ollama -n 30 | grep -iE "cuda|vulkan"` — expect the cuda_v12 runner and no `OLLAMA_VULKAN` env.

litelotus added 1 commit 2026-09-14 12:52:26 +00:00

01-server: ollama native CUDA 12.9 backend (sm_50) instead of Vulkan ... e750cc2dc3

Build current ollama 0.32.3 with the CUDA 12 backend targeting Maxwell:

- acceleration=cuda, cudaPackages=cudaPackages_12 (12.9.79 — last 12.x line
  that still compiles sm_50; CUDA 13 dropped Maxwell entirely)
- cudaArches=[sm_50 sm_52] — nixpkgs' 12.x realArches default starts at
  sm_75, so the M10s would get no kernels without explicit arches
- llama.cpp b9509 (ollama 0.32.x pin) still ships '50 == Maxwell' in its
  default CUDA arch list; all feature gates are __CUDA_ARCH__ >= X guards
  that degrade gracefully on sm_50 — no version pin needed

Stacks on feat/ollama-vulkan-fixes (MDWE sandbox fix stays; harmless for
CUDA and preserves the one-line fallback to pkgs.ollama-vulkan).

Expectation: native quantized GEMM kernels + no shader-JIT dependency vs
the Vulkan backend; A/B benchmark on qwen3:8b.

litelotus merged commit 5ce5778922 into main 2026-09-14 12:52:41 +00:00

litelotus referenced this pull request from a commit 2026-09-14 12:52:43 +00:00

Merge pull request '01-server: ollama native CUDA 12.9 backend (sm_50) — carry-over of #23 into main' (#24) from feat/ollama-cuda12 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

litelotus/.nix!24

No description provided.