ollama likely remains stuck in an unclassified userspace loop. A diagnosis report and patch proposal were created locally.
Published session
Prompt
## Plan Pass
You are planning a fixer patch before any edits happen.
Read the evidence bundle at `./evidence.json`. The prepared workspace is `./workspace` and it was acquired via `local-executable-upstream-git`. The original pre-edit snapshot is available at `./source` if you need to inspect it. For interpreter processes, plan from the script/application entrypoint evidence first and include the runtime only as a second investigation target unless the evidence proves a runtime bug.
Validation expectation: try the project-level build/test entrypoint from the workspace root before reporting only a focused leaf compile. Detected candidate(s): `cmake -S . -B build-fix -G Ninja && cmake --build build-fix && ctest --test-dir build-fix --output-on-failure`. If the project-level command fails because dependencies or generated files are missing, include the exact command and failure reason in `## Validation`, then run the narrowest relevant compile/test that is still reproducible from a clean checkout.
Upstream-style expectation: before planning or editing, check for contribution/style docs (`CONTRIBUTING`, `HACKING`, `README-hacking`, `README.md`, `docs/`, `dev-docs/`) and scan the touched subsystem for local helpers. If the project has wrappers for file IO, path-relative IO, process spawning, memory allocation, logging, locking, or platform compatibility, prefer those wrappers over generic libc/std APIs. For public upstream review, prepare the final patch against the upstream default integration branch, usually `master` or `main`; use installed or stable source trees for reproduction/backport validation, not as the default merge-request base unless the contribution docs explicitly say to target a stable branch. Do not invent a reproducer or user-visible failure that is not in the evidence bundle; if the evidence is profiler-only or indirect, describe it as a targeted mitigation or stop with a diagnosis instead of presenting a speculative patch as a confirmed bug fix. In the plan and final validation, name any such helper, convention, branch-base choice, or evidence limit you found, or say that no relevant local helper was found. Treat this as a `github.com_ollama_ollama` upstream patch, not just a Debian-local workaround.
Codex subagent/fan-out expectation: for any non-trivial investigation, do not keep all checking in the lead context. Proactively spawn bounded leaf subagents for independent source inspection, duplicate/upstream search, implementation sanity checks, and verification while the lead owns integration. Use the newest available mini model for read-only scouts, and use `gpt-5.3-codex-spark` aggressively for tiny coding, test, and verification leaves even when the lead model is `gpt-5.6-luna`. Pass model names explicitly when spawning. Keep subagents as leaves: tell them not to spawn their own agents, not to run recursive `codex review`, not to perform public/external/destructive actions, and not to duplicate the same live-state operation. Merge their evidence into the final patch/report instead of merely saying a subagent looked. Inspect the relevant code, nearby callers, project contribution docs, and local helper/compat APIs, but do not edit files in this pass.
Return a short markdown plan with these exact sections:
## Problem
## Evidence Confidence
## Proposed Subject
## Patch Plan
## Risks
## Validation
Classify `## Evidence Confidence` as exactly one of `reproduced`, `observed`, or `inferred`. Use `inferred` only for a no-patch diagnosis/report plan unless you can name the extra evidence you will collect before editing; inferred source patches are blocked by Fixer because they are not pull-request-ready. For `observed` source-patch plans, plan to say in the final `## Issue Connection` that the failure was observed by Fixer and not independently reproduced. If the plan touches authentication, authorization, credentials, cryptography, sandboxing, permissions, timing/throttling behavior, or PostgreSQL database-core behavior such as dynamic library resolution, extension loading, transaction semantics, locking, planner/executor behavior, catalog changes, or storage changes, treat it as high-risk: do not plan a source patch unless validation can reproduce the behavior and analyze the semantic impact; otherwise plan a no-patch diagnosis/report. The plan must explain how the proposed code change addresses the observed issue evidence, call out any prior Fixer patch that should be improved or replaced, reject awkward control flow such as avoidable `goto` if there is a cleaner bounded alternative, name any local helper APIs or maintainer conventions the patch should follow, and keep the intended maintainer-facing explanation clear enough that someone unfamiliar with the local complaint wording can still follow the fix. In `## Validation`, name the reproducible configure/build/test entrypoint you will try from the workspace root before any focused leaf compile or smoke check, and include one bounded independent reproduction attempt for the collected failure signal when it is safe and cheap. Do not plan to claim `reproduced` unless that reproduction command or test can actually show the failure.
## Patch Pass
You are working on a bounded fixer proposal.
Read the evidence bundle at `./evidence.json`. The prepared workspace is `./workspace` and it was acquired via `local-executable-upstream-git`. Produce the smallest reasonable patch for the target repository, keep the change upstreamable, prefer the clearest control flow available, and do not keep avoidable `goto` when a simpler structure would read better. Before introducing new file, process, allocation, locking, networking, or platform APIs, inspect nearby code and project contribution docs for existing helpers or compatibility wrappers and use those local patterns unless you can explain why they do not fit. Validate from a reproducible workspace-root entrypoint before falling back to focused leaf commands; if a build or test cannot run, report the exact command, the exact blocker, and any narrower check you ran instead. During validation, also try one bounded independent reproduction of the collected failure signal when it is safe and cheap, such as a failing test, smoke command, perf/strace comparison, or before/after runtime check. Only use `reproduced` if that command or test actually reproduced the failure; otherwise keep `observed` and report the reproduction blocker. The final explanation must connect the observed issue evidence to the actual code change, not just paraphrase the diff. Write like a maintainer is going to read the patch mail cold: explain the bug in plain language, define subsystem-specific jargon the first time you need it, and make the causal story obvious. Explicitly classify evidence confidence as `reproduced`, `observed`, or `inferred`: `reproduced` means you reproduced the failure locally; `observed` means Fixer has direct crash/log/trace evidence but you did not independently reproduce it; `inferred` means the source patch is not pull-request-ready, so do not leave a source diff unless you first gather stronger observed/reproduced evidence; otherwise return a no-patch diagnosis/report. For any source-changing `observed` patch, say explicitly in `## Issue Connection` that the failure was observed by Fixer and not independently reproduced. Security-sensitive areas such as authentication, authorization, credentials, cryptography, sandboxing, permissions, and timing/throttling behavior need reproduced evidence plus explicit security-impact analysis before leaving a source diff; otherwise return a no-patch diagnosis/report for human review. If you introduce non-obvious state translation, index remapping, or backend split logic, add a short source comment that explains the invariant being preserved.
Start by explaining the likely root cause from the collected perf, strace, and /proc evidence. If you cannot land a safe patch, leave a diagnosis that is strong enough for an upstream bug report.
Validation expectation: try the project-level build/test entrypoint from the workspace root before reporting only a focused leaf compile. Detected candidate(s): `cmake -S . -B build-fix -G Ninja && cmake --build build-fix && ctest --test-dir build-fix --output-on-failure`. If the project-level command fails because dependencies or generated files are missing, include the exact command and failure reason in `## Validation`, then run the narrowest relevant compile/test that is still reproducible from a clean checkout.
Upstream-style expectation: before planning or editing, check for contribution/style docs (`CONTRIBUTING`, `HACKING`, `README-hacking`, `README.md`, `docs/`, `dev-docs/`) and scan the touched subsystem for local helpers. If the project has wrappers for file IO, path-relative IO, process spawning, memory allocation, logging, locking, or platform compatibility, prefer those wrappers over generic libc/std APIs. For public upstream review, prepare the final patch against the upstream default integration branch, usually `master` or `main`; use installed or stable source trees for reproduction/backport validation, not as the default merge-request base unless the contribution docs explicitly say to target a stable branch. Do not invent a reproducer or user-visible failure that is not in the evidence bundle; if the evidence is profiler-only or indirect, describe it as a targeted mitigation or stop with a diagnosis instead of presenting a speculative patch as a confirmed bug fix. In the plan and final validation, name any such helper, convention, branch-base choice, or evidence limit you found, or say that no relevant local helper was found. Treat this as a `github.com_ollama_ollama` upstream patch, not just a Debian-local workaround.
Codex subagent/fan-out expectation: for any non-trivial investigation, do not keep all checking in the lead context. Proactively spawn bounded leaf subagents for independent source inspection, duplicate/upstream search, implementation sanity checks, and verification while the lead owns integration. Use the newest available mini model for read-only scouts, and use `gpt-5.3-codex-spark` aggressively for tiny coding, test, and verification leaves even when the lead model is `gpt-5.6-luna`. Pass model names explicitly when spawning. Keep subagents as leaves: tell them not to spawn their own agents, not to run recursive `codex review`, not to perform public/external/destructive actions, and not to duplicate the same live-state operation. Merge their evidence into the final patch/report instead of merely saying a subagent looked.
Keep the change narrowly scoped and summarize validation clearly.
In every authoring pass, your final response must start with `Subject: <single-line git commit subject>` and then include these markdown sections exactly:
## Commit Message
A short upstream-friendly explanation of what changed and why. Write it in plain language that a maintainer can follow without local complaint context. If you use subsystem jargon, define it immediately.
## Evidence Confidence
Exactly one word: `reproduced`, `observed`, or `inferred`. Use `reproduced` only when you reproduced the failure locally with a command or test, and include that command/test in `## Validation`. Use `observed` when Fixer has direct crash/log/trace evidence but you did not independently reproduce it. If `## Git Add Paths` lists source files for an `observed` patch, `## Issue Connection` must explicitly say the failure was observed by Fixer and not independently reproduced. Security-sensitive source changes touching authentication, authorization, credentials, cryptography, sandboxing, permissions, or timing/throttling behavior require reproduced evidence and explicit security-impact analysis. PostgreSQL database-core source changes touching dynamic library resolution, extension loading, transaction semantics, locking, planner/executor behavior, catalog changes, or storage changes require reproduced evidence plus explicit semantic-impact analysis. Otherwise list `None` under `## Git Add Paths` and write a no-patch diagnosis/report. Use `inferred` for profiler/strace/indirect evidence; inferred responses may be no-patch diagnoses or reports, but inferred source patches are not pull-request-ready until stronger evidence is gathered.
## Issue Connection
Write this as maintainer-facing patch mail, not as local Fixer notes. Cover four things explicitly in readable sentences: the user-visible symptom or the exact collected signal, the code-level cause or the cautious inference from evidence, the specific change you made, and the expected effect. Do not invent a reproducer, command line, crash, or user-visible failure that is not present in the evidence bundle. If the evidence is direct-but-not-reproduced, say it was observed by Fixer and not independently reproduced. If the evidence is indirect and you did not gather stronger evidence, do not leave a source diff; write a no-patch diagnosis/report instead. Include an explicit effect sentence such as `The expected effect is ...`, `This should reduce ...`, or `This prevents ...` for source patches. If the logic is non-obvious in code, mention that you added a short explanatory comment.
## Git Add Paths
List the repo-relative paths that belong in the final patch, one per line. Use `None` only when you intentionally made no source changes. Include intentionally new files, and do not list generated build artifacts.
## Validation
List the checks you ran, or say clearly that you could not run them. Include the independent reproduction command/test and result when `## Evidence Confidence` is `reproduced`; if reproduction was attempted but blocked, name the exact blocker and keep confidence at `observed` or `inferred`.
Before editing, read the plan at `./plan-output.txt` and follow it unless the code proves part of it wrong. If you change course, say so explicitly in the final write-up instead of silently drifting from the plan.
## Review Pass 1 (local metadata check)
Local metadata consistency check
## Refinement Pass 1
You are refining a fixer patch after an explicit code review.
Read the evidence bundle at `./evidence.json`. The prepared workspace is `./workspace` and it was acquired via `local-executable-upstream-git`. Read the latest author response at `./patch-output.txt`. Read the review report at `./review-1-output.txt`. This is refinement round 1. The original pre-edit snapshot is available at `./source` if you need to compare the current patch against it. Re-read the planning pass at `./plan-output.txt` before editing. The workspace currently changes these repo-relative paths: build-fix/.ninja_log, build-fix/CMakeCache.txt, build-fix/CMakeFiles/4.3.4/CMakeCCompiler.cmake, build-fix/CMakeFiles/4.3.4/CMakeCXXCompiler.cmake, build-fix/CMakeFiles/4.3.4/CMakeDetermineCompilerABI_C.bin, build-fix/CMakeFiles/4.3.4/CMakeDetermineCompilerABI_CXX.bin, build-fix/CMakeFiles/4.3.4/CMakeSystem.cmake, build-fix/CMakeFiles/4.3.4/CompilerIdC/CMakeCCompilerId.c, build-fix/CMakeFiles/4.3.4/CompilerIdC/a.out, build-fix/CMakeFiles/4.3.4/CompilerIdCXX/CMakeCXXCompilerId.cpp, build-fix/CMakeFiles/4.3.4/CompilerIdCXX/a.out, build-fix/CMakeFiles/CMakeConfigureLog.yaml, build-fix/CMakeFiles/InstallScripts.json, build-fix/CMakeFiles/TargetDirectories.txt, build-fix/CMakeFiles/cmake.check_cache, build-fix/CMakeFiles/ollama-llama-cpp-source.dir/Labels.json, build-fix/CMakeFiles/ollama-llama-cpp-source.dir/Labels.txt, build-fix/CMakeFiles/ollama-llama-server-local.dir/Labels.json, build-fix/CMakeFiles/ollama-llama-server-local.dir/Labels.txt, build-fix/CMakeFiles/rules.ninja, build-fix/Testing/Temporary/CTestCostData.txt, build-fix/Testing/Temporary/LastTest.log, build-fix/build.ninja, build-fix/cmake_install.cmake, build-fix/ollama-llama-cpp-source-prefix/src/ollama-llama-cpp-source-stamp/ollama-llama-cpp-source-gitinfo.txt, build-fix/ollama-llama-cpp-source-prefix/src/ollama-llama-cpp-source-stamp/ollama-llama-cpp-source-mkdir, build-fix/ollama-llama-cpp-source-prefix/src/ollama-llama-cpp-source-stamp/ollama-llama-cpp-source-patch-info.txt, build-fix/ollama-llama-cpp-source-prefix/src/ollama-llama-cpp-source-stamp/ollama-llama-cpp-source-update-info.txt, build-fix/ollama-llama-cpp-source-prefix/tmp/ollama-llama-cpp-source-cfgcmd.txt, build-fix/ollama-llama
[truncated 12729 bytes]
Response
## Plan Pass
## Problem
Fixer observed `/usr/local/bin/ollama serve` with unresolved userspace frames and futex-heavy samples. The evidence classifies the loop as unknown-userspace-loop with only 0.42 confidence; no functional failure, hang, or user-visible regression is established.
## Evidence Confidence
inferred
The signal is profiler-only and indirect. Source inspection found expected Go synchronization, scheduler, channel, and subprocess-wait paths, but no confirmed defect. The actual inference work runs in the external `llama-server` process.
## Proposed Subject
No source patch proposed. Prepare a diagnosis/report requesting symbolized, process-separated profiling before any upstream fix.
## Patch Plan
- Keep the upstream patch base at `master`; the repository contains only the `Fixer baseline` commit and no prior Fixer patch to improve or replace.
- Do not modify `server/routes.go`, `server/sched.go`, or `llm/llama_server.go` based on this evidence alone.
- If investigation continues, collect debug-symbol profiles separately for Ollama and `llama-server`, with workload and lifecycle state recorded.
- Preserve local conventions such as `envconfig.Host`, `logutil.NewLogger`, scheduler contexts, and existing `exec.Command` handling if a later confirmed fix is identified.
- No new `goto` or other awkward control flow is warranted.
## Risks
A speculative scheduler or runtime change could alter request ordering, shutdown behavior, model loading, or concurrency without addressing the sampled process. The observed futex waits may be normal blocking rather than contention.
## Validation
- Attempted the required project command:
`cmake -S . -B build-fix -G Ninja && cmake --build build-fix && ctest --test-dir build-fix --output-on-failure`
Configure succeeded, but the build failed because `llama.cpp` could not be cloned (`Could not resolve host: github.com`) and Go reported `module cache not found: neither GOMODCACHE nor GOPATH is set`; CTest therefore was not reached.
- Narrow compile/test attempt:
`go test ./server`
Failed for the same missing Go module-cache configuration.
- Bounded independent runtime attempt:
`timeout 8s env HOME=/tmp/ollama-repro-home OLLAMA_HOST=127.0.0.1:19876 /usr/local/bin/ollama serve`
Did not reproduce the profiler signal; the environment rejected the listener with `socket: operation not permitted`.
No source files were changed.
## Patch Pass
Subject: server: diagnose unresolved futex profile without speculative patch
## Commit Message
No source change. The collected profile does not identify a confirmed Ollama defect: `/usr/local/bin/ollama serve` shows unresolved Go frames and futex waits, while the actual inference work runs in a separate `llama-server` process. Symbolized, process-separated profiling is needed before proposing an upstream fix.
## Evidence Confidence
inferred
## Issue Connection
Fixer observed a futex-heavy profile with unresolved userspace frames, but no crash, hang, or user-visible failure was reported. The cautious source-level inference is that the samples are consistent with normal Go scheduler, channel, mutex, context, and subprocess-wait blocking; no futex-specific implementation or confirmed busy loop was found.
Accordingly, no source change was made. A speculative change to scheduler retries, goroutine creation, or shutdown could alter concurrency without addressing the sampled process. The expected next step is to collect symbolized profiles separately for Ollama and `llama-server`, including workload and lifecycle state.
No prior Fixer patch exists to improve or replace. The repository remains based on upstream `master`; contribution guidance requires behavior-focused tests and package-prefixed lowercase commit subjects. Existing helpers and conventions include `envconfig.Host`, `logutil.NewLogger`, scheduler contexts, and the existing `exec.Command` subprocess boundary.
## Git Add Paths
None
## Validation
- Project-level command attempted:
`cmake -S . -B build-fix -G Ninja && cmake --build build-fix && ctest --test-dir build-fix --output-on-failure`
Configure succeeded, but the build failed because cloning `https://github.com/ggml-org/llama.cpp.git` failed with `Could not resolve host: github.com`; Go also reported `module cache not found: neither GOMODCACHE nor GOPATH is set`. CTest was not reached.
- Focused check attempted:
`go test ./server`
Failed with `go: module cache not found: neither GOMODCACHE nor GOPATH is set`.
- Bounded independent reproduction attempted:
`timeout 8s env HOME=/tmp/ollama-repro-home OLLAMA_HOST=127.0.0.1:19876 /usr/local/bin/ollama serve`
The profiler signal was not reproduced because the environment rejected the listener with `socket: operation not permitted`.
No source files were changed.
## Review Pass 1 (local metadata check)
RESULT: fix-needed
## Findings
1. Patch metadata drift: the workspace currently changes `build-fix/.ninja_log`, `build-fix/CMakeCache.txt`, `build-fix/CMakeFiles/4.3.4/CMakeCCompiler.cmake`, `build-fix/CMakeFiles/4.3.4/CMakeCXXCompiler.cmake`, `build-fix/CMakeFiles/4.3.4/CMakeDetermineCompilerABI_C.bin`, `build-fix/CMakeFiles/4.3.4/CMakeDetermineCompilerABI_CXX.bin`, `build-fix/CMakeFiles/4.3.4/CMakeSystem.cmake`, `build-fix/CMakeFiles/4.3.4/CompilerIdC/CMakeCCompilerId.c`, `build-fix/CMakeFiles/4.3.4/CompilerIdC/a.out`, `build-fix/CMakeFiles/4.3.4/CompilerIdCXX/CMakeCXXCompilerId.cpp`, `build-fix/CMakeFiles/4.3.4/CompilerIdCXX/a.out`, `build-fix/CMakeFiles/CMakeConfigureLog.yaml`, `build-fix/CMakeFiles/InstallScripts.json`, `build-fix/CMakeFiles/TargetDirectories.txt`, `build-fix/CMakeFiles/cmake.check_cache`, `build-fix/CMakeFiles/ollama-llama-cpp-source.dir/Labels.json`, `build-fix/CMakeFiles/ollama-llama-cpp-source.dir/Labels.txt`, `build-fix/CMakeFiles/ollama-llama-server-local.dir/Labels.json`, `build-fix/CMakeFiles/ollama-llama-server-local.dir/Labels.txt`, `build-fix/CMakeFiles/rules.ninja`, `build-fix/Testing/Temporary/CTestCostData.txt`, `build-fix/Testing/Temporary/LastTest.log`, `build-fix/build.ninja`, `build-fix/cmake_install.cmake`, `build-fix/ollama-llama-cpp-source-prefix/src/ollama-llama-cpp-source-stamp/ollama-llama-cpp-source-gitinfo.txt`, `build-fix/ollama-llama-cpp-source-prefix/src/ollama-llama-cpp-source-stamp/ollama-llama-cpp-source-mkdir`, `build-fix/ollama-llama-cpp-source-prefix/src/ollama-llama-cpp-source-stamp/ollama-llama-cpp-source-patch-info.txt`, `build-fix/ollama-llama-cpp-source-prefix/src/ollama-llama-cpp-source-stamp/ollama-llama-cpp-source-update-info.txt`, `build-fix/ollama-llama-cpp-source-prefix/tmp/ollama-llama-cpp-source-cfgcmd.txt`, `build-fix/ollama-llama-cpp-source-prefix/tmp/ollama-llama-cpp-source-gitclone.cmake`, `build-fix/ollama-llama-cpp-source-prefix/tmp/ollama-llama-cpp-source-gitupdate.cmake`, `build-fix/ollama-llama-cpp-source-prefix/tmp/ollama-llama-cpp-source-mkdirs.cmake`, `build-fix/ollama-llama-server-local-prefix/src/ollama-llama-server-local-stamp/ollama-llama-server-local-patch-info.txt`, `build-fix/ollama-llama-server-local-prefix/src/ollama-llama-server-local-stamp/ollama-llama-server-local-source_dirinfo.txt`, `build-fix/ollama-llama-server-local-prefix/src/ollama-llama-server-local-stamp/ollama-llama-server-local-update-info.txt`, `build-fix/ollama-llama-server-local-prefix/tmp/ollama-llama-server-local-cfgcmd.txt`, `build-fix/ollama-llama-server-local-prefix/tmp/ollama-llama-server-local-mkdirs.cmake`, but `## Git Add Paths` lists `None`. Update `## Git Add Paths` to match the real shipped file set exactly, and make sure `## Issue Connection` explains every functional file that remains in the patch.
2. Missing from `## Git Add Paths`: `build-fix/.ninja_log`, `build-fix/CMakeCache.txt`, `build-fix/CMakeFiles/4.3.4/CMakeCCompiler.cmake`, `build-fix/CMakeFiles/4.3.4/CMakeCXXCompiler.cmake`, `build-fix/CMakeFiles/4.3.4/CMakeDetermineCompilerABI_C.bin`, `build-fix/CMakeFiles/4.3.4/CMakeDetermineCompilerABI_CXX.bin`, `build-fix/CMakeFiles/4.3.4/CMakeSystem.cmake`, `build-fix/CMakeFiles/4.3.4/CompilerIdC/CMakeCCompilerId.c`, `build-fix/CMakeFiles/4.3.4/CompilerIdC/a.out`, `build-fix/CMakeFiles/4.3.4/CompilerIdCXX/CMakeCXXCompilerId.cpp`, `build-fix/CMakeFiles/4.3.4/CompilerIdCXX/a.out`, `build-fix/CMakeFiles/CMakeConfigureLog.yaml`, `build-fix/CMakeFiles/InstallScripts.json`, `build-fix/CMakeFiles/TargetDirectories.txt`, `build-fix/CMakeFiles/cmake.check_cache`, `build-fix/CMakeFiles/ollama-llama-cpp-source.dir/Labels.json`, `build-fix/CMakeFiles/ollama-llama-cpp-source.dir/Labels.txt`, `build-fix/CMakeFiles/ollama-llama-server-local.dir/Labels.json`, `build-fix/CMakeFiles/ollama-llama-server-local.dir/Labels.txt`, `build-fix/CMakeFiles/rules.ninja`, `build-fix/Testing/Temporary/CTestCostData.txt`, `build-fix/Testing/Temporary/LastTest.log`, `build-fix/build.ninja`, `build-fix/cmake_install.cmake`, `build-fix/ollama-llama-cpp-source-prefix/src/ollama-llama-cpp-source-stamp/ollama-llama-cpp-source-gitinfo.txt`, `build-fix/ollama-llama-cpp-source-prefix/src/ollama-llama-cpp-source-stamp/ollama-llama-cpp-source-mkdir`, `build-fix/ollama-llama-cpp-source-prefix/src/ollama-llama-cpp-source-stamp/ollama-llama-cpp-source-patch-info.txt`, `build-fix/ollama-llama-cpp-source-prefix/src/ollama-llama-cpp-source-stamp/ollama-llama-cpp-source-update-info.txt`, `build-fix/ollama-llama-cpp-source-prefix/tmp/ollama-llama-cpp-source-cfgcmd.txt`, `build-fix/ollama-llama-cpp-source-prefix/tmp/ollama-llama-cpp-source-gitclone.cmake`, `build-fix/ollama-llama-cpp-source-prefix/tmp/ollama-llama-cpp-source-gitupdate.cmake`, `build-fix/ollama-llama-cpp-source-prefix/tmp/ollama-llama-cpp-source-mkdirs.cmake`, `build-fix/ollama-llama-server-local-prefix/src/ollama-llama-server-local-stamp/ollama-llama-server-local-patch-info.txt`, `build-fix/ollama-llama-server-local-prefix/src/ollama-llama-server-local-stamp/ollama-llama-server-local-source_dirinfo.txt`, `build-fix/ollama-llama-server-local-prefix/src/ollama-llama-server-local-stamp/ollama-llama-server-local-update-info.txt`, `build-fix/ollama-llama-server-local-prefix/tmp/ollama-llama-server-local-cfgcmd.txt`, `build-fix/ollama-llama-server-local-prefix/tmp/ollama-llama-server-local-mkdirs.cmake`.
2. Listed in `## Git Add Paths` but not actually changed: `None`.
## Refinement Pass 1
Subject: server: report unresolved futex profile without speculative patch
## Commit Message
No source change. The review finding was metadata drift from generated `build-fix/` files, not a runtime defect. Those generated artifacts were removed, leaving no patch files.
## Evidence Confidence
inferred
## Issue Connection
Fixer observed unresolved userspace frames and repeated futex waits from `/usr/local/bin/ollama serve`. Source inspection shows that `serve` starts the Go scheduler, while model inference runs in an external `llama-server` subprocess launched through the existing `exec.Command` boundary. The evidence cannot distinguish normal Go synchronization from scheduler contention, native llama.cpp behavior, or driver activity.
No source change was made because the profile does not identify a safe code-level cause. The expected next step is version-matched, symbolized profiling of Ollama and `llama-server` separately, with workload and lifecycle state recorded.
The review’s generated-artifact finding was addressed by removing `build-fix/`; no prior Fixer patch exists to improve or replace. The repository remains based on upstream `master`, following the documented package-prefixed commit and behavior-focused test conventions.
## Git Add Paths
None
## Validation
- Project-level command attempted:
`cmake -S . -B build-fix -G Ninja && cmake --build build-fix && ctest --test-dir build-fix --output-on-failure`
Configure succeeded, but the build failed because cloning `https://github.com/ggml-org/llama.cpp.git` failed with `Could not resolve host: github.com`; Go also reported `module cache not found: neither GOMODCACHE nor GOPATH is set`. CTest was not reached.
- Focused check attempted:
`go test ./server`
Failed with `go: module cache not found: neither GOMODCACHE nor GOPATH is set`.
- Bounded independent reproduction attempted:
`timeout 8s env HOME=/tmp/ollama-repro-home OLLAMA_HOST=127.0.0.1:19876 /usr/local/bin/ollama serve`
The profiler signal was not reproduced because the environment rejected the listener with `socket: operation not permitted`.
The worktree is clean and contains no source or generated build changes.
## Review Pass 2
RESULT: ok
The refined response is a valid no-patch diagnosis:
- `inferred` correctly reflects the profiler-only evidence and lack of independent reproduction.
- It does not invent a user-visible failure or claim a source fix.
- The serve/scheduler versus external `llama-server` boundary is accurately described.
- Validation includes the required project-level command, its exact blockers, a focused test attempt, and a bounded reproduction attempt.
- `## Git Add Paths` correctly lists `None`, and the generated `build-fix/` artifacts are no longer present.
- The subject, upstream `master` base, helper conventions, and maintainer-facing explanation are appropriate.
Diff
--- a/harmony/harmonyparser.go 2026-09-22 22:46:27.469844484 +0400
+++ b/harmony/harmonyparser.go 2026-09-22 20:07:30.000000000 +0400
@@ -268,7 +268,6 @@
FunctionNameMap *FunctionNameMap
toolAccumulator *HarmonyToolCallAccumulator
convertedTools map[string]struct{}
- contentPrefill bool // the prompt continues a final-channel message
}
// NewHarmonyMessageHandler creates a new message handler
@@ -407,7 +406,6 @@
} else {
h.HarmonyParser.AddImplicitStart()
}
- h.contentPrefill = lastMessage != nil && lastMessage.Role == "assistant" && lastMessage.Content != ""
// Initialize tool accumulator
h.toolAccumulator = h.CreateToolParser()
@@ -463,23 +461,6 @@
return true
}
-// The analysis ends when the model opens a message whose body is content: the
-// final channel in any constrain spelling parseHeader accepts, a commentary
-// message with no recipient, or a message with no channel.
-func (h *HarmonyMessageHandler) ThinkingClose() []string {
- if h.contentPrefill {
- return nil
- }
- return []string{
- "<|end|><|start|>assistant<|channel|>final<|message|>",
- "<|end|><|start|>assistant<|channel|>final <|constrain|>json<|message|>",
- "<|end|><|start|>assistant<|channel|>final<|constrain|>json<|message|>",
- "<|end|><|start|>assistant<|channel|>final json<|message|>",
- "<|end|><|start|>assistant<|channel|>commentary<|message|>",
- "<|end|><|start|>assistant<|message|>",
- }
-}
-
func (h *HarmonyMessageHandler) PreservedTokens() []string {
// <|call|> is an EOG marker for tool calls. Preserve structural tokens
// used by the parser, but let llama-server stop on the call terminator.
--- a/llm/gbnf.go 2026-09-22 22:46:27.469844484 +0400
+++ b/llm/gbnf.go 1970-01-01 04:00:00.000000000 +0400
@@ -1,180 +0,0 @@
-package llm
-
-import (
- "fmt"
- "slices"
- "strings"
- "unicode"
-)
-
-// thinkingGrammar returns a grammar that leaves the text before any of the
-// closings unconstrained, then constrains what follows the first complete
-// closing to the root of the format grammar. The response may end before any
-// closing. A closing that is a prefix of another ends the thinking as soon
-// as it is complete.
-func thinkingGrammar(closings []string, format string) string {
- // The rules added here take a prefix no rule of the format grammar starts
- // with.
- prefix := "ollama-"
- for grammarHasRulePrefix(format, prefix) {
- prefix += "ollama-"
- }
- thinking, formatted := prefix+"thinking-", prefix+"format"
-
- a := newClosingAutomaton(closings)
- var b strings.Builder
- fmt.Fprintf(&b, "root ::= %s0\n", thinking)
- for state := range a.states {
- fmt.Fprintf(&b, "%s%d ::=", thinking, state)
- var matching, terminal []rune
- next := map[int][]rune{}
- for _, r := range a.alphabet {
- to, done := a.next(state, r)
- if done {
- terminal = append(terminal, r)
- } else if to != 0 {
- next[to] = append(next[to], r)
- }
- if done || to != 0 {
- matching = append(matching, r)
- }
- }
- // Any other character restarts the search from the empty prefix.
- fmt.Fprintf(&b, " | %s %s0", gbnfCharClass(matching, true), thinking)
- for to := range a.states {
- if rs := next[to]; len(rs) > 0 {
- fmt.Fprintf(&b, " | %s %s%d", gbnfCharClass(rs, false), thinking, to)
- }
- }
- if len(terminal) > 0 {
- fmt.Fprintf(&b, " | %s %s", gbnfCharClass(terminal, false), formatted)
- }
- b.WriteByte('\n')
- }
- b.WriteString(renameGrammarRule(format, "root", formatted))
- return b.String()
-}
-
-// closingAutomaton is the Aho-Corasick automaton over the closings: each
-// state is a proper prefix of a closing, state 0 the empty one.
-type closingAutomaton struct {
- closings []string
- states []string
- index map[string]int
- alphabet []rune
-}
-
-func newClosingAutomaton(closings []string) *closingAutomaton {
- a := &closingAutomaton{closings: closings, index: map[string]int{}}
- add := func(prefix string) {
- if _, ok := a.index[prefix]; !ok {
- a.index[prefix] = len(a.states)
- a.states = append(a.states, prefix)
- }
- }
- add("")
- for _, closing := range closings {
- for i := range closing {
- add(closing[:i])
- }
- for _, r := range closing {
- if !slices.Contains(a.alphabet, r) {
- a.alphabet = append(a.alphabet, r)
- }
- }
- }
- slices.Sort(a.alphabet)
- return a
-}
-
-// next returns the state after reading r in state, or done when the text
-// read so far ends with a closing.
-func (a *closingAutomaton) next(state int, r rune) (to int, done bool) {
- s := a.states[state] + string(r)
- for _, closing := range a.closings {
- if strings.HasSuffix(s, closing) {
- return 0, true
- }
- }
- for i := range s {
- if to, ok := a.index[s[i:]]; ok {
- return to, false
- }
- }
- return 0, false
-}
-
-func gbnfCharClass(rs []rune, negate bool) string {
- var b strings.Builder
- b.WriteByte('[')
- if negate {
- b.WriteByte('^')
- }
- for _, r := range rs {
- switch {
- case r > unicode.MaxASCII || !unicode.IsPrint(r) || strings.ContainsRune(`]\^-`, r):
- fmt.Fprintf(&b, `\U%08X`, r)
- default:
- b.WriteRune(r)
- }
- }
- b.WriteByte(']')
- return b.String()
-}
-
-func renameGrammarRule(grammar, from, to string) string {
- var b strings.Builder
- scanGrammar(grammar, func(text string, name bool) {
- if name && text == from {
- text = to
- }
- b.WriteString(text)
- })
- return b.String()
-}
-
-func grammarHasRulePrefix(grammar, prefix string) bool {
- found := false
- scanGrammar(grammar, func(text string, name bool) {
- found = found || name && strings.HasPrefix(text, prefix)
- })
- return found
-}
-
-// scanGrammar splits a grammar into rule names and everything else. Comments,
-// quoted strings and character classes are passed whole, so a word inside
-// them is never taken for a name.
-func scanGrammar(grammar string, fn func(text string, name bool)) {
- isName := func(c byte) bool {
- return c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c == '-' || c == '_'
- }
- for i := 0; i < len(grammar); {
- start := i
- switch c := grammar[i]; {
- case c == '#':
- for i < len(grammar) && grammar[i] != '\n' {
- i++
- }
- case c == '"' || c == '[':
- end := byte('"')
- if c == '[' {
- end = ']'
- }
- for i++; i < len(grammar) && grammar[i] != end; i++ {
- if grammar[i] == '\\' {
- i++
- }
- }
- i = min(i+1, len(grammar))
- case isName(c):
- for i < len(grammar) && isName(grammar[i]) {
- i++
- }
- fn(grammar[start:i], true)
- continue
- default:
- i++
- }
- fn(grammar[start:i], false)
- }
-}
--- a/llm/gbnf_test.go 2026-09-22 22:46:27.469844484 +0400
+++ b/llm/gbnf_test.go 1970-01-01 04:00:00.000000000 +0400
@@ -1,79 +0,0 @@
-package llm
-
-import (
- "strings"
- "testing"
-)
-
-// TestRenameGrammarRule checks that only rule names change, not the same word
-// in a comment, a string, a character class or a longer name.
-func TestRenameGrammarRule(t *testing.T) {
- grammar := "# root stays in a comment\n" +
- "root ::= \"root\" | [root] | \"[\" root \"]\" | root-item\n" +
- "root-item ::= \"\\\"root\\\"\" root\n"
- want := "# root stays in a comment\n" +
- "out ::= \"root\" | [root] | \"[\" out \"]\" | root-item\n" +
- "root-item ::= \"\\\"root\\\"\" out\n"
- if got := renameGrammarRule(grammar, "root", "out"); got != want {
- t.Errorf("renamed grammar:\n%s\nwant:\n%s", got, want)
- }
-}
-
-// TestClosingAutomatonNext checks that the state after a character is the
-// longest suffix of the text so far that is a proper prefix of a closing, and
-// that the automaton is done exactly when the text ends with a closing. The
-// closings share a prefix and overlap, so a reset to the empty prefix on a
-// mismatch would lose matches.
-func TestClosingAutomatonNext(t *testing.T) {
- a := newClosingAutomaton([]string{"<|x|>", "<|y|>", "y|>|"})
- state := 0
- for _, step := range []struct {
- r rune
- want string
- done bool
- }{
- {r: '<', want: "<"},
- {r: '|', want: "<|"},
- {r: '<', want: "<"},
- {r: '|', want: "<|"},
- {r: 'y', want: "<|y"},
- {r: '|', want: "<|y|"},
- {r: '>', done: true},
- {r: '|', want: ""},
- {r: 'y', want: "y"},
- {r: '|', want: "y|"},
- {r: '>', want: "y|>"},
- {r: '<', want: "<"},
- {r: 'z', want: ""},
- } {
- to, done := a.next(state, step.r)
- if done != step.done || (!done && a.states[to] != step.want) {
- t.Fatalf("after %q: state %q done %v, want state %q done %v", step.r, a.states[to], done, step.want, step.done)
- }
- state = to
- }
-}
-
-// TestThinkingGrammarRules checks the emitted rules: every prefix state may
-// end the response, completing the closing leads to the renamed format root,
-// the format grammar's other rules keep their names, and the added rules
-// avoid a prefix the format grammar already uses.
-func TestThinkingGrammarRules(t *testing.T) {
- format := "root ::= \"{\" space \"}\"\nspace ::= | \" \"\n"
- grammar := thinkingGrammar([]string{"</think>"}, format)
- for _, want := range []string{
- "root ::= ollama-thinking-0\n",
- "ollama-thinking-0 ::= | [^<] ollama-thinking-0 | [<] ollama-thinking-1\n",
- "ollama-thinking-7 ::= | [^<>] ollama-thinking-0 | [<] ollama-thinking-1 | [>] ollama-format\n",
- "ollama-format ::= \"{\" space \"}\"\nspace ::= | \" \"\n",
- } {
- if !strings.Contains(grammar, want) {
- t.Errorf("grammar lacks %q:\n%s", want, grammar)
- }
- }
-
- grammar = thinkingGrammar([]string{"</think>"}, "root ::= ollama-x\nollama-x ::= \"{}\"\n")
- if !strings.HasPrefix(grammar, "root ::= ollama-ollama-thinking-0\n") || !strings.Contains(grammar, "ollama-ollama-format ::= ollama-x\n") {
- t.Errorf("grammar does not avoid the format's rule prefix:\n%s", grammar)
- }
-}
--- a/llm/llama_server.go 2026-09-22 22:46:27.469844484 +0400
+++ b/llm/llama_server.go 2026-09-22 20:07:30.000000000 +0400
@@ -5,10 +5,9 @@
// still render prompts in Go and call /completion. Other GGUF chat models use
// llama-server's chat_template handling through /v1/chat/completions.
//
-// For structured output, a JSON schema is passed to llama-server via its
-// json_schema field and the "json" format as a builtin grammar via the grammar
-// field. A format that applies after a response's thinking is sent as a
-// grammar built around the GBNF llama-server itself derives from the schema.
+// For structured output, JSON schemas are passed directly to llama-server via
+// its json_schema field (avoiding the CGO SchemaToGrammar dependency). Raw BNF
+// grammars are passed via the grammar field.
//
// llama-server auto-detects GPU layers (-ngl), thread count (-t), and flash
// attention (--flash-attn).
@@ -17,7 +16,6 @@
import (
"bufio"
"bytes"
- "container/list"
"context"
crand "crypto/rand"
"encoding/base64"
@@ -1427,85 +1425,6 @@
TimingsPerToken bool `json:"timings_per_token,omitempty"`
}
-// schemaGrammars caches the grammars llama-server derives from JSON schemas,
-// most recently used first. The conversion depends only on the llama-server
-// build, which is fixed for the process, so entries are shared by every
-// runner and never expire.
-var schemaGrammars = struct {
- sync.Mutex
- entries map[string]*list.Element
- order list.List
-}{entries: map[string]*list.Element{}}
-
-const schemaGrammarsSize = 64
-
-type schemaGrammar struct {
- schema, grammar string
-}
-
-// schemaGrammar converts a JSON schema to GBNF with llama-server's own
-// converter. An empty completion evaluates and generates nothing, but its
-// final response still reports the grammar the schema was converted to.
-func (s *llamaServerRunner) schemaGrammar(ctx context.Context, schema json.RawMessage) (string, error) {
- key := string(schema)
- schemaGrammars.Lock()
- if e, ok := schemaGrammars.entries[key]; ok {
- schemaGrammars.order.MoveToFront(e)
- schemaGrammars.Unlock()
- return e.Value.(*schemaGrammar).grammar, nil
- }
- schemaGrammars.Unlock()
-
- body, err := json.Marshal(struct {
- Prompt [][]int `json:"prompt"`
- NPredict int `json:"n_predict"`
- JsonSchema json.RawMessage `json:"json_schema"`
- ResponseFields []string `json:"response_fields"`
- }{Prompt: [][]int{{}}, JsonSchema: schema, ResponseFields: []string{"generation_settings/grammar"}})
- if err != nil {
- return "", fmt.Errorf("failed to marshal grammar request: %v", err)
- }
- endpoint := fmt.Sprintf("http://127.0.0.1:%d/completion", s.port)
- serverReq, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
- if err != nil {
- return "", fmt.Errorf("error creating grammar request: %v", err)
- }
- serverReq.Header.Set("Content-Type", "application/json")
- res, err := s.httpClient().Do(serverReq)
- if err != nil {
- return "", fmt.Errorf("llama-server grammar request failed: %v", err)
- }
- defer res.Body.Close()
- resBody, err := io.ReadAll(res.Body)
- if err != nil {
- return "", fmt.Errorf("failed reading llama-server grammar response: %w", err)
- }
- if res.StatusCode >= 400 {
- return "", api.StatusError{StatusCode: res.StatusCode, ErrorMessage: s.statusErrorMessage(resBody)}
- }
- var lsRes struct {
- Grammar string `json:"generation_settings/grammar"`
- }
- if err := json.Unmarshal(resBody, &lsRes); err != nil {
- return "", fmt.Errorf("error unmarshalling llama-server grammar response: %v", err)
- }
- if lsRes.Grammar == "" {
- return "", errors.New("llama-server returned no grammar for the schema")
- }
-
- schemaGrammars.Lock()
- defer schemaGrammars.Unlock()
- if _, ok := schemaGrammars.entries[key]; !ok {
- if schemaGrammars.order.Len() == schemaGrammarsSize {
- oldest := schemaGrammars.order.Back()
- delete(schemaGrammars.entries, oldest.Value.(*schemaGrammar).schema)
- schemaGrammars.order.Remove(oldest)
- }
- schemaGrammars.entries[key] = schemaGrammars.order.PushFront(&schemaGrammar{schema: key, grammar: lsRes.Grammar})
- }
- return lsRes.Grammar, nil
-}
-
func llamaServerPreservedTokens(parserTokens []string, toolCallTag string) []string {
tokens := append([]string{}, parserTokens...)
tokens = append(tokens, llamaServerPreservedTokensForToolTag(toolCallTag)...)
@@ -1669,6 +1588,7 @@
TypicalP: req.Options.TypicalP,
Seed: req.Options.Seed,
PreservedTokens: llamaServerPreservedTokens(req.PreservedTokens, req.ToolCallTag),
+ TimingsPerToken: req.IncludeIntermediateMetrics,
}
if req.Logprobs {
@@ -1691,19 +1611,6 @@
}
}
- // A format on a thinking response applies after the closing string, which
- // only a grammar of our own can express, so a schema is converted first.
- if len(req.ThinkingClose) > 0 && (lsReq.Grammar != "" || lsReq.JsonSchema != nil) {
- if lsReq.Grammar == "" {
- grammar, err := s.schemaGrammar(ctx, lsReq.JsonSchema)
- if err != nil {
- return err
- }
- lsReq.Grammar, lsReq.JsonSchema = grammar, nil
- }
- lsReq.Grammar = thinkingGrammar(req.ThinkingClose, lsReq.Grammar)
- }
-
// Convert media: replace Ollama's stable [img-N] markers with the per-process
// llama-server media marker and package the matching payloads as base64.
if len(req.Media) > 0 {
@@ -1809,10 +1716,16 @@
}
if lsResp.Content != "" && !lsResp.Stop {
- fn(CompletionResponse{
- Content: lsResp.Content,
- Logprobs: convertLogprobs(lsResp.CompletionProbabilities, req.TopLogprobs > 0),
- })
+ resp := CompletionResponse{Content: lsResp.Content}
+ if req.IncludeIntermediateMetrics {
+ resp.PromptEvalCount = lsResp.Timings.promptEvalCount()
+ resp.PromptEvalCachedCount = lsResp.Timings.CacheN
+ resp.PromptEvalDuration = time.Duration(lsResp.Timings.PromptMS * float64(time.Millisecond))
+ resp.EvalCount = lsResp.Timings.PredictN
+ resp.EvalDuration = time.Duration(lsResp.Timings.PredictMS * float64(time.Millisecond))
+ }
+ resp.Logprobs = convertLogprobs(lsResp.CompletionProbabilities, req.TopLogprobs > 0)
+ fn(resp)
}
if lsResp.Stop {
--- a/llm/llama_server_test.go 2026-09-22 22:46:27.469844484 +0400
+++ b/llm/llama_server_test.go 2026-09-22 20:07:30.000000000 +0400
@@ -187,6 +187,9 @@
if !reqBody.Stream {
t.Error("stream should be true")
}
+ if !reqBody.TimingsPerToken {
+ t.Error("timings_per_token should be true")
+ }
w.Header().Set("Content-Type", "text/event-stream")
for _, line := range sseLines {
@@ -209,8 +212,9 @@
var responses []CompletionResponse
opts := api.DefaultOptions()
err := runner.Completion(t.Context(), CompletionRequest{
- Prompt: "test prompt",
- Options: &opts,
+ Prompt: "test prompt",
+ Options: &opts,
+ IncludeIntermediateMetrics: true,
}, func(cr CompletionResponse) {
responses = append(responses, cr)
})
@@ -229,11 +233,29 @@
if responses[0].Done {
t.Error("response[0] should not be done")
}
+ if responses[0].PromptEvalCount != 5 || responses[0].EvalCount != 1 {
+ t.Errorf("response[0] counts = (%d, %d), want (5, 1)", responses[0].PromptEvalCount, responses[0].EvalCount)
+ }
+ if got := responses[0].PromptEvalCachedCount; got == nil || *got != 2 {
+ t.Errorf("response[0] cached prompt count = %v, want 2", got)
+ }
+ if responses[0].PromptEvalDuration != 10500*time.Microsecond || responses[0].EvalDuration != 9100*time.Microsecond {
+ t.Errorf("response[0] durations = (%s, %s), want (10.5ms, 9.1ms)", responses[0].PromptEvalDuration, responses[0].EvalDuration)
+ }
// Second token
if responses[1].Content != " world" {
t.Errorf("response[1].Content = %q, want %q", responses[1].Content, " world")
}
+ if responses[1].PromptEvalCount != 5 || responses[1].EvalCount != 2 {
+ t.Errorf("response[1] counts = (%d, %d), want (5, 2)", responses[1].PromptEvalCount, responses[1].EvalCount)
+ }
+ if got := responses[1].PromptEvalCachedCount; got == nil || *got != 2 {
+ t.Errorf("response[1] cached prompt count = %v, want 2", got)
+ }
+ if responses[1].PromptEvalDuration != 10500*time.Microsecond || responses[1].EvalDuration != 20300*time.Microsecond {
+ t.Errorf("response[1] durations = (%s, %s), want (10.5ms, 20.3ms)", responses[1].PromptEvalDuration, responses[1].EvalDuration)
+ }
// Final response
if !responses[2].Done {
@@ -3825,116 +3847,3 @@
// SIGKILL children when the test process exits.
return cmd
}
-
-// TestLlamaServerCompletionThinkingFormat checks that a format on a thinking
-// response is one request carrying a grammar: a schema is converted by an
-// empty completion once per schema, "json" needs no conversion, the grammar
-// wraps the format rules behind the closing string, and content and metrics
-// pass through unchanged.
-func TestLlamaServerCompletionThinkingFormat(t *testing.T) {
- schema := json.RawMessage(`{"type":"object"}`)
- converted := "root ::= \"{\" space \"}\"\nspace ::= | \" \"\n"
- sseLines := []string{
- `data: {"content":"Let me think.</think>","stop":false,"timings":{"cache_n":2,"prompt_n":3,"prompt_ms":10,"predicted_n":3,"predicted_ms":15}}`,
- `data: {"content":"{}","stop":false,"timings":{"cache_n":2,"prompt_n":3,"prompt_ms":10,"predicted_n":4,"predicted_ms":20}}`,
- `data: {"content":"","stop":true,"stop_type":"eos","timings":{"cache_n":2,"prompt_n":3,"prompt_ms":10,"predicted_n":4,"predicted_ms":20}}`,
- }
-
- var conversions atomic.Int32
- grammars := make(chan string, 8)
- srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
- if r.URL.Path == "/health" {
- fmt.Fprint(w, `{"status":"ok"}`)
- return
- }
- var reqBody struct {
- Prompt any `json:"prompt"`
- NPredict *int `json:"n_predict"`
- JsonSchema json.RawMessage `json:"json_schema"`
- Grammar string `json:"grammar"`
- ResponseFields []string `json:"response_fields"`
- }
- if err := json.NewDecoder(r.Body).Decode(&reqBody); err != nil {
- t.Errorf("invalid request body: %v", err)
- return
- }
- if len(reqBody.ResponseFields) > 0 {
- conversions.Add(1)
- if !reflect.DeepEqual(reqBody.Prompt, []any{[]any{}}) || reqBody.NPredict == nil || *reqBody.NPredict != 0 {
- t.Errorf("conversion request prompt %v n_predict %v, want an empty token prompt and no generation", reqBody.Prompt, reqBody.NPredict)
- }
- if !bytes.Equal(reqBody.JsonSchema, schema) || !reflect.DeepEqual(reqBody.ResponseFields, []string{"generation_settings/grammar"}) {
- t.Errorf("conversion request schema %s fields %v", reqBody.JsonSchema, reqBody.ResponseFields)
- }
- json.NewEncoder(w).Encode(map[string]string{"generation_settings/grammar": converted})
- return
- }
- if reqBody.JsonSchema != nil {
- t.Errorf("completion carried schema %s", reqBody.JsonSchema)
- }
- grammars <- reqBody.Grammar
- w.Header().Set("Content-Type", "text/event-stream")
- for _, line := range sseLines {
- fmt.Fprintln(w, line)
- fmt.Fprintln(w)
- }
- }))
- defer srv.Close()
-
- parts := strings.Split(srv.URL, ":")
- var portInt int
- fmt.Sscanf(parts[len(parts)-1], "%d", &portInt)
-
- runner := &llamaServerRunner{
- port: portInt,
- cmd: fakeRunningCmd(),
- sem: semaphore.NewWeighted(1),
- options: api.Options{Runner: api.Runner{NumCtx: 2048}},
- }
-
- complete := func(format json.RawMessage) string {
- var responses []CompletionResponse
- opts := api.DefaultOptions()
- err := runner.Completion(t.Context(), CompletionRequest{
- Prompt: "test prompt",
- Format: format,
- ThinkingClose: []string{"</think>"},
- Options: &opts,
- }, func(cr CompletionResponse) {
- responses = append(responses, cr)
- })
- if err != nil {
- t.Fatalf("Completion error: %v", err)
- }
- var content strings.Builder
- for _, resp := range responses[:len(responses)-1] {
- content.WriteString(resp.Content)
- }
- if got := content.String(); got != "Let me think.</think>{}" {
- t.Errorf("streamed content = %q", got)
- }
- final := responses[len(responses)-1]
- if !final.Done || final.DoneReason != DoneReasonStop || final.EvalCount != 4 || final.PromptEvalCount != 5 {
- t.Errorf("final response = %+v, want done with 4 generated and 5 prompt tokens", final)
- }
- return <-grammars
- }
-
- for range 2 {
- grammar := complete(schema)
- if !strings.HasPrefix(grammar, "root ::= ollama-thinking-0\n") || !strings.Contains(grammar, "ollama-format ::= \"{\" space \"}\"\n") {
- t.Errorf("grammar does not wrap the converted schema:\n%s", grammar)
- }
- }
- if got := conversions.Load(); got != 1 {
- t.Errorf("schema converted %d times, want once", got)
- }
-
- grammar := complete(json.RawMessage(`"json"`))
- if !strings.HasPrefix(grammar, "root ::= ollama-thinking-0\n") || !strings.Contains(grammar, "ollama-format ::= object\n") {
- t.Errorf("grammar does not wrap the json grammar:\n%s", grammar)
- }
- if got := conversions.Load(); got != 1 {
- t.Errorf("json format converted a schema, %d conversions", got)
- }
-}
--- a/llm/server.go 2026-09-22 22:46:27.469844484 +0400
+++ b/llm/server.go 2026-09-22 20:07:30.000000000 +0400
@@ -205,10 +205,8 @@
PreservedTokens []string // parser tokens to render as text; ignored by non-llama-server runners
ToolCallTag string // raw generic tool parser tag, if any
LeadingBOS string // textual BOS emitted by Go rendering, if any
- // ThinkingClose holds the strings any of which ends the thinking the
- // response begins with, which Format leaves free; none when the response
- // starts in content.
- ThinkingClose []string
+ // IncludeIntermediateMetrics adds cumulative metrics to non-final responses; final responses always include metrics.
+ IncludeIntermediateMetrics bool
// Logprobs specifies whether to include log probabilities in the response
Logprobs bool
--- a/mlxrunner/client.go 2026-09-22 22:46:27.469844484 +0400
+++ b/mlxrunner/client.go 2026-09-22 20:07:30.000000000 +0400
@@ -114,12 +114,13 @@
}
type CompletionRequest struct {
- Prompt string
- Media []llm.MediaData
- Format json.RawMessage
- Options api.Options
- Logprobs bool
- TopLogprobs int
+ Prompt string
+ Media []llm.MediaData
+ Format json.RawMessage
+ Options api.Options
+ Logprobs bool
+ TopLogprobs int
+ IncludeIntermediateMetrics bool
}
type CompletionResponse struct {
@@ -154,8 +155,7 @@
}
// requestGrammar returns the structural tag the runner decodes under: the
-// API's format as a json_schema tag, behind the free thinking the response
-// begins with when there is any.
+// API's format wrapped into a json_schema tag.
func requestGrammar(req llm.CompletionRequest) json.RawMessage {
schema := req.Format
switch string(schema) {
@@ -165,42 +165,21 @@
// The API documents "json" as producing a JSON object.
schema = json.RawMessage(`{"type":"object"}`)
}
- format := `{"type":"json_schema","json_schema":` + string(schema) + `}`
- if len(req.ThinkingClose) > 0 {
- excludes := make([]string, len(req.ThinkingClose))
- closings := make([]string, len(req.ThinkingClose))
- for i, closing := range req.ThinkingClose {
- excludes[i] = jsonString(closing)
- closings[i] = `{"type":"const_string","value":` + excludes[i] + `}`
- }
- closing := closings[0]
- if len(closings) > 1 {
- closing = `{"type":"or","elements":[` + strings.Join(closings, ",") + `]}`
- }
- // The tail is optional so EOS stays legal mid-thinking.
- format = `{"type":"sequence","elements":[{"type":"any_text","excludes":[` + strings.Join(excludes, ",") + `]},` +
- `{"type":"optional","content":{"type":"sequence","elements":[` + closing + `,` + format + `]}}]}`
- }
- return json.RawMessage(`{"type":"structural_tag","format":` + format + `}`)
-}
-
-// jsonString quotes s without escaping the HTML characters tags carry.
-func jsonString(s string) string {
- var b strings.Builder
- enc := json.NewEncoder(&b)
- enc.SetEscapeHTML(false)
- _ = enc.Encode(s)
- return strings.TrimSuffix(b.String(), "\n")
+ tag := make(json.RawMessage, 0, len(schema)+64)
+ tag = append(tag, `{"type":"structural_tag","format":{"type":"json_schema","json_schema":`...)
+ tag = append(tag, schema...)
+ return append(tag, `}}`...)
}
// Completion implements llm.LlamaServer.
func (c *Client) Completion(ctx context.Context, req llm.CompletionRequest, fn func(llm.CompletionResponse)) error {
creq := CompletionRequest{
- Prompt: req.Prompt,
- Media: req.Media,
- Format: requestGrammar(req),
- Logprobs: req.Logprobs,
- TopLogprobs: req.TopLogprobs,
+ Prompt: req.Prompt,
+ Media: req.Media,
+ Format: requestGrammar(req),
+ Logprobs: req.Logprobs,
+ TopLogprobs: req.TopLogprobs,
+ IncludeIntermediateMetrics: req.IncludeIntermediateMetrics,
}
if req.Options != nil {
creq.Options = *req.Options
--- a/mlxrunner/client_test.go 2026-09-22 22:46:27.469844484 +0400
+++ b/mlxrunner/client_test.go 2026-09-22 20:07:30.000000000 +0400
@@ -2,11 +2,20 @@
import (
"encoding/json"
+ "net"
+ "net/http"
+ "net/http/httptest"
+ "strconv"
"testing"
+ "github.com/ollama/ollama/api"
"github.com/ollama/ollama/llm"
)
+func testIntPtr(v int) *int {
+ return &v
+}
+
func TestRequestGrammar(t *testing.T) {
schema := `{"type":"object","properties":{"answer":{"type":"string"}}}`
tag := `{"type":"structural_tag","format":{"type":"json_schema","json_schema":` + schema + `}}`
@@ -24,21 +33,6 @@
want: `{"type":"structural_tag","format":{"type":"json_schema","json_schema":{"type":"object"}}}`,
},
{name: "schema", req: llm.CompletionRequest{Format: json.RawMessage(schema)}, want: tag},
- {
- name: "schema after thinking",
- req: llm.CompletionRequest{Format: json.RawMessage(schema), ThinkingClose: []string{"</think>"}},
- want: `{"type":"structural_tag","format":{"type":"sequence","elements":[{"type":"any_text","excludes":["</think>"]},` +
- `{"type":"optional","content":{"type":"sequence","elements":[{"type":"const_string","value":"</think>"},` +
- `{"type":"json_schema","json_schema":` + schema + `}]}}]}}`,
- },
- {
- name: "schema after thinking with two closings",
- req: llm.CompletionRequest{Format: json.RawMessage(schema), ThinkingClose: []string{"<|final|>", "<|final|>json"}},
- want: `{"type":"structural_tag","format":{"type":"sequence","elements":[{"type":"any_text","excludes":["<|final|>","<|final|>json"]},` +
- `{"type":"optional","content":{"type":"sequence","elements":[{"type":"or","elements":[{"type":"const_string","value":"<|final|>"},{"type":"const_string","value":"<|final|>json"}]},` +
- `{"type":"json_schema","json_schema":` + schema + `}]}}]}}`,
- },
- {name: "thinking without a format", req: llm.CompletionRequest{ThinkingClose: []string{"</think>"}}},
} {
t.Run(tt.name, func(t *testing.T) {
if got := string(requestGrammar(tt.req)); got != tt.want {
@@ -47,3 +41,46 @@
})
}
}
+
+func TestClientCompletionRequestsIntermediateMetrics(t *testing.T) {
+ var request CompletionRequest
+ want := CompletionResponse{
+ Done: true,
+ PromptEvalCount: 10,
+ PromptEvalCachedCount: testIntPtr(4),
+ }
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
+ t.Errorf("decode request: %v", err)
+ return
+ }
+ if err := json.NewEncoder(w).Encode(want); err != nil {
+ t.Errorf("encode response: %v", err)
+ }
+ }))
+ t.Cleanup(srv.Close)
+
+ _, portString, err := net.SplitHostPort(srv.Listener.Addr().String())
+ if err != nil {
+ t.Fatalf("parse server port: %v", err)
+ }
+ port, err := strconv.Atoi(portString)
+ if err != nil {
+ t.Fatalf("parse server port: %v", err)
+ }
+ client := &Client{port: port, client: srv.Client()}
+ opts := api.DefaultOptions()
+ var got llm.CompletionResponse
+ if err := client.Completion(t.Context(), llm.CompletionRequest{
+ Options: &opts,
+ IncludeIntermediateMetrics: true,
+ }, func(response llm.CompletionResponse) { got = response }); err != nil {
+ t.Fatalf("Completion: %v", err)
+ }
+ if !request.IncludeIntermediateMetrics {
+ t.Fatal("metrics per token was not forwarded to the MLX runner")
+ }
+ if got.PromptEvalCount != want.PromptEvalCount || got.PromptEvalCachedCount == nil || *got.PromptEvalCachedCount != *want.PromptEvalCachedCount {
+ t.Errorf("prompt counts = (%d, %v), want (%d, %d)", got.PromptEvalCount, got.PromptEvalCachedCount, want.PromptEvalCount, *want.PromptEvalCachedCount)
+ }
+}
--- a/mlxrunner/mtp_test.go 2026-09-22 22:46:27.469844484 +0400
+++ b/mlxrunner/mtp_test.go 2026-09-22 20:07:30.000000000 +0400
@@ -9,6 +9,7 @@
"slices"
"strings"
"testing"
+ "time"
"github.com/ollama/ollama/api"
"github.com/ollama/ollama/mlx"
@@ -431,6 +432,56 @@
}
})
}
+
+func TestDecodeIntermediateMetrics(t *testing.T) {
+ mlxtest.Run(t, func(t *mlxtest.T) {
+ const eos int32 = 7
+ predict := map[int32]int32{1: 2, 2: 3, 3: eos, eos: 0}
+ r := mtpTestRunner(t, predict, []int32{eos}, sampler.Options{})
+ caches, _ := newMTPTestCaches(1)
+ session, ch := newMTPTestSession(caches)
+ session.inputs = []int32{0, 1}
+ session.remaining = []int32{1}
+ req := Request{
+ Responses: ch,
+ Tokens: session.inputs,
+ CompletionRequest: CompletionRequest{
+ Options: api.Options{NumPredict: 20},
+ IncludeIntermediateMetrics: true,
+ },
+ }
+ d := testDecoder(t, r, req, caches, []int32{1}, 1)
+ promptEval := 5 * time.Millisecond
+ if err := r.decode(context.Background(), req, session, d, promptEval); err != nil {
+ t.Fatalf("decode: %v", err)
+ }
+ d.close()
+
+ var responses []CompletionResponse
+ for len(ch) > 0 {
+ resp := <-ch
+ responses = append(responses, resp)
+ }
+ if len(responses) != 3 {
+ t.Fatalf("got %d responses, want 3", len(responses))
+ }
+ for i, resp := range responses[:2] {
+ if resp.PromptEvalCount != 2 || resp.PromptEvalCachedCount == nil || *resp.PromptEvalCachedCount != 1 || resp.PromptEvalDuration != promptEval {
+ t.Errorf("response[%d] prompt metrics = (%d, %v, %s), want (2, 1, %s)", i, resp.PromptEvalCount, resp.PromptEvalCachedCount, resp.PromptEvalDuration, promptEval)
+ }
+ if resp.EvalCount != i+1 || resp.EvalDuration <= 0 {
+ t.Errorf("response[%d] eval metrics = (%d, %s), want count %d and positive duration", i, resp.EvalCount, resp.EvalDuration, i+1)
+ }
+ }
+ final := responses[2]
+ if final.PromptEvalCount != 2 || final.PromptEvalCachedCount == nil || *final.PromptEvalCachedCount != 1 || final.PromptEvalDuration != promptEval {
+ t.Errorf("final prompt metrics = (%d, %v, %s), want (2, 1, %s)", final.PromptEvalCount, final.PromptEvalCachedCount, final.PromptEvalDuration, promptEval)
+ }
+ if final.EvalCount != 2 || final.EvalDuration <= 0 {
+ t.Errorf("final eval metrics = (%d, %s), want count 2 and positive duration", final.EvalCount, final.EvalDuration)
+ }
+ })
+}
func TestRunMTPDecodeSampled(t *testing.T) {
mlxtest.Run(t, func(t *mlxtest.T) {
--- a/mlxrunner/pipeline.go 2026-09-22 22:46:27.469844484 +0400
+++ b/mlxrunner/pipeline.go 2026-09-22 20:07:30.000000000 +0400
@@ -327,6 +327,14 @@
if !ok {
continue
}
+ // Two-pass structured output cancels the first pass before its final response.
+ if request.IncludeIntermediateMetrics {
+ resp.PromptEvalCount = len(request.Tokens)
+ resp.PromptEvalCachedCount = final.PromptEvalCachedCount
+ resp.PromptEvalDuration = promptEval
+ resp.EvalCount = generated
+ resp.EvalDuration = time.Since(now)
+ }
select {
case <-ctx.Done():
err = ctx.Err()
--- a/model/parsers/cogito.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/cogito.go 2026-09-22 20:07:30.000000000 +0400
@@ -46,13 +46,6 @@
return true
}
-func (p *CogitoParser) ThinkingClose() []string {
- if p.state == CogitoCollectingThinking {
- return []string{cogitoThinkingCloseTag}
- }
- return nil
-}
-
func (p *CogitoParser) PreservedTokens() []string {
return []string{
cogitoThinkingCloseTag,
--- a/model/parsers/cohere.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/cohere.go 2026-09-22 20:07:30.000000000 +0400
@@ -54,13 +54,6 @@
return true
}
-func (p *CohereParser) ThinkingClose() []string {
- if p.state == cohereCollectingThinking {
- return []string{cohereEndThinking}
- }
- return nil
-}
-
func (p *CohereParser) PreservedTokens() []string {
return []string{
"<|START_THINKING|>", cohereEndThinking,
--- a/model/parsers/deepseek3.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/deepseek3.go 2026-09-22 20:07:30.000000000 +0400
@@ -45,13 +45,6 @@
return p.hasThinkingSupport
}
-func (p *DeepSeek3Parser) ThinkingClose() []string {
- if p.state == DeepSeekCollectingThinking {
- return []string{deepseekThinkingCloseTag}
- }
- return nil
-}
-
func (p *DeepSeek3Parser) PreservedTokens() []string {
return []string{
deepseekThinkingCloseTag,
--- a/model/parsers/functiongemma.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/functiongemma.go 2026-09-22 20:07:30.000000000 +0400
@@ -30,7 +30,6 @@
func (p *FunctionGemmaParser) HasToolSupport() bool { return true }
func (p *FunctionGemmaParser) HasThinkingSupport() bool { return false }
-func (p *FunctionGemmaParser) ThinkingClose() []string { return nil }
func (p *FunctionGemmaParser) PreservedTokens() []string {
return []string{
--- a/model/parsers/gemma4.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/gemma4.go 2026-09-22 20:07:30.000000000 +0400
@@ -39,7 +39,6 @@
callIndex int
hasThinkingSupport bool
thinkingEnabled bool // true when both model supports and user requested thinking
- contentPrefill bool // true when the prompt continues assistant content
needsChannelNameStrip bool // true when we just entered thinking and need to strip "thought\n"
}
@@ -51,13 +50,6 @@
return p.hasThinkingSupport
}
-func (p *Gemma4Parser) ThinkingClose() []string {
- if p.thinkingEnabled && !p.contentPrefill {
- return []string{gemma4ThinkingCloseTag}
- }
- return nil
-}
-
func (p *Gemma4Parser) PreservedTokens() []string {
return []string{
gemma4ThinkingOpenTag,
@@ -73,7 +65,7 @@
p.tools = tools
p.callIndex = 0
- p.contentPrefill = lastMessage != nil && lastMessage.Role == "assistant" && lastMessage.Content != ""
+ prefill := lastMessage != nil && lastMessage.Role == "assistant"
p.thinkingEnabled = p.HasThinkingSupport() && (thinkValue != nil && thinkValue.Bool())
@@ -88,7 +80,7 @@
return tools
}
- if p.contentPrefill {
+ if prefill && lastMessage.Content != "" {
p.state = Gemma4CollectingContent
return tools
}
--- a/model/parsers/glimmer.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/glimmer.go 2026-09-22 20:07:30.000000000 +0400
@@ -51,19 +51,6 @@
func (p *GlimmerParser) HasToolSupport() bool { return true }
func (p *GlimmerParser) HasThinkingSupport() bool { return true }
-// The model opens the message to the user, with or without naming the
-// recipient, before any content; its self message may end implicitly, so
-// the header alone marks the boundary.
-func (p *GlimmerParser) ThinkingClose() []string {
- if p.emitThinking {
- return []string{
- glimmerStartTag + "assistant to=user" + glimmerMessageTag,
- glimmerStartTag + "assistant" + glimmerMessageTag,
- }
- }
- return nil
-}
-
func (p *GlimmerParser) PreservedTokens() []string {
return []string{glimmerStartTag, glimmerMessageTag, glimmerEndMessageTag, glimmerEndTurnTag}
}
--- a/model/parsers/glm46.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/glm46.go 2026-09-22 20:07:30.000000000 +0400
@@ -36,11 +36,10 @@
)
type GLM46Parser struct {
- state glm46ParserState
- buffer strings.Builder
- tools []api.Tool
- callIndex int
- thinkingEnabled bool
+ state glm46ParserState
+ buffer strings.Builder
+ tools []api.Tool
+ callIndex int
}
func (p *GLM46Parser) HasToolSupport() bool {
@@ -51,13 +50,6 @@
return true
}
-func (p *GLM46Parser) ThinkingClose() []string {
- if p.thinkingEnabled {
- return []string{glm46ThinkingCloseTag}
- }
- return nil
-}
-
func (p *GLM46Parser) PreservedTokens() []string {
return []string{
glm46ThinkingOpenTag,
@@ -75,7 +67,6 @@
func (p *GLM46Parser) Init(tools []api.Tool, lastMessage *api.Message, thinkValue *api.ThinkValue) []api.Tool {
p.tools = tools
p.callIndex = 0
- p.thinkingEnabled = thinkValue == nil || thinkValue.Bool()
return tools
}
--- a/model/parsers/glm47.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/glm47.go 2026-09-22 20:07:30.000000000 +0400
@@ -12,10 +12,9 @@
func (p *GLM47Parser) Init(tools []api.Tool, lastMessage *api.Message, thinkValue *api.ThinkValue) []api.Tool {
p.tools = tools
p.callIndex = 0
- p.thinkingEnabled = thinkValue == nil || thinkValue.Bool()
// When thinking is enabled (nil or true), the prompt ends with <think>,
// so model output starts directly with thinking content (no opening tag).
- if p.thinkingEnabled {
+ if thinkValue == nil || thinkValue.Bool() {
p.state = glm46ParserState_CollectingThinking
}
return tools
--- a/model/parsers/laguna.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/laguna.go 2026-09-22 20:07:30.000000000 +0400
@@ -50,13 +50,6 @@
return true
}
-func (p *LagunaParser) ThinkingClose() []string {
- if p.state == lagunaParserStateThinking {
- return []string{lagunaThinkingCloseTag}
- }
- return nil
-}
-
func (p *LagunaParser) PreservedTokens() []string {
return []string{
lagunaThinkingOpenTag,
--- a/model/parsers/lfm2.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/lfm2.go 2026-09-22 20:07:30.000000000 +0400
@@ -50,13 +50,6 @@
return p.hasThinkingSupport
}
-func (p *LFM2Parser) ThinkingClose() []string {
- if p.state == LFM2LookingForThinking || p.state == LFM2CollectingThinking {
- return []string{lfm2ThinkingCloseTag}
- }
- return nil
-}
-
func (p *LFM2Parser) PreservedTokens() []string {
return []string{
lfm2ThinkingOpenTag,
--- a/model/parsers/ministral.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/ministral.go 2026-09-22 20:07:30.000000000 +0400
@@ -57,13 +57,6 @@
return p.hasThinkingSupport
}
-func (p *MinistralParser) ThinkingClose() []string {
- if p.state == ministralCollectingThinkingContent {
- return []string{ministralThinkEndTag}
- }
- return nil
-}
-
func (p *MinistralParser) PreservedTokens() []string {
return []string{
ministralToolCallsTag,
--- a/model/parsers/nemotron3nano.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/nemotron3nano.go 2026-09-22 20:07:30.000000000 +0400
@@ -32,13 +32,6 @@
func (p *Nemotron3NanoParser) HasToolSupport() bool { return true }
func (p *Nemotron3NanoParser) HasThinkingSupport() bool { return true }
-func (p *Nemotron3NanoParser) ThinkingClose() []string {
- if p.state == Nemotron3NanoCollectingThinking {
- return []string{nemotronThinkClose}
- }
- return nil
-}
-
func (p *Nemotron3NanoParser) PreservedTokens() []string {
return []string{
nemotronThinkOpen,
--- a/model/parsers/olmo3.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/olmo3.go 2026-09-22 20:07:30.000000000 +0400
@@ -39,10 +39,6 @@
return false
}
-func (p *Olmo3Parser) ThinkingClose() []string {
- return nil
-}
-
func (p *Olmo3Parser) PreservedTokens() []string {
return []string{
olmo3FuncCallsOpenTag,
--- a/model/parsers/olmo3_think.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/olmo3_think.go 2026-09-22 20:07:30.000000000 +0400
@@ -34,13 +34,6 @@
return true
}
-func (p *Olmo3ThinkParser) ThinkingClose() []string {
- if p.state == olmo3CollectingThink {
- return []string{olmo3ThinkCloseTag}
- }
- return nil
-}
-
func (p *Olmo3ThinkParser) PreservedTokens() []string {
return []string{
olmo3ThinkCloseTag,
--- a/model/parsers/parsers.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/parsers.go 2026-09-22 20:07:30.000000000 +0400
@@ -19,11 +19,6 @@
// PreservedTokens returns parser grammar tokens that must remain visible in
// llama-server detokenized output for this parser to recognize boundaries.
PreservedTokens() []string
- // ThinkingClose returns the strings any of which ends the thinking the
- // response begins with, or none when it starts in content. Answered after
- // Init, which decides that from the think value, a prefill, and the
- // parser's default.
- ThinkingClose() []string
HasToolSupport() bool
HasThinkingSupport() bool
}
@@ -125,10 +120,6 @@
return nil
}
-func (p *PassthroughParser) ThinkingClose() []string {
- return nil
-}
-
func (p *PassthroughParser) HasToolSupport() bool {
return false
}
--- a/model/parsers/parsers_test.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/parsers_test.go 2026-09-22 20:07:30.000000000 +0400
@@ -24,10 +24,6 @@
return nil
}
-func (m *mockParser) ThinkingClose() []string {
- return nil
-}
-
func (m *mockParser) HasToolSupport() bool {
return false
}
@@ -58,82 +54,6 @@
}
}
-// TestThinkingClose checks that a parser reports the strings ending its
-// thinking exactly when its response begins inside thinking: not when the
-// request turns thinking off, when an assistant prefill continues content, or
-// when the parser suppresses thinking for tools.
-func TestThinkingClose(t *testing.T) {
- think := func(v bool) *api.ThinkValue { return &api.ThinkValue{Value: v} }
- contentPrefill := &api.Message{Role: "assistant", Content: "The answer"}
- toolResponse := &api.Message{Role: "tool", Content: "42"}
- tool := api.Tool{Type: "function", Function: api.ToolFunction{Name: "get_weather"}}
- thinkTag := []string{"</think>"}
-
- tests := []struct {
- parser string
- think *api.ThinkValue
- lastMessage *api.Message
- tools []api.Tool
- want []string
- }{
- {parser: "passthrough", think: think(true)},
- {parser: "qwen3", think: think(true)},
- {parser: "qwen3-thinking", want: thinkTag},
- {parser: "qwen3-thinking", think: think(false)},
- {parser: "qwen3.5", want: thinkTag},
- {parser: "qwen3.5", think: think(false)},
- {parser: "qwen3.5", lastMessage: contentPrefill},
- {parser: "qwen3-vl-thinking", want: thinkTag},
- {parser: "qwen3-vl-thinking", lastMessage: contentPrefill},
- {parser: "qwen3-vl-instruct", think: think(true)},
- {parser: "deepseek3", think: think(true), want: thinkTag},
- {parser: "deepseek3"},
- {parser: "cogito", think: think(true), want: thinkTag},
- {parser: "cogito", think: think(true), tools: []api.Tool{tool}},
- {parser: "cohere", want: []string{"<|END_THINKING|>"}},
- {parser: "cohere", think: think(false)},
- {parser: "gemma4", think: think(true), want: []string{"<channel|>"}},
- {parser: "gemma4", think: think(true), lastMessage: toolResponse, want: []string{"<channel|>"}},
- {parser: "gemma4", think: think(true), lastMessage: contentPrefill},
- {parser: "gemma4"},
- {parser: "gemma4-no-thinking", think: think(true)},
- {parser: "glm-4.7", want: thinkTag},
- {parser: "glm-4.7", think: think(false)},
- {parser: "glm-ocr"},
- {parser: "lfm2-thinking", think: think(true), want: thinkTag},
- {parser: "lfm2", think: think(true)},
- {parser: "nemotron-3-nano", want: thinkTag},
- {parser: "nemotron-3-nano", think: think(false)},
- {parser: "olmo3-think", want: thinkTag},
- {parser: "olmo3-think", lastMessage: contentPrefill},
- {parser: "olmo3"},
- {parser: "laguna", think: think(true), want: thinkTag},
- {parser: "laguna"},
- {parser: "ministral", think: think(true)},
- {parser: "glimmer", want: []string{"<|start|>assistant to=user<|message|>", "<|start|>assistant<|message|>"}},
- {parser: "glimmer", think: think(false)},
- {parser: "harmony", want: []string{
- "<|end|><|start|>assistant<|channel|>final<|message|>",
- "<|end|><|start|>assistant<|channel|>final <|constrain|>json<|message|>",
- "<|end|><|start|>assistant<|channel|>final<|constrain|>json<|message|>",
- "<|end|><|start|>assistant<|channel|>final json<|message|>",
- "<|end|><|start|>assistant<|channel|>commentary<|message|>",
- "<|end|><|start|>assistant<|message|>",
- }},
- {parser: "harmony", lastMessage: contentPrefill},
- }
-
- for _, tt := range tests {
- t.Run(tt.parser, func(t *testing.T) {
- parser := ParserForName(tt.parser)
- parser.Init(tt.tools, tt.lastMessage, tt.think)
- if got := parser.ThinkingClose(); !slices.Equal(got, tt.want) {
- t.Errorf("ThinkingClose() = %q, want %q", got, tt.want)
- }
- })
- }
-}
-
func TestBuiltInParsersStillWork(t *testing.T) {
tests := []struct {
name string
--- a/model/parsers/qwen35.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/qwen35.go 2026-09-22 20:07:30.000000000 +0400
@@ -45,13 +45,6 @@
return true
}
-func (p *Qwen35Parser) ThinkingClose() []string {
- if p.state == qwen35ParserStateCollectingThinking {
- return []string{qwen35ThinkingCloseTag}
- }
- return nil
-}
-
func (p *Qwen35Parser) PreservedTokens() []string {
return []string{
qwen35ThinkingOpenTag,
--- a/model/parsers/qwen3coder.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/qwen3coder.go 2026-09-22 20:07:30.000000000 +0400
@@ -43,10 +43,6 @@
return false
}
-func (p *Qwen3CoderParser) ThinkingClose() []string {
- return nil
-}
-
func (p *Qwen3CoderParser) PreservedTokens() []string {
return []string{
toolOpenTag,
--- a/model/parsers/qwen3.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/qwen3.go 2026-09-22 20:07:30.000000000 +0400
@@ -52,13 +52,6 @@
return p.hasThinkingSupport
}
-func (p *Qwen3Parser) ThinkingClose() []string {
- if p.state == qwen3ParserStateCollectingThinking {
- return []string{qwen3ThinkingCloseTag}
- }
- return nil
-}
-
func (p *Qwen3Parser) PreservedTokens() []string {
return []string{
qwen3ThinkingOpenTag,
--- a/model/parsers/qwen3vl.go 2026-09-22 22:46:27.469844484 +0400
+++ b/model/parsers/qwen3vl.go 2026-09-22 20:07:30.000000000 +0400
@@ -40,13 +40,6 @@
return p.hasThinkingSupport
}
-func (p *Qwen3VLParser) ThinkingClose() []string {
- if p.state == CollectingThinkingContent {
- return []string{thinkingCloseTag}
- }
- return nil
-}
-
func (p *Qwen3VLParser) PreservedTokens() []string {
return []string{
thinkingCloseTag,
--- a/server/routes_generate_test.go 2026-09-22 22:46:27.473844473 +0400
+++ b/server/routes_generate_test.go 2026-09-22 20:07:30.000000000 +0400
@@ -2646,6 +2646,331 @@
t.Errorf("expected content %q, got %q", "Based on my analysis, the solution is straightforward.", got)
}
})
+
+ earlyFirstPassMetrics := api.Metrics{
+ PromptEvalCount: 4,
+ PromptEvalCachedCount: testIntPtr(1),
+ PromptEvalDuration: 5 * time.Millisecond,
+ EvalCount: 6,
+ EvalDuration: 7 * time.Millisecond,
+ }
+ firstPassMetrics := api.Metrics{
+ PromptEvalCount: 10,
+ PromptEvalCachedCount: testIntPtr(4),
+ PromptEvalDuration: 11 * time.Millisecond,
+ EvalCount: 12,
+ EvalDuration: 13 * time.Millisecond,
+ }
+ secondPassMetrics := api.Metrics{
+ PromptEvalCount: 20_000,
+ PromptEvalCachedCount: testIntPtr(19_000),
+ PromptEvalDuration: 21 * time.Millisecond,
+ EvalCount: 22,
+ EvalDuration: 23 * time.Millisecond,
+ }
+ wantMetrics := api.Metrics{
+ PromptEvalCount: firstPassMetrics.PromptEvalCount,
+ PromptEvalCachedCount: firstPassMetrics.PromptEvalCachedCount,
+ PromptEvalDuration: firstPassMetrics.PromptEvalDuration,
+ EvalCount: firstPassMetrics.EvalCount + secondPassMetrics.EvalCount,
+ EvalDuration: firstPassMetrics.EvalDuration + secondPassMetrics.PromptEvalDuration + secondPassMetrics.EvalDuration,
+ }
+
+ t.Run("structured outputs restart non-stream", func(t *testing.T) {
+ var (
+ requestsMu sync.Mutex
+ requests []llm.CompletionRequest
+ wg sync.WaitGroup
+ )
+
+ wg.Add(2)
+
+ format := json.RawMessage(`{"type":"object","properties":{"answer":{"type":"string"}}}`)
+
+ mock.CompletionFn = func(ctx context.Context, r llm.CompletionRequest, fn func(r llm.CompletionResponse)) error {
+ defer wg.Done()
+
+ requestsMu.Lock()
+ requests = append(requests, r)
+ callNum := len(requests)
+ requestsMu.Unlock()
+
+ switch callNum {
+ case 1:
+ fn(llm.CompletionResponse{
+ Content: " I am thinking through this problem.",
+ Done: false,
+ PromptEvalCount: earlyFirstPassMetrics.PromptEvalCount,
+ PromptEvalCachedCount: earlyFirstPassMetrics.PromptEvalCachedCount,
+ PromptEvalDuration: earlyFirstPassMetrics.PromptEvalDuration,
+ EvalCount: earlyFirstPassMetrics.EvalCount,
+ EvalDuration: earlyFirstPassMetrics.EvalDuration,
+ })
+ fn(llm.CompletionResponse{
+ Content: " </think> {\"answer\":\"42\"}",
+ Done: false,
+ PromptEvalCount: firstPassMetrics.PromptEvalCount,
+ PromptEvalCachedCount: firstPassMetrics.PromptEvalCachedCount,
+ PromptEvalDuration: firstPassMetrics.PromptEvalDuration,
+ EvalCount: firstPassMetrics.EvalCount,
+ EvalDuration: firstPassMetrics.EvalDuration,
+ })
+
+ select {
+ case <-ctx.Done():
+ return ctx.Err()
+ case <-time.After(time.Second):
+ t.Fatalf("timeout waiting for structured outputs cancellation")
+ return nil
+ }
+ case 2:
+ fn(llm.CompletionResponse{
+ Content: `{"answer":"42"}`,
+ Done: true,
+ DoneReason: llm.DoneReasonStop,
+ PromptEvalCount: secondPassMetrics.PromptEvalCount,
+ PromptEvalCachedCount: secondPassMetrics.PromptEvalCachedCount,
+ PromptEvalDuration: secondPassMetrics.PromptEvalDuration,
+ EvalCount: secondPassMetrics.EvalCount,
+ EvalDuration: secondPassMetrics.EvalDuration,
+ })
+ return nil
+ default:
+ t.Fatalf("unexpected number of completion calls: %d", callNum)
+ return nil
+ }
+ }
+
+ think := true
+ streamRequest := false
+ w := createRequest(t, s.ChatHandler, api.ChatRequest{
+ Model: "test-thinking",
+ Messages: []api.Message{{Role: "user", Content: "Please respond in JSON."}},
+ Think: &api.ThinkValue{Value: think},
+ Stream: &streamRequest,
+ Format: format,
+ })
+
+ wg.Wait()
+ mock.CompletionFn = nil
+
+ if w.Code != http.StatusOK {
+ t.Fatalf("expected status 200, got %d", w.Code)
+ }
+
+ if len(requests) != 2 {
+ t.Fatalf("expected two completion calls, got %d", len(requests))
+ }
+
+ if requests[0].Format != nil {
+ t.Errorf("expected first completion format to be nil, got %q", requests[0].Format)
+ }
+ if !requests[0].IncludeIntermediateMetrics {
+ t.Error("expected first completion to request per-token metrics")
+ }
+
+ if !bytes.Equal([]byte(format), []byte(requests[1].Format)) {
+ t.Errorf("expected second completion format to match original format")
+ }
+ if requests[1].IncludeIntermediateMetrics {
+ t.Error("expected second completion to use terminal metrics")
+ }
+
+ var resp api.ChatResponse
+ if err := json.NewDecoder(w.Body).Decode(&resp); err != nil {
+ t.Fatal(err)
+ }
+
+ if resp.Message.Thinking != "I am thinking through this problem. " {
+ t.Errorf("expected thinking %q, got %q", "I am thinking through this problem. ", resp.Message.Thinking)
+ }
+
+ if resp.Message.Content != `{"answer":"42"}` {
+ t.Errorf("expected content %q, got %q", `{"answer":"42"}`, resp.Message.Content)
+ }
+
+ if !resp.Done {
+ t.Errorf("expected response to be done")
+ }
+
+ if resp.DoneReason != "stop" {
+ t.Errorf("expected done reason stop, got %s", resp.DoneReason)
+ }
+ if resp.PromptEvalCount != wantMetrics.PromptEvalCount || resp.EvalCount != wantMetrics.EvalCount {
+ t.Errorf("response counts = (%d, %d), want (%d, %d)", resp.PromptEvalCount, resp.EvalCount, wantMetrics.PromptEvalCount, wantMetrics.EvalCount)
+ }
+ if diff := cmp.Diff(wantMetrics.PromptEvalCachedCount, resp.PromptEvalCachedCount); diff != "" {
+ t.Errorf("response cached prompt count mismatch (-want +got):\n%s", diff)
+ }
+ if resp.PromptEvalDuration != wantMetrics.PromptEvalDuration || resp.EvalDuration != wantMetrics.EvalDuration {
+ t.Errorf("response durations = (%s, %s), want (%s, %s)", resp.PromptEvalDuration, resp.EvalDuration, wantMetrics.PromptEvalDuration, wantMetrics.EvalDuration)
+ }
+ })
+
+ t.Run("structured outputs restart streaming", func(t *testing.T) {
+ var (
+ requestsMu sync.Mutex
+ requests []llm.CompletionRequest
+ wg sync.WaitGroup
+ )
+
+ wg.Add(2)
+
+ format := json.RawMessage(`{"type":"object","properties":{"answer":{"type":"string"}}}`)
+
+ mock.CompletionFn = func(ctx context.Context, r llm.CompletionRequest, fn func(r llm.CompletionResponse)) error {
+ defer wg.Done()
+
+ requestsMu.Lock()
+ requests = append(requests, r)
+ callNum := len(requests)
+ requestsMu.Unlock()
+
+ switch callNum {
+ case 1:
+ fn(llm.CompletionResponse{
+ Content: " I am thinking through this problem.",
+ Done: false,
+ PromptEvalCount: earlyFirstPassMetrics.PromptEvalCount,
+ PromptEvalCachedCount: earlyFirstPassMetrics.PromptEvalCachedCount,
+ PromptEvalDuration: earlyFirstPassMetrics.PromptEvalDuration,
+ EvalCount: earlyFirstPassMetrics.EvalCount,
+ EvalDuration: earlyFirstPassMetrics.EvalDuration,
+ })
+ fn(llm.CompletionResponse{
+ Content: " </think> {\"answer\":\"42\"}",
+ Done: false,
+ PromptEvalCount: firstPassMetrics.PromptEvalCount,
+ PromptEvalCachedCount: firstPassMetrics.PromptEvalCachedCount,
+ PromptEvalDuration: firstPassMetrics.PromptEvalDuration,
+ EvalCount: firstPassMetrics.EvalCount,
+ EvalDuration: firstPassMetrics.EvalDuration,
+ })
+
+ select {
+ case <-ctx.Done():
+ return ctx.Err()
+ case <-time.After(time.Second):
+ t.Fatalf("timeout waiting for structured outputs cancellation")
+ return nil
+ }
+ case 2:
+ fn(llm.CompletionResponse{
+ Content: `{"answer":"42"}`,
+ Done: true,
+ DoneReason: llm.DoneReasonStop,
+ PromptEvalCount: secondPassMetrics.PromptEvalCount,
+ PromptEvalCachedCount: secondPassMetrics.PromptEvalCachedCount,
+ PromptEvalDuration: secondPassMetrics.PromptEvalDuration,
+ EvalCount: secondPassMetrics.EvalCount,
+ EvalDuration: secondPassMetrics.EvalDuration,
+ })
+ return nil
+ default:
+ t.Fatalf("unexpected number of completion calls: %d", callNum)
+ return nil
+ }
+ }
+
+ think := true
+ streamRequest := true
+ w := createRequest(t, s.ChatHandler, api.ChatRequest{
+ Model: "test-thinking",
+ Messages: []api.Message{{Role: "user", Content: "Please respond in JSON."}},
+ Think: &api.ThinkValue{Value: think},
+ Stream: &streamRequest,
+ Format: format,
+ })
+
+ wg.Wait()
+ mock.CompletionFn = nil
+
+ if w.Code != http.StatusOK {
+ t.Fatalf("expected status 200, got %d", w.Code)
+ }
+
+ if len(requests) != 2 {
+ t.Fatalf("expected two completion calls, got %d", len(requests))
+ }
+
+ if requests[0].Format != nil {
+ t.Errorf("expected first completion format to be nil, got %q", requests[0].Format)
+ }
+ if !requests[0].IncludeIntermediateMetrics {
+ t.Error("expected first completion to request per-token metrics")
+ }
+
+ if !bytes.Equal([]byte(format), []byte(requests[1].Format)) {
+ t.Errorf("expected second completion format to match original format")
+ }
+ if requests[1].IncludeIntermediateMetrics {
+ t.Error("expected second completion to use terminal metrics")
+ }
+
+ decoder := json.NewDecoder(w.Body)
+ var events []api.ChatResponse
+ for {
+ var event api.ChatResponse
+ if err := decoder.Decode(&event); err == io.EOF {
+ break
+ } else if err != nil {
+ t.Fatal(err)
+ }
+ events = append(events, event)
+ if event.Done {
+ break
+ }
+ }
+
+ if len(events) < 2 {
+ t.Fatalf("expected at least two streaming events, got %d", len(events))
+ }
+
+ first := events[0]
+ var thinking strings.Builder
+ for _, event := range events {
+ thinking.WriteString(event.Message.Thinking)
+ if !event.Done && (event.PromptEvalCount != 0 || event.PromptEvalCachedCount != nil || event.PromptEvalDuration != 0 || event.EvalCount != 0 || event.EvalDuration != 0) {
+ t.Errorf("non-terminal event unexpectedly exposed metrics: %+v", event.Metrics)
+ }
+ }
+ if got := thinking.String(); got != "I am thinking through this problem. " {
+ t.Errorf("thinking = %q, want %q", got, "I am thinking through this problem. ")
+ }
+
+ if first.Message.Content != "" {
+ t.Errorf("expected first event content to be empty, got %q", first.Message.Content)
+ }
+
+ if first.Done {
+ t.Error("expected first event to be non-terminal")
+ }
+ last := events[len(events)-1]
+ if last.Message.Thinking != "" {
+ t.Errorf("expected final event thinking to be empty, got %q", last.Message.Thinking)
+ }
+
+ if last.Message.Content != `{"answer":"42"}` {
+ t.Errorf("expected final event content %q, got %q", `{"answer":"42"}`, last.Message.Content)
+ }
+
+ if !last.Done {
+ t.Error("expected final event to be done")
+ }
+
+ if last.DoneReason != "stop" {
+ t.Errorf("expected final done reason stop, got %s", last.DoneReason)
+ }
+ if last.PromptEvalCount != wantMetrics.PromptEvalCount || last.EvalCount != wantMetrics.EvalCount {
+ t.Errorf("final counts = (%d, %d), want (%d, %d)", last.PromptEvalCount, last.EvalCount, wantMetrics.PromptEvalCount, wantMetrics.EvalCount)
+ }
+ if diff := cmp.Diff(wantMetrics.PromptEvalCachedCount, last.PromptEvalCachedCount); diff != "" {
+ t.Errorf("final cached prompt count mismatch (-want +got):\n%s", diff)
+ }
+ if last.PromptEvalDuration != wantMetrics.PromptEvalDuration || last.EvalDuration != wantMetrics.EvalDuration {
+ t.Errorf("final durations = (%s, %s), want (%s, %s)", last.PromptEvalDuration, last.EvalDuration, wantMetrics.PromptEvalDuration, wantMetrics.EvalDuration)
+ }
+ })
}
// TestChatFormatWithThinkFalse verifies that when a model uses a builtin
--- a/server/routes.go 2026-09-22 22:46:27.473844473 +0400
+++ b/server/routes.go 2026-09-22 20:07:30.000000000 +0400
@@ -452,6 +452,27 @@
}
}
+ var builtinParser parsers.Parser
+ if shouldUseHarmony(m) {
+ // harmony's Reasoning field only understands low/medium/high; map "max" to "high"
+ if req.Think != nil {
+ if s, ok := req.Think.Value.(string); ok && s == "max" {
+ req.Think.Value = "high"
+ }
+ }
+ if m.Config.Parser == "" {
+ m.Config.Parser = "harmony"
+ }
+ }
+
+ if thinking == nil && !req.Raw && m.Config.Parser != "" {
+ builtinParser = parsers.ParserForName(m.Config.Parser)
+ if builtinParser != nil {
+ // no tools or last message for generate endpoint
+ builtinParser.Init(nil, nil, req.Think)
+ }
+ }
+
caps := []model.Capability{model.CapabilityCompletion}
if req.Suffix != "" {
caps = append(caps, model.CapabilityInsert)
@@ -473,24 +494,10 @@
}
}
- var builtinParser parsers.Parser
- if shouldUseHarmony(m) {
- // harmony's Reasoning field only understands low/medium/high; map "max" to "high"
- if req.Think != nil {
- if s, ok := req.Think.Value.(string); ok && s == "max" {
- req.Think.Value = "high"
- }
- }
- if m.Config.Parser == "" {
- m.Config.Parser = "harmony"
- }
- }
-
- if !req.Raw && m.Config.Parser != "" {
+ if thinking != nil && !req.Raw && m.Config.Parser != "" {
builtinParser = parsers.ParserForName(m.Config.Parser)
if builtinParser != nil {
- // no tools or last message for generate endpoint
- builtinParser.Init(nil, nil, req.Think)
+ builtinParser.Init(nil, nil, think)
}
}
@@ -687,13 +694,6 @@
defer cancel()
var parserErr error
- // A raw prompt gives no way to tell where the response starts, so the
- // format applies from its first token.
- var thinkingClose []string
- if !req.Raw {
- thinkingClose = thinkingCloseForCompletion(builtinParser, thinkTagParser)
- }
-
if err := r.Completion(ctx, llm.CompletionRequest{
Prompt: prompt,
Media: media,
@@ -705,7 +705,6 @@
TopLogprobs: req.TopLogprobs,
PreservedTokens: preservedTokensForCompletion(builtinParser),
LeadingBOS: leadingBOS,
- ThinkingClose: thinkingClose,
}, func(cr llm.CompletionResponse) {
res := api.GenerateResponse{
Model: req.Model,
@@ -2382,16 +2381,6 @@
return toolParser.Tag()
}
-func thinkingCloseForCompletion(builtinParser parsers.Parser, thinkTagParser *thinkingparser.Parser) []string {
- if builtinParser != nil {
- return builtinParser.ThinkingClose()
- }
- if thinkTagParser != nil {
- return []string{thinkTagParser.ClosingTag}
- }
- return nil
-}
-
func leadingBOSForModel(m *Model) string {
if m == nil || m.Config.Renderer == "" {
return ""
@@ -2827,129 +2816,224 @@
toolParser = tools.NewParser(m.Template.Template, req.Tools)
}
+ type structuredOutputsState int
+ const (
+ structuredOutputsState_None structuredOutputsState = iota
+ structuredOutputsState_ReadyToApply
+ structuredOutputsState_Applying
+ )
+
ch := make(chan any)
go func() {
defer close(ch)
- ctx, cancel := context.WithCancel(c.Request.Context())
- defer cancel()
+ structuredOutputsState := structuredOutputsState_None
+ var firstPassMetrics api.Metrics
- var parserErr error
+ for {
+ var tb strings.Builder
- err := r.Completion(ctx, llm.CompletionRequest{
- Prompt: prompt,
- Media: media,
- Format: req.Format,
- Options: opts,
- Shift: req.Shift == nil || *req.Shift,
- Truncate: truncate,
- Logprobs: req.Logprobs,
- TopLogprobs: req.TopLogprobs,
- PreservedTokens: preservedTokensForCompletion(builtinParser),
- ToolCallTag: toolCallTagForCompletion(toolParser),
- LeadingBOS: leadingBOSForModel(m),
- ThinkingClose: thinkingCloseForCompletion(builtinParser, thinkTagParser),
- }, func(r llm.CompletionResponse) {
- res := api.ChatResponse{
- Model: req.Model,
- CreatedAt: time.Now().UTC(),
- Message: api.Message{Role: "assistant", Content: r.Content},
- Done: r.Done,
- Metrics: api.Metrics{
+ currentFormat := req.Format
+ // structured outputs via double request is enabled when:
+ // 1. the model supports the thinking capability and
+ // 2. it uses a built-in parser or our generic thinking parser
+
+ // Note that the current approach does not work for (potential future)
+ // non-thinking models that emit anything before actual content. This
+ // current approach uses the transition from parsed thinking content to
+ // parsed non-thinking content as the signal to turn constraining on
+
+ forceImmediate := builtinParser != nil && builtinParser.HasThinkingSupport() && req.Think != nil && !req.Think.Bool()
+ if req.Format != nil && structuredOutputsState == structuredOutputsState_None && !forceImmediate && ((builtinParser != nil || thinkTagParser != nil) && slices.Contains(m.Capabilities(), model.CapabilityThinking)) {
+ currentFormat = nil
+ }
+ includeIntermediateMetrics := req.Format != nil && currentFormat == nil
+
+ // sets up new context given parent context per request
+ ctx, cancel := context.WithCancel(c.Request.Context())
+
+ var parserErr error
+
+ err := r.Completion(ctx, llm.CompletionRequest{
+ Prompt: prompt,
+ Media: media,
+ Format: currentFormat,
+ Options: opts,
+ Shift: req.Shift == nil || *req.Shift,
+ Truncate: truncate,
+ Logprobs: req.Logprobs,
+ TopLogprobs: req.TopLogprobs,
+ PreservedTokens: preservedTokensForCompletion(builtinParser),
+ ToolCallTag: toolCallTagForCompletion(toolParser),
+ LeadingBOS: leadingBOSForModel(m),
+ IncludeIntermediateMetrics: includeIntermediateMetrics,
+ }, func(r llm.CompletionResponse) {
+ metrics := api.Metrics{
PromptEvalCount: r.PromptEvalCount,
PromptEvalCachedCount: r.PromptEvalCachedCount,
PromptEvalDuration: r.PromptEvalDuration,
EvalCount: r.EvalCount,
EvalDuration: r.EvalDuration,
- },
- Logprobs: toAPILogprobs(r.Logprobs),
- }
+ }
+ if includeIntermediateMetrics {
+ firstPassMetrics = metrics
+ if !r.Done {
+ metrics = api.Metrics{}
+ }
+ } else if structuredOutputsState == structuredOutputsState_Applying && r.Done {
+ // Treat the restart as generation work: retain the original prompt metrics and fold in the second prefill.
+ metrics.PromptEvalCount = firstPassMetrics.PromptEvalCount
+ metrics.PromptEvalCachedCount = firstPassMetrics.PromptEvalCachedCount
+ metrics.PromptEvalDuration = firstPassMetrics.PromptEvalDuration
+ metrics.EvalCount += firstPassMetrics.EvalCount
+ metrics.EvalDuration += firstPassMetrics.EvalDuration + r.PromptEvalDuration
+ }
+
+ res := api.ChatResponse{
+ Model: req.Model,
+ CreatedAt: time.Now().UTC(),
+ Message: api.Message{Role: "assistant", Content: r.Content},
+ Done: r.Done,
+ Metrics: metrics,
+ Logprobs: toAPILogprobs(r.Logprobs),
+ }
+
+ if r.Done {
+ res.DoneReason = r.DoneReason.String()
+ res.TotalDuration = time.Since(checkpointStart)
+ res.LoadDuration = checkpointLoaded.Sub(checkpointStart)
+ }
- if r.Done {
- res.DoneReason = r.DoneReason.String()
- res.TotalDuration = time.Since(checkpointStart)
- res.LoadDuration = checkpointLoaded.Sub(checkpointStart)
- }
+ if builtinParser != nil {
+ slog.Log(context.TODO(), logutil.LevelTrace, "builtin parser input", "parser", m.Config.Parser, "content", r.Content)
- if builtinParser != nil {
- slog.Log(context.TODO(), logutil.LevelTrace, "builtin parser input", "parser", m.Config.Parser, "content", r.Content)
+ content, thinking, toolCalls, err := builtinParser.Add(r.Content, r.Done)
+ if err != nil {
+ parserErr = err
+ cancel()
+ return
+ }
- content, thinking, toolCalls, err := builtinParser.Add(r.Content, r.Done)
- if err != nil {
- parserErr = err
- cancel()
+ res.Message.Content = content
+ res.Message.Thinking = thinking
+ for i := range toolCalls {
+ toolCalls[i].ID = toolCallId()
+ }
+ res.Message.ToolCalls = toolCalls
+
+ tb.WriteString(thinking)
+ // we are now receiving content from the model - we should start applying structured outputs
+ if structuredOutputsState == structuredOutputsState_None && req.Format != nil && tb.String() != "" && res.Message.Content != "" {
+ structuredOutputsState = structuredOutputsState_ReadyToApply
+ cancel()
+ return
+ }
+
+ if res.Message.Content != "" || res.Message.Thinking != "" || len(res.Message.ToolCalls) > 0 || r.Done || len(res.Logprobs) > 0 {
+ slog.Log(context.TODO(), logutil.LevelTrace, "builtin parser output", "parser", m.Config.Parser, "content", content, "thinking", thinking, "toolCalls", toolCalls, "done", r.Done)
+ ch <- res
+ } else {
+ slog.Log(context.TODO(), logutil.LevelTrace, "builtin parser empty output", "parser", m.Config.Parser)
+ }
return
}
- res.Message.Content = content
- res.Message.Thinking = thinking
- for i := range toolCalls {
- toolCalls[i].ID = toolCallId()
+ if thinkTagParser != nil {
+ thinkingContent, remainingContent := thinkTagParser.AddContent(res.Message.Content)
+ if thinkingContent == "" && remainingContent == "" && !r.Done {
+ // need to accumulate more to decide what to send
+ return
+ }
+ res.Message.Thinking = thinkingContent
+ tb.WriteString(thinkingContent)
+ // emit the collected thinking text before restarting with structured outputs and clear unstructured content
+ // to avoid leaking mixed tokens like "</think>Hello"
+ if structuredOutputsState == structuredOutputsState_None && req.Format != nil && tb.String() != "" && remainingContent != "" {
+ structuredOutputsState = structuredOutputsState_ReadyToApply
+ res.Message.Content = ""
+ ch <- res
+ cancel()
+ return
+ }
+ res.Message.Content = remainingContent
}
- res.Message.ToolCalls = toolCalls
- if res.Message.Content != "" || res.Message.Thinking != "" || len(res.Message.ToolCalls) > 0 || r.Done || len(res.Logprobs) > 0 {
- slog.Log(context.TODO(), logutil.LevelTrace, "builtin parser output", "parser", m.Config.Parser, "content", content, "thinking", thinking, "toolCalls", toolCalls, "done", r.Done)
- ch <- res
- } else {
- slog.Log(context.TODO(), logutil.LevelTrace, "builtin parser empty output", "parser", m.Config.Parser)
+ if len(req.Tools) > 0 {
+ toolCalls, content := toolParser.Add(res.Message.Content)
+ if len(content) > 0 {
+ res.Message.Content = content
+ } else if len(toolCalls) > 0 {
+ for i := range toolCalls {
+ toolCalls[i].ID = toolCallId()
+ }
+ res.Message.ToolCalls = toolCalls
+ res.Message.Content = ""
+ } else if res.Message.Thinking != "" {
+ // don't return, fall through to send
+ } else {
+ // Send logprobs while content is being buffered by the parser for tool calls
+ if len(res.Logprobs) > 0 && !r.Done {
+ logprobRes := res
+ logprobRes.Message.Content = ""
+ logprobRes.Message.ToolCalls = nil
+ ch <- logprobRes
+ }
+
+ if r.Done {
+ res.Message.Content = toolParser.Content()
+ ch <- res
+ }
+ return
+ }
}
+
+ ch <- res
+ })
+ if parserErr != nil {
+ ch <- gin.H{"error": parserErr.Error()}
return
}
-
- if thinkTagParser != nil {
- thinkingContent, remainingContent := thinkTagParser.AddContent(res.Message.Content)
- if thinkingContent == "" && remainingContent == "" && !r.Done {
- // need to accumulate more to decide what to send
+ if err != nil {
+ if structuredOutputsState == structuredOutputsState_ReadyToApply && strings.Contains(err.Error(), "context canceled") && c.Request.Context().Err() == nil {
+ // only ignores error if it's a context cancellation due to setting structured outputs
+ } else {
+ s.sched.expireRunnersForRuntimeOOM(m, err)
+ var serr api.StatusError
+ if errors.As(err, &serr) {
+ ch <- gin.H{"error": serr.ErrorMessage, "status": serr.StatusCode}
+ } else {
+ ch <- gin.H{"error": err.Error()}
+ }
return
}
- res.Message.Thinking = thinkingContent
- res.Message.Content = remainingContent
}
- if len(req.Tools) > 0 {
- toolCalls, content := toolParser.Add(res.Message.Content)
- if len(content) > 0 {
- res.Message.Content = content
- } else if len(toolCalls) > 0 {
- for i := range toolCalls {
- toolCalls[i].ID = toolCallId()
- }
- res.Message.ToolCalls = toolCalls
- res.Message.Content = ""
- } else if res.Message.Thinking != "" {
- // don't return, fall through to send
- } else {
- // Send logprobs while content is being buffered by the parser for tool calls
- if len(res.Logprobs) > 0 && !r.Done {
- logprobRes := res
- logprobRes.Message.Content = ""
- logprobRes.Message.ToolCalls = nil
- ch <- logprobRes
- }
+ // ignored structured outputs cancellation falls through to here, start a new request with the structured outputs and updated prompt. use the
+ if structuredOutputsState == structuredOutputsState_ReadyToApply {
+ structuredOutputsState = structuredOutputsState_Applying
+ msg := api.Message{
+ Role: "assistant",
+ Thinking: tb.String(),
+ }
- if r.Done {
- res.Message.Content = toolParser.Content()
- ch <- res
- }
+ msgs = append(msgs, msg)
+ prompt, _, err = chatPrompt(c.Request.Context(), m, r.Tokenize, promptOpts, msgs, processedTools, req.Think, truncate)
+ if err != nil {
+ slog.Error("chat prompt error applying structured outputs", "error", err)
+ ch <- gin.H{"error": err.Error()}
return
}
+ // force constraining by terminating thinking header, the parser is already at this state
+ // when the last message is thinking, the rendered for gpt-oss cannot disambiguate between having the
+ // model continue thinking or ending thinking and outputting the final message.
+ // TODO(parthsareen): consider adding prefill disambiguation logic to the renderer for structured outputs.
+ if shouldUseHarmony(m) || (builtinParser != nil && m.Config.Parser == "harmony") {
+ prompt += "<|end|><|start|>assistant<|channel|>final<|message|>"
+ }
+ continue
}
- ch <- res
- })
- if parserErr != nil {
- ch <- gin.H{"error": parserErr.Error()}
- return
- }
- if err != nil {
- s.sched.expireRunnersForRuntimeOOM(m, err)
- var serr api.StatusError
- if errors.As(err, &serr) {
- ch <- gin.H{"error": serr.ErrorMessage, "status": serr.StatusCode}
- } else {
- ch <- gin.H{"error": err.Error()}
- }
+ break
}
}()