xserv-chat: harmony channel routing + repetition penalty for gpt-oss

- Let the model generate its own <|channel|> routing instead of forcing
  <|channel|>final<|message|> — matches the GGUF chat template behavior.
- State machine tracks harmony channels: analysis channel rendered gray,
  final channel printed normally, <|end|> stops on final channel only.
- Add repetition penalty (default 1.3 for MoE, 1.0 for Qwen) with 512
  token window to prevent greedy decode loops. Configurable via
  XSERV_REP_PENALTY and XSERV_REP_WINDOW env vars.
- Fix Length path: use <|end|> instead of <|im_end|> for gpt-oss to
  avoid poisoning the KV cache with garbage tokens on truncation.
- Server api.rs: append <|channel|>final<|message|> to the hardcoded
  gpt-oss prompt (server expects to post-process the JSON output).
- Add startswith filter to minijinja for harmony template compatibility.

Known issue: gpt-oss multi-turn NaN when total context exceeds ~256
tokens — likely a flash_attention_sinks kernel bug with sliding window
layers at large kv_len + small q_len. Single-turn and short multi-turn
conversations work correctly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-02 12:40:17 +08:00
parent 3ee8df2c0f
commit f2e60218b4
2 changed files with 92 additions and 7 deletions

View File

@@ -114,6 +114,11 @@ impl ChatTemplate {
env.add_function("strftime_now", strftime_now);
env.add_function("raise_exception", raise_exception);
// Python str methods used by harmony/gpt-oss templates.
env.add_filter("startswith", |s: String, prefix: String| -> bool {
s.starts_with(&prefix)
});
env.add_template("chat", &self.source)?;
let tmpl = env.get_template("chat")?;