Localize Qwen27B harness ablation doc

docs/harness-ablation/qwen27b-tight-2x2-model-ablation-20260623.md

@@ -1,37 +1,38 @@
-# Qwen27B Tight-SLO 2x2 Harness Ablation - 2026-06-23
+# Qwen27B tight-SLO 2x2 harness ablation - 2026-06-23
 
-This note organizes the aggregate report generated at:
+本文整理以下 aggregate report，并解释 harness 为什么能够让 tuning 更快、更有效：
 
 ```text
 .aituner-reports/qwen27b-tight-2x2-aggregate-20260623T005838Z/report.md
 ```
 
-The experiment is a 2x2 ablation: model strength crossed with `use_harness`.
+这个实验是一个 2x2 ablation：模型强弱和是否启用 `use_harness` 交叉。
-It asks whether the harness supplies reusable search structure beyond a stronger
+核心问题是：harness 是否提供了可复用的搜索结构，而不仅仅是更强 LLM
-LLM's free-form tuning proposals.
+或者更长 prompt 带来的偶然收益。
 
-## Experiment Design
+## 实验设计
 
-Case: `qwen27b-tight-slo-2x2-aggregate`.
+Case: `qwen27b-tight-slo-2x2-aggregate`。
 
-Substrate:
+实验基座：
 
-- Model served: `qwen3.5-27b-256k-0223-internal`.
+- Served model: `qwen3.5-27b-256k-0223-internal`。
-- Hardware: H20, up to 8 GPUs.
+- Hardware: H20，最多 8 GPUs。
-- Trace: `chat_w20260311_1000`, input length filtered to 0-8192 tokens,
+- Trace: `chat_w20260311_1000`，输入长度过滤到 0-8192 tokens，
-  `replay_time_scale=1.0`, `max_concurrency=32`.
+  `replay_time_scale=1.0`，`max_concurrency=32`。
-- SLO: pass rate >= 0.95, TTFT step rule of 2s for <=4096 input tokens,
+- SLO: pass rate >= 0.95；TTFT step rule 为 <=4096 input tokens 时 2s，
-  4s for <=32768 input tokens, 6s above that, and TPOT <= 50 ms.
+  <=32768 input tokens 时 4s，更长输入时 6s；TPOT <= 50 ms。
-- Search: `sampling_u` in `[0, 0.0625]`, tolerance 0.001, max 6 probes.
+- Search: 在 `sampling_u in [0, 0.0625]` 上二分探测，tolerance 0.001，
-- Tunable envs: `VLLM_ENABLE_TORCH_COMPILE`.
+  max 6 probes。
+- Tunable envs: `VLLM_ENABLE_TORCH_COMPILE`。
 - Tunable flags: `tensor-parallel-size`, `data-parallel-size`,
   `expert-parallel-size`, `gpu-memory-utilization`, `block-size`,
   `max-num-batched-tokens`, `max-num-seqs`, `enable-prefix-caching`,
-  `enable-chunked-prefill`.
+  `enable-chunked-prefill`。
-- Topology constraints: TP and DP in `{1,2,4,8}`, allowed TP*DP products in
+- Topology constraints: TP 和 DP 均在 `{1,2,4,8}` 中，允许的 TP*DP product 为
-  `{1,2,4,8}`, EP fixed to 1 for this case.
+  `{1,2,4,8}`，本 case 中 EP 固定为 1。
 
-Arms:
+2x2 arms:
 
 | Arm | Tuner model | Harness | Trial budget used |
 | --- | --- | --- | ---: |
@@ -40,15 +41,13 @@ Arms:
 | `gpt54mini_harness` | `gpt-5.4-mini` | on | 2 |
 | `gpt54mini_naive` | `gpt-5.4-mini` | off | 10 |
 
-The only intended axis inside each model pair is `use_harness`. The aggregate
+同一个 tuner model 内，主要差异是 `use_harness`。跨模型比较则用来判断：
-then compares whether the weaker model plus harness can match or exceed the
+更弱模型加 harness 是否能匹配或超过更强模型的 naive tuning。
-stronger model without harness.
 
-## Aggregate Result
+## Aggregate result
 
-Reference best: `0.4429 req/s/GPU`.
+Reference best: `0.4429 req/s/GPU`。
-Target threshold for convergence comparisons: 95% of reference, or
+Convergence target: reference 的 95%，即 `0.4208 req/s/GPU`。
-`0.4208 req/s/GPU`.
 
 | Arm | Kind | Trials | Final req/s/GPU | Final/ref | Trials to target | Normalized AUC | Failed | No feasible |
 | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
@@ -57,42 +56,154 @@ Target threshold for convergence comparisons: 95% of reference, or
 | `gpt54mini_harness` | harness | 2 | 0.4429 | 1.0000 | 2 | 0.9450 | 0 | 0 |
 | `gpt54mini_naive` | naive | 10 | 0.0231 | 0.0522 | - | 0.0498 | 1 | 1 |
 
-Harness wins both harness-vs-naive checks:
+Harness-vs-naive 检查全部通过：
 
 | Harness arm | Final vs best naive | AUC vs best naive | Pass |
 | --- | ---: | ---: | --- |
 | `gpt55_harness` | 16.2290x | 16.1296x | true |
 | `gpt54mini_harness` | 16.2290x | 16.0720x | true |
 
-The strongest ablation observation is that `gpt-5.4-mini + harness` matches
+最关键的 ablation 信号是：`gpt-5.4-mini + harness` 和
-`gpt-5.5 + harness` at the same final throughput and the same trials-to-target,
+`gpt-5.5 + harness` 达到同一个 final throughput，也都是 2 trials 达到 target；
-while both naive arms remain more than 16x below the harness arms by final
+而两个 naive arms 用满 10 trials 后仍低于 harness arms 16x 以上。
-per-GPU throughput and AUC.
 
-## What The Harness Actually Did
+## Agent loop 流程图
 
-The harness did not perform generic "better prompting". It inserted a measured,
+下面是当前 harness 化 agent loop 的抽象流程。LLM 仍然可以参与 proposal，
-structured decision protocol between trial results and the next proposal.
+但它拿到的不是裸文本历史，而是结构化 observation、bottleneck diagnosis、
+candidate actions 和 validator 约束；同时 validator 可以授权 stop，也可以阻止
+重复失败或不合法配置。
 
-Formally, after each trial `t`, AITuner observes:
+```mermaid
-
+flowchart TD
-```text
+    A[Study spec: trace, SLO, search range, tunable knobs] --> B[Run one engine config]
-o_t = (config_t, probe history_t, pass-rate_t, latency/SLO failures_t,
+    B --> C[Binary-search probes over sampling_u]
-       request_rate_t, parallel_size_t, launch status_t)
+    C --> D[Build observation o_t]
+    D --> E[Bottleneck classifier]
+    E --> F[Candidate family generator]
+    F --> G[Score candidate actions]
+    G --> H[Prompt renderer / planner]
+    H --> I[LLM or deterministic harness proposal]
+    I --> J{Config validator}
+    J -- invalid, repeated, unsafe --> F
+    J -- valid config_patch --> B
+    G --> K{Stop validator}
+    K -- search_high_saturated_by_incumbent --> L[Stop and keep incumbent]
+    K -- useful candidates remain --> H
 ```
 
-and optimizes:
+这个 loop 中，harness 的作用不是把 prompt 写得更漂亮，而是把 tuning 变成
+一个受测量约束的决策过程：
+
+```text
+measurement -> diagnosis -> candidate family -> scored action -> validated proposal/stop
+```
+
+## 形式化设计：observation
+
+每个 trial 结束后，AITuner 不只记录一段自然语言总结，而是形成结构化 observation：
+
+```text
+o_t = (
+  config_t,
+  probe_history_t,
+  pass_rate_t,
+  latency/SLO_failure_profile_t,
+  request_rate_t,
+  parallel_size_t,
+  launch_status_t,
+  prior_failures_t,
+  incumbent_t
+)
+```
+
+本实验里 observation 中最重要的字段是：
+
+- `config_t`: 当前 trial 的 `flag_patch` 和 `env_patch`，例如 `TP=2, DP=1`。
+- `probe_history_t`: 在不同 `sampling_u` 下二分探测得到的 feasible/infeasible
+  结果。
+- `pass_rate_t`: 是否满足 target pass rate 0.95。
+- `latency/SLO_failure_profile_t`: TTFT 和 TPOT 哪个先触发 SLO pressure。
+- `request_rate_t`: 当前配置在 SLO 下能承载的 request rate。
+- `parallel_size_t`: 该配置实际使用的并行规模，用于归一化 per-GPU objective。
+- `prior_failures_t`: 之前哪些配置 launch failed 或 no feasible，避免重复试错。
+- `incumbent_t`: 当前最优配置及其 `request_rate_per_gpu`。
+
+目标函数是：
 
 ```text
 J(config_t) = request_rate_t / parallel_size_t
-subject to pass_rate_t >= 0.95.
+subject to pass_rate_t >= 0.95
 ```
 
-The harness maps the observation into:
+也就是说，harness 优化的是满足 SLO 后的 `req/s/GPU`，不是 raw throughput，
+也不是 LLM 主观认为“更强”的配置。
+
+## 形式化设计：bottleneck classifier
+
+`bottleneck classifier` 把 observation 映射成 ranked bottleneck hypotheses：
 
 ```text
 b_t = ranked_bottleneck(o_t)
-A_t = candidate_knob_families(b_t, topology_constraints, prior_failures)
+```
+
+它判断的不是“哪个 knob 看起来常用”，而是“当前 SLO failure 和 latency profile
+说明哪个系统环节在限制 objective”。
+
+常见分类包括：
+
+| Bottleneck | 典型证据 | 倾向 knob family |
+| --- | --- | --- |
+| `ttft_prefill` | 长 prompt 下 TTFT 接近或超过 SLO，prefill service time 是瓶颈 | 提高 TP，调整 prefill batching |
+| `decode_tpot` | TPOT p95/p99 超 SLO，decode token latency 是瓶颈 | 调整 `max-num-seqs`，提高 TP，降低 decode contention |
+| `admission_queueing` | waiting/arrival lag 增长，服务时间未必单独变差 | 提高 DP，调整 admission/concurrency knobs |
+| `memory_kv` | KV cache pressure、preemption、OOM、launch failure | 调整 `gpu-memory-utilization`、`block-size`、sequence/token caps |
+| `topology_comm` | TP 增加降低 latency 但 per-GPU efficiency 下降 | 回退 TP，比较 DP/TP tradeoff |
+
+本实验里，两个 harness arms 都把 ranked bottleneck 识别为
+`ttft_prefill`。原因是 workload 有 heavy-tailed long prompts，并且 TTFT SLO 很紧；
+这意味着单个请求的 prefill service time 是主要限制。DP-only 只能增加 replica，
+不能缩短一个长 prompt 的 prefill 路径，因此不是第一优先级。
+
+## 形式化设计：candidate family
+
+`candidate family generator` 根据 bottleneck 和 topology constraints 生成可比较的
+action family：
+
+```text
+A_t = candidate_knob_families(
+  b_t,
+  topology_constraints,
+  prior_failures_t,
+  incumbent_t
+)
+```
+
+在这个 case 中：
+
+- `b_t = ttft_prefill`。
+- 允许的 TP frontier 是 `TP=1 -> TP=2 -> TP=4 -> TP=8`。
+- 允许的 DP frontier 是 `DP=1,2,4,8`，但 DP-only 不直接缓解单请求 prefill
+  latency。
+- EP 固定为 1，因此不探索 expert parallel。
+- 之前没有 failed topology，因此相邻 TP probe launch risk 低。
+
+所以 harness 选择了：
+
+```text
+trial-0001: TP=2, DP=1
+trial-0002: TP=4, DP=1
+```
+
+这不是写死“Qwen27B 应该 TP4”。如果 classifier 输出的是
+`admission_queueing`，candidate family 会更偏向 DP 或 `max-num-seqs`；如果输出是
+`memory_kv`，则会更偏向 memory/cache/sequence knobs。
+
+## 形式化设计：scoring
+
+每个 candidate action 都按同一个抽象打分：
+
+```text
 score(a) = expected_bottleneck_relief(a)
          + information_gain(a)
          + launch_safety(a)
@@ -100,118 +211,122 @@ score(a) = expected_bottleneck_relief(a)
          - measurement_cost(a)
 ```
 
-For this workload, the ranked bottleneck was `ttft_prefill`: long, heavy-tailed
+这些项在本实验里的含义是：
-prompts and a tight TTFT SLO made single-request prefill service time the
-active limiter. Under that bottleneck, the high-value candidate family is a
-legal TP frontier probe, because increasing TP can reduce prefill compute
-latency for one request. DP-only scaling adds replicas but does not shorten the
-single-request prefill path, so it can improve aggregate admission while still
-failing the per-request TTFT bottleneck and the per-GPU objective.
 
-The actual harness trajectory was:
+- `expected_bottleneck_relief`: TP2/TP4 预计能降低 long-prefill compute latency，
+  直接作用于 `ttft_prefill`。
+- `information_gain`: TP frontier probe 可以区分“需要 compute-latency relief”
+  还是“只是 admission/replica 不够”。
+- `launch_safety`: TP2/TP4 均满足 topology constraints，没有重复 failed signature。
+- `regression_risk`: TP 增加会带来通信开销，可能损害 per-GPU efficiency，所以必须用
+  `request_rate_per_gpu` 验证。
+- `measurement_cost`: 每个 GPU trial 成本高；因此高信息量的 topology probe 优先于
+  多个局部 runtime tweak。
 
-| Arm | Trial | Patch | req/s/GPU | Pass rate | Diagnosis |
+实际结果验证了这个 scoring：
+
+| Arm | Trial | Patch | req/s/GPU | Pass rate | 解释 |
 | --- | ---: | --- | ---: | ---: | --- |
-| `gpt55_harness` | 1 | `TP=2, DP=1` | 0.2142 | 0.9572 | TTFT/prefill; adjacent TP increase should reduce long-prefill latency. |
+| `gpt55_harness` | 1 | `TP=2, DP=1` | 0.2142 | 0.9572 | 相邻 TP probe 已满足 SLO，但仍未饱和 search high。 |
-| `gpt55_harness` | 2 | `TP=4, DP=1` | 0.4429 | 0.9718 | Ranked bottleneck is `ttft_prefill`; compare TP4 vs TP2 to distinguish compute-latency relief from replica/admission effects. |
+| `gpt55_harness` | 2 | `TP=4, DP=1` | 0.4429 | 0.9718 | TP frontier 继续缓解 prefill bottleneck，达到 reference best。 |
-| `gpt54mini_harness` | 1 | `TP=2, DP=1` | 0.1992 | 0.9707 | TTFT/prefill; adjacent TP increase is the safest throughput-improving probe. |
+| `gpt54mini_harness` | 1 | `TP=2, DP=1` | 0.1992 | 0.9707 | 弱模型也选择同一机制路径。 |
-| `gpt54mini_harness` | 2 | `TP=4, DP=1` | 0.4429 | 0.9727 | Same `ttft_prefill` topology test as the stronger model. |
+| `gpt54mini_harness` | 2 | `TP=4, DP=1` | 0.4429 | 0.9727 | 弱模型加 harness 匹配强模型加 harness。 |
 
-The stop was also harness-mediated. Both harness arms stopped after trial 2
+## 形式化设计：validator stop
-because the validator authorized `harness_stop` with:
+
+Stop 不是 LLM 自己说“我觉得差不多了”。Stop 必须通过 `stop validator`：
 
 ```text
-search_high_saturated_by_incumbent
+stop(o_t, incumbent_t, search_state_t, candidate_set_t) -> true/false
 ```
 
-The recorded stop diagnosis was:
+本实验里 stop 的记录是：
 
 ```text
-The incumbent's highest measured probe is feasible and is within the configured
+tuning_stop_reason: harness_stop
-binary-search resolution of search.high.
+validator_reason: search_high_saturated_by_incumbent
+diagnosis: The incumbent's highest measured probe is feasible and is within the
+configured binary-search resolution of search.high.
 ```
 
-So the loop did not stop because an LLM guessed that tuning was done. It stopped
+含义是：
-because the incumbent saturated the configured search interval under the SLO
-within binary-search tolerance.
 
-## Which Knobs Were Tuned
+1. 当前 incumbent 的最高测量 probe 已经 feasible。
+2. 该 feasible probe 距离 `search.high` 已经在 binary-search tolerance 内。
+3. 在当前搜索区间和 SLO 约束下，继续花 GPU trial 很难提高 measured objective。
+4. 因此 validator 授权 stop，并保留当前 incumbent。
 
-The winning harness configuration only changed topology:
+这给 harness 带来了 stop discipline：它既不会因为 LLM 过早自信而随便停，也不会在
+已经 saturate search high 后继续 burn budget。
+
+## 实际 tune 了哪些 knobs
+
+Harness winning path 只改了 topology：
 
 ```text
 base config + tensor-parallel-size=4, data-parallel-size=1
 ```
 
-The harness did not tune local scheduler/cache/memory knobs in the winning path.
+它没有在 winning path 中调 scheduler/cache/memory knobs，因为 `ttft_prefill`
-It deliberately tested topology before local runtime knobs because the active
+bottleneck 下，首要动作是缩短单请求 prefill service time。
-bottleneck was single-request TTFT/prefill service time.
 
-The naive arms tuned a different knob family:
+Naive arms 则走了另一个方向：
 
-| Arm | Topology used in all trials | Runtime knobs varied | Best req/s/GPU |
+| Arm | 所有 trials 使用的 topology | 变化过的 runtime knobs | Best req/s/GPU |
 | --- | --- | --- | ---: |
 | `gpt55_naive` | `TP=1, DP=8` | `max-num-batched-tokens`, `max-num-seqs`, `block-size`, `gpu-memory-utilization`, prefix caching, chunked prefill | 0.0273 |
 | `gpt54mini_naive` | `TP=1, DP=8` | `max-num-batched-tokens`, `max-num-seqs`, `block-size`, `gpu-memory-utilization` | 0.0231 |
 
-The first `gpt55_naive` proposal explicitly chose `TP=1, DP=8`, reasoning that
+`gpt55_naive` 的第一个 proposal 明确选择 `TP=1, DP=8`，理由是模型能单卡放下，
-horizontal data parallelism should maximize request rate because the model fits
+因此 horizontal data parallelism 应该最大化 request rate，而 TP 会带来通信开销。
-per GPU and TP would add communication overhead. Subsequent naive proposals kept
+之后 naive proposals 一直保留 DP-heavy topology，只围绕 runtime knobs 搜索。
-that DP-heavy topology and searched scheduler/cache/memory details around it.
+两个 naive arms 合计 20 个 trial slots 都没有进入 TP2/TP4 topology frontier。
-Across 20 naive trial slots total, neither model entered the TP2/TP4 topology
-frontier that solved the bottleneck.
 
-## Why This Beats Baseline
+## 为什么比 baseline 更好
 
-The baseline failed because it optimized the wrong causal path.
+Baseline 失败的原因是优化了错误的因果路径。
 
-For a TTFT/prefill-bound workload, the relevant service-time term is the latency
+对 `ttft_prefill`-bound workload，关键服务时间是单个请求的 prefill latency。
-of one request's prefill path. A DP-heavy topology can run more independent
+DP-heavy topology 可以增加 replica 数，但每个 replica 仍用 TP1 处理长 prompt；
-replicas, but each replica still handles a long prompt with TP1 compute latency.
+它不能显著缩短单请求 prefill path。在 tight TTFT SLO 下，这会导致 feasible
-Under a tight per-request TTFT SLO, those replicas do not unlock a much higher
+`sampling_u` 很低；再除以 GPU 数得到 `req/s/GPU` 后，结果只有
-feasible `sampling_u`, and the objective divides by GPU usage. This is why
+`0.02-0.027 req/s/GPU`。
-`TP=1, DP=8` stayed near `0.02-0.027 req/s/GPU` despite using all GPUs.
 
-The harness changed the optimization direction:
+Harness 的优化路径是：
 
 ```text
-observed SLO pressure -> classify as TTFT/prefill -> prefer legal TP frontier
+observed SLO pressure
--> measure per-GPU feasible rate under the same SLO -> stop when search.high is saturated
+-> classify as ttft_prefill
+-> choose legal TP frontier probe
+-> measure feasible req/s/GPU under the same SLO
+-> stop only when search.high is saturated by incumbent
 ```
 
-That sequence is measurable and falsifiable. If TP4 had improved raw latency but
+这条路径是可测量、可反驳的。如果 TP4 降低了 latency 但
-materially regressed `request_rate_per_gpu`, the harness proposal said it should
+`request_rate_per_gpu` 明显下降，harness 会 reject 这个 hypothesis。如果
-reject the hypothesis. If the bottleneck had been admission/queueing with healthy
+bottleneck 是 admission/queueing 而不是 TTFT/prefill，同一个 knob-effect model
-TTFT/TPOT service times, the same knob-effect model would have favored DP or
+会偏向 DP 或 `max-num-seqs`，而不是 TP frontier。
-`max-num-seqs` instead. The decision was not "Qwen27B needs TP4"; it was
-"`ttft_prefill` evidence makes TP frontier the next highest-information probe
-under current constraints."
 
-This is also why the weak-model arm matters. The weaker `gpt-5.4-mini` with the
+因此，这个结果不是“Qwen27B case 里我们 prompt 诱导模型说 TP4”。更准确的结论是：
-harness converged to exactly the same TP frontier and final throughput as
+harness 用 SLO-derived bottleneck evidence 把搜索导向了正确的 knob family，
-`gpt-5.5 + harness`, while the stronger `gpt-5.5` without harness stayed in the
+再用 per-GPU objective 和 validator stop 验证这个方向。
-wrong DP-heavy family for its whole budget. The ablation therefore attributes the
-gain to the structured harness state and validators, not merely to a stronger
-language model or a more verbose prompt.
 
-## Evidence Boundary
+## 证据边界
 
-This report strongly supports the harness mechanism on the Qwen27B tight-SLO
+这份报告强支撑 Qwen27B tight-SLO case 上的 harness 机制，但不能单独当作通用性证明。
-case and the model-strength ablation. It should not be overclaimed as universal
+当前可成立的结论是：
-proof by itself. The correct generalization claim is narrower:
 
-- In this case, the harness improved final quality, convergence speed, AUC, and
+- 在这个 case 中，harness 同时提升了 final quality、convergence speed、AUC 和
-  stop discipline.
+  stop discipline。
-- The harness made a weaker model match the stronger harnessed model and beat
+- `gpt-5.4-mini + harness` 匹配 `gpt-5.5 + harness`，并显著超过
-  the stronger naive model by more than 16x.
+  `gpt-5.5 + naive`，说明收益主要来自 harness 的结构化状态和 validator，而不是
-- The successful decision was expressed in generic terms: SLO-derived
+  单纯来自更强模型。
-  bottleneck classification, topology constraints, knob-effect scoring,
+- 成功路径用的是通用机制：SLO-derived bottleneck classification、topology
-  per-GPU objective, and validator-authorized stop.
+  constraints、knob-effect scoring、per-GPU objective、validator-authorized stop。
-- Additional cases are still needed to show the same mechanism across different
+- 还需要在其他 bottleneck/case 上继续验证，例如 prefill scheduler pressure、
-  bottlenecks, for example prefill scheduler pressure, decode TPOT pressure,
+  decode TPOT pressure、memory/KV pressure、admission/queueing pressure。
-  memory/KV pressure, and admission/queueing pressure.
 
-## Original Aggregate Report
+## 原始 aggregate report 摘录
 
 ```text
 # qwen27b-tight-2x2-aggregate-20260623T005838Z