32 lines
7.2 KiB
Markdown
32 lines
7.2 KiB
Markdown
两阶段
|
|
1. 使用 Rust framework 实现 vLLM replay 进行正确推理，提取出 compute flow
|
|
2. 编排 compute flow
|
|
|
|
|
|
|
|
### Rust Framework
|
|
|
|
|
|
### Compute Flow Arrangement
|
|
|
|
|
|
|
|
### Challenges
|
|
|
|
- vllm 为不同模型会选择不同算子、同一算子面临不同的输入 batch 也会采用不同的 kernel 现实，在哪个 level 做抽象？
|
|
- 如果我们希望自动化搜索到最优的 serve 部署方案，直接 trace vllm 的算子流有什么道理是最优的？
|
|
|
|
|
|
|
|
283: 6.276874845s, 0.313328s
|
|
544: 6.322570442s, 0.314596s
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
```
|
|
odict_keys(['transformer.wte.weight', 'transformer.h.0.ln_1.weight', 'transformer.h.0.attn.c_attn.weight', 'transformer.h.0.attn.c_attn.bias', 'transformer.h.0.attn.c_proj.weight', 'transformer.h.0.ln_2.weight', 'transformer.h.0.mlp.w1.weight', 'transformer.h.0.mlp.w2.weight', 'transformer.h.0.mlp.c_proj.weight', 'transformer.h.1.ln_1.weight', 'transformer.h.1.attn.c_attn.weight', 'transformer.h.1.attn.c_attn.bias', 'transformer.h.1.attn.c_proj.weight', 'transformer.h.1.ln_2.weight', 'transformer.h.1.mlp.w1.weight', 'transformer.h.1.mlp.w2.weight', 'transformer.h.1.mlp.c_proj.weight', 'transformer.h.2.ln_1.weight', 'transformer.h.2.attn.c_attn.weight', 'transformer.h.2.attn.c_attn.bias', 'transformer.h.2.attn.c_proj.weight', 'transformer.h.2.ln_2.weight', 'transformer.h.2.mlp.w1.weight', 'transformer.h.2.mlp.w2.weight', 'transformer.h.2.mlp.c_proj.weight', 'transformer.h.3.ln_1.weight', 'transformer.h.3.attn.c_attn.weight', 'transformer.h.3.attn.c_attn.bias', 'transformer.h.3.attn.c_proj.weight', 'transformer.h.3.ln_2.weight', 'transformer.h.3.mlp.w1.weight', 'transformer.h.3.mlp.w2.weight', 'transformer.h.3.mlp.c_proj.weight', 'transformer.h.4.ln_1.weight', 'transformer.h.4.attn.c_attn.weight', 'transformer.h.4.attn.c_attn.bias', 'transformer.h.4.attn.c_proj.weight', 'transformer.h.4.ln_2.weight', 'transformer.h.4.mlp.w1.weight', 'transformer.h.4.mlp.w2.weight', 'transformer.h.4.mlp.c_proj.weight', 'transformer.h.5.ln_1.weight', 'transformer.h.5.attn.c_attn.weight', 'transformer.h.5.attn.c_attn.bias', 'transformer.h.5.attn.c_proj.weight', 'transformer.h.5.ln_2.weight', 'transformer.h.5.mlp.w1.weight', 'transformer.h.5.mlp.w2.weight', 'transformer.h.5.mlp.c_proj.weight', 'transformer.h.6.ln_1.weight', 'transformer.h.6.attn.c_attn.weight', 'transformer.h.6.attn.c_attn.bias', 'transformer.h.6.attn.c_proj.weight', 'transformer.h.6.ln_2.weight', 'transformer.h.6.mlp.w1.weight', 'transformer.h.6.mlp.w2.weight', 'transformer.h.6.mlp.c_proj.weight', 'transformer.h.7.ln_1.weight', 'transformer.h.7.attn.c_attn.weight', 'transformer.h.7.attn.c_attn.bias', 'transformer.h.7.attn.c_proj.weight', 'transformer.h.7.ln_2.weight', 'transformer.h.7.mlp.w1.weight', 'transformer.h.7.mlp.w2.weight', 'transformer.h.7.mlp.c_proj.weight', 'transformer.h.8.ln_1.weight', 'transformer.h.8.attn.c_attn.weight', 'transformer.h.8.attn.c_attn.bias', 'transformer.h.8.attn.c_proj.weight', 'transformer.h.8.ln_2.weight', 'transformer.h.8.mlp.w1.weight', 'transformer.h.8.mlp.w2.weight', 'transformer.h.8.mlp.c_proj.weight', 'transformer.h.9.ln_1.weight', 'transformer.h.9.attn.c_attn.weight', 'transformer.h.9.attn.c_attn.bias', 'transformer.h.9.attn.c_proj.weight', 'transformer.h.9.ln_2.weight', 'transformer.h.9.mlp.w1.weight', 'transformer.h.9.mlp.w2.weight', 'transformer.h.9.mlp.c_proj.weight', 'transformer.h.10.ln_1.weight', 'transformer.h.10.attn.c_attn.weight', 'transformer.h.10.attn.c_attn.bias', 'transformer.h.10.attn.c_proj.weight', 'transformer.h.10.ln_2.weight', 'transformer.h.10.mlp.w1.weight', 'transformer.h.10.mlp.w2.weight', 'transformer.h.10.mlp.c_proj.weight', 'transformer.h.11.ln_1.weight', 'transformer.h.11.attn.c_attn.weight', 'transformer.h.11.attn.c_attn.bias', 'transformer.h.11.attn.c_proj.weight', 'transformer.h.11.ln_2.weight', 'transformer.h.11.mlp.w1.weight', 'transformer.h.11.mlp.w2.weight', 'transformer.h.11.mlp.c_proj.weight', 'transformer.h.12.ln_1.weight', 'transformer.h.12.attn.c_attn.weight', 'transformer.h.12.attn.c_attn.bias', 'transformer.h.12.attn.c_proj.weight', 'transformer.h.12.ln_2.weight', 'transformer.h.12.mlp.w1.weight', 'transformer.h.12.mlp.w2.weight', 'transformer.h.12.mlp.c_proj.weight', 'transformer.h.13.ln_1.weight', 'transformer.h.13.attn.c_attn.weight', 'transformer.h.13.attn.c_attn.bias', 'transformer.h.13.attn.c_proj.weight', 'transformer.h.13.ln_2.weight', 'transformer.h.13.mlp.w1.weight', 'transformer.h.13.mlp.w2.weight', 'transformer.h.13.mlp.c_proj.weight', 'transformer.h.14.ln_1.weight', 'transformer.h.14.attn.c_attn.weight', 'transformer.h.14.attn.c_attn.bias', 'transformer.h.14.attn.c_proj.weight', 'transformer.h.14.ln_2.weight', 'transformer.h.14.mlp.w1.weight', 'transformer.h.14.mlp.w2.weight', 'transformer.h.14.mlp.c_proj.weight', 'transformer.h.15.ln_1.weight', 'transformer.h.15.attn.c_attn.weight', 'transformer.h.15.attn.c_attn.bias', 'transformer.h.15.attn.c_proj.weight', 'transformer.h.15.ln_2.weight', 'transformer.h.15.mlp.w1.weight', 'transformer.h.15.mlp.w2.weight', 'transformer.h.15.mlp.c_proj.weight', 'transformer.h.16.ln_1.weight', 'transformer.h.16.attn.c_attn.weight', 'transformer.h.16.attn.c_attn.bias', 'transformer.h.16.attn.c_proj.weight', 'transformer.h.16.ln_2.weight', 'transformer.h.16.mlp.w1.weight', 'transformer.h.16.mlp.w2.weight', 'transformer.h.16.mlp.c_proj.weight', 'transformer.h.17.ln_1.weight', 'transformer.h.17.attn.c_attn.weight', 'transformer.h.17.attn.c_attn.bias', 'transformer.h.17.attn.c_proj.weight', 'transformer.h.17.ln_2.weight', 'transformer.h.17.mlp.w1.weight', 'transformer.h.17.mlp.w2.weight', 'transformer.h.17.mlp.c_proj.weight', 'transformer.h.18.ln_1.weight', 'transformer.h.18.attn.c_attn.weight', 'transformer.h.18.attn.c_attn.bias', 'transformer.h.18.attn.c_proj.weight', 'transformer.h.18.ln_2.weight', 'transformer.h.18.mlp.w1.weight', 'transformer.h.18.mlp.w2.weight', 'transformer.h.18.mlp.c_proj.weight', 'transformer.h.19.ln_1.weight', 'transformer.h.19.attn.c_attn.weight', 'transformer.h.19.attn.c_attn.bias', 'transformer.h.19.attn.c_proj.weight', 'transformer.h.19.ln_2.weight', 'transformer.h.19.mlp.w1.weight', 'transformer.h.19.mlp.w2.weight', 'transformer.h.19.mlp.c_proj.weight', 'transformer.h.20.ln_1.weight', 'transformer.h.20.attn.c_attn.weight', 'transformer.h.20.attn.c_attn.bias', 'transformer.h.20.attn.c_proj.weight', 'transformer.h.20.ln_2.weight', 'transformer.h.20.mlp.w1.weight', 'transformer.h.20.mlp.w2.weight', 'transformer.h.20.mlp.c_proj.weight', 'transformer.h.21.ln_1.weight', 'transformer.h.21.attn.c_attn.weight', 'transformer.h.21.attn.c_attn.bias', 'transformer.h.21.attn.c_proj.weight', 'transformer.h.21.ln_2.weight', 'transformer.h.21.mlp.w1.weight', 'transformer.h.21.mlp.w2.weight', 'transformer.h.21.mlp.c_proj.weight', 'transformer.h.22.ln_1.weight', 'transformer.h.22.attn.c_attn.weight', 'transformer.h.22.attn.c_attn.bias', 'transformer.h.22.attn.c_proj.weight', 'transformer.h.22.ln_2.weight', 'transformer.h.22.mlp.w1.weight', 'transformer.h.22.mlp.w2.weight', 'transformer.h.22.mlp.c_proj.weight', 'transformer.h.23.ln_1.weight', 'transformer.h.23.attn.c_attn.weight', 'transformer.h.23.attn.c_attn.bias', 'transformer.h.23.attn.c_proj.weight', 'transformer.h.23.ln_2.weight', 'transformer.h.23.mlp.w1.weight', 'transformer.h.23.mlp.w2.weight', 'transformer.h.23.mlp.c_proj.weight', 'transformer.ln_f.weight', 'lm_head.weight'])
|
|
|
|
``` |