phase 6+7+8: model loading, BPE tokenizer, GPT-2 inference (Milestone ①)

Phase 6 — Model Loading (xserv-model):
- safetensors parser with single/sharded file support
- ModelConfig with dual naming (GPT-2 n_embd/n_head + modern HF naming)
- Weight loading flow: safetensors → mmap → CPU Tensor → GPU

Phase 7 — BPE Tokenizer (xserv-tokenizer):
- Full BPE encode/decode from tokenizer.json
- GPT-2 byte-to-unicode mapping (printable ASCII identity + shifted bytes)
- Pre-tokenization regex, special token handling
- Chat template support structure

Phase 8 — GPT-2 Complete Inference:
- GPT-2 model definition: wte, wpe, 12 transformer blocks, ln_f
- Forward pass: embedding → (LayerNorm → MHA → residual → LayerNorm → MLP → residual) × 12 → LN → logits
- QKV split with correct [batch, heads, seq, dim] layout (fixed reshape bug)
- Greedy sampling from last-position logits
- Interactive CLI: xserv-cli <model-dir> [--max-tokens N]

Verified: GPT-2 124M generates coherent English text on RTX 5090.
"The future of AI is uncertain. The future of AI is uncertain..."
"Once upon a time, the world was a place of great beauty..."

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-21 22:04:00 +08:00
parent 6035ffdc0b
commit e1e75fc7f6
13 changed files with 971 additions and 0 deletions

View File

@@ -0,0 +1,96 @@
use serde::Deserialize;
use std::path::Path;
#[derive(Debug, Clone, Deserialize)]
pub struct ModelConfig {
pub architectures: Option<Vec<String>>,
pub model_type: Option<String>,
// Modern HF naming
#[serde(default)]
pub hidden_size: Option<usize>,
#[serde(default)]
pub intermediate_size: Option<usize>,
#[serde(default)]
pub num_attention_heads: Option<usize>,
#[serde(default)]
pub num_key_value_heads: Option<usize>,
#[serde(default)]
pub num_hidden_layers: Option<usize>,
pub vocab_size: usize,
#[serde(default)]
pub max_position_embeddings: Option<usize>,
// GPT-2 naming
#[serde(default)]
pub n_embd: Option<usize>,
#[serde(default)]
pub n_head: Option<usize>,
#[serde(default)]
pub n_layer: Option<usize>,
#[serde(default)]
pub n_positions: Option<usize>,
#[serde(default)]
pub n_inner: Option<usize>,
// Normalization
#[serde(default)]
pub layer_norm_eps: Option<f64>,
#[serde(default)]
pub layer_norm_epsilon: Option<f64>,
#[serde(default)]
pub rms_norm_eps: Option<f64>,
// Other
#[serde(default)]
pub rope_theta: Option<f64>,
#[serde(default)]
pub tie_word_embeddings: Option<bool>,
}
impl ModelConfig {
pub fn from_file(path: &Path) -> Self {
let data = std::fs::read_to_string(path)
.unwrap_or_else(|e| panic!("failed to read {}: {e}", path.display()));
serde_json::from_str(&data)
.unwrap_or_else(|e| panic!("failed to parse {}: {e}", path.display()))
}
pub fn hidden(&self) -> usize {
self.hidden_size.or(self.n_embd).expect("hidden_size or n_embd required")
}
pub fn num_heads(&self) -> usize {
self.num_attention_heads.or(self.n_head).expect("num_attention_heads or n_head required")
}
pub fn num_layers(&self) -> usize {
self.num_hidden_layers.or(self.n_layer).expect("num_hidden_layers or n_layer required")
}
pub fn max_seq_len(&self) -> usize {
self.max_position_embeddings.or(self.n_positions).unwrap_or(2048)
}
pub fn ffn_hidden(&self) -> usize {
self.intermediate_size.or(self.n_inner).unwrap_or(self.hidden() * 4)
}
pub fn num_kv_heads(&self) -> usize {
self.num_key_value_heads.unwrap_or(self.num_heads())
}
pub fn head_dim(&self) -> usize {
self.hidden() / self.num_heads()
}
pub fn ln_eps(&self) -> f32 {
self.layer_norm_eps
.or(self.layer_norm_epsilon)
.unwrap_or(1e-5) as f32
}
pub fn tied_embeddings(&self) -> bool {
self.tie_word_embeddings.unwrap_or(true)
}
}