Gahow Wang e11f15e009 tokenizer: support multiple end-of-generation tokens ...

Track an ordered eos_token_ids list (not just one id) and add is_eos().
gpt-oss/harmony ends the assistant turn on <|return|> and also treats
<|call|> and <|endoftext|> as terminators (generation_config.json
eos_token_id = [200002, 199999, 200012]); single-eos families are
unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-05-31 00:56:21 +08:00

..

src

tokenizer: support multiple end-of-generation tokens

2026-05-31 00:56:21 +08:00

Cargo.toml

phase 6+7+8: model loading, BPE tokenizer, GPT-2 inference (Milestone ①)

2026-05-21 22:04:00 +08:00