Architecture
Audited base
SmolLM3-3B is a decoder-only Transformer published by the Hugging Face SmolLM team.
| Component | Value |
|---|---|
| Parameters | 3,075,098,624 |
| Layers | 36 |
| Hidden size | 2,048 |
| MLP intermediate | 11,008 |
| Query / KV heads | 16 / 4 |
| Vocabulary | 128,256 |
| Trained positions | 65,536 |
| Weight tying | input embedding = language-model head |
| Position design | NoPE 3:1 with periodic RoPE |
| Weight format | two BF16 SafeTensors shards |
The exact parameter count comes from all 326 tensor headers at the pinned revision. See the base audit.
First TideLM adapter
The planned reference adapter adds rank-8 low-rank matrices to q_proj and v_proj in layer 35, with alpha 16 and no dropout. Only those adapter parameters receive gradients. The base weights stay immutable and users need the exact pinned base to load the adapter.
frozen layers 0 … 34
↓
layer 35 attention: q_proj + LoRA, v_proj + LoRA
↓
frozen MLP / final norm / tied output headTargeting one layer is a compute compromise, not a new architecture claim. The runtime records the actual trainable count after PEFT injection.
Identity
An adapter plus its base revision, tokenizer, PEFT config, generation config, and checksums defines a TideLM release. Renaming the base or changing metadata without gradients does not.