Cartesia Research published results for an updated hierarchical network, or H-Net, on Oct. 28, 2026. The company reports that its two-stage byte-level model reached SmolLM3's final average across six benchmark tasks after roughly 1 trillion training tokens, compared with roughly 11 trillion for SmolLM3. That is a comparison of averages, not a finding that the models scored equally on every task.
Most language models use a tokenizer to split text into a fixed vocabulary of chunks before training. Cartesia's H-Net instead learns to group bytes inside the model. Cartesia shows examples in which the model groups an unfamiliar word into fewer chunks as it recurs in an input.
What the comparison does and doesn't match
Cartesia says the models' training compute per byte was matched to within 2%, making the saving in training compute proportional to the shorter run. But its H-Net had 3.67 billion parameters, compared with SmolLM3's 3.08 billion. Cartesia used a best-effort reproduction of SmolLM3's first-stage data mixture; SmolLM3 trained in three phases. The company cautions that matching both data and compute across the architectures is not possible.
In a separate, more tightly controlled scaling experiment, Cartesia compared one-stage H-Nets with tokenized Transformers trained on identical text and using matched backbone architectures. It reports 1.28 times better compute efficiency on in-domain validation loss, roughly twice the efficiency on downstream evaluations and up to 5.37 times on out-of-distribution Prolog. Those figures belong to the scaling experiment, not the SmolLM3 run. Cartesia says it plans to release the improved H-Net architecture.
Comments