Roko AI · ANATOMY中文

How a language model generates step by step

Samrat Chowdhury · Original post

Video explanation

Using 17×24, a ring holds prompt, intermediate steps and output while a central layer stack computes next-token probabilities. An exploded view closes the demonstration.

Context to next token

The model uses context to produce a probability distribution for the next token, then selects a token to continue.

Attention links information

Attention relates sequence information as layers transform representations. The tower and lines are a metaphor, not hardware layout.

Intermediate steps and answer

The demo distinguishes intermediate steps from the answer. More steps may help computation, without proving all models reason through the same sequence.

Further reading

Related videos