Roko AI · ANATOMY中文

LLM inference as machinery

Ameer Ashhab - oss/acc · Original post

Video explanation

Five mechanical scenes visualize tokenization, context limits, output generation, caching and usage metering. Belts and moving parts turn software stages into spatial metaphors for how inputs become tokens and outputs.

Text becomes tokens

Tokens may represent words, word fragments or other symbols, rather than a fixed one-character or one-word unit.

Context has a limit

Input, history and output budget share a model-specific context limit; the mechanical window represents that capacity boundary.

Caching and metering differ

Caching can reuse work while metering tracks consumption. Actual behavior and accounting depend on the provider.

Further reading

Related videos