I want to point out that LLMs can be completely deterministic if the final sampler is run with 0 temperature (picking the highest probability token), no top-k, fixed seed, etc.
Highest probability token can still vary nondeterministically when the computation is essentially racing GPU cores or even separate hosts against each other. Float math evaluation order can change the end result.
