Torvalds: "100% of Their Code Is Written by Compilers" Too. The Linux creator argued that AI-written code claims are misleading — and warned that a wave of low-quality, AI-generated bug reports is quietly burning out the maintainers who keep open source running.
C/C++ compilers are non deterministic due to support of super macros that change run to run, non-deterministic optimisation strategies or ordering due to parallelism, and linkers often produce different outputs every time they are run where subtle bugs can cause crashes when addresses don’t line up how you expect. And that’s without mentioning projects that use a configuration step.
Can’t most serious compilers produce reproducible builds these days given the same build environment. I know there has been a drive towards reproducible builds in general for security verification purposes.
I’m not very good with C/C++ so please correct me, isn’t that what’s called a “race condition”? Parallelism can cause non-determinism but not in the same sense LLMs generate non-deterministic output. Compilers are not statistical machines.
You don’t need parallelism to have a race condition, just not handling an event with expected timing can cause one - like when two keys are pressed within one polling cycle and you depend on one being pressed before the other for some logic like up and right arrow for a diagonal but they register as right and up so the diagonal movement doesn’t trigger
Compiler optimisation strategies sometimes use statistical machines and link time optimisation does use random number generators for producing output
An LLM is not inherently non-deterministic though - if you don’t randomly sample and instead have a fixed rule (which is what the recent fingerprint embedding approach does), if applied in all cases the output is deterministic, as the neural net at its core is deterministic function. A lot of the randomness beyond that is due to optimizations [source].
LLMs are however unreliable at ‘compiling’. Whether or not it will be able to complete the requested task (translate human language into code) correctly it not guaranteed - at least nowhere near the compilers we use.
If you do that it will basically produce the most average possible output, given the context. I don’t know if that’s going to be useful in a compiler context…
Exactly this. If you made an LLM that had deterministic code output then I’m all up for saying “AI is a compiler for human language”. But until then AI is most definitely not a compiler.
Never seen a non-deterministic compiler though
C/C++ compilers are non deterministic due to support of super macros that change run to run, non-deterministic optimisation strategies or ordering due to parallelism, and linkers often produce different outputs every time they are run where subtle bugs can cause crashes when addresses don’t line up how you expect. And that’s without mentioning projects that use a configuration step.
Can’t most serious compilers produce reproducible builds these days given the same build environment. I know there has been a drive towards reproducible builds in general for security verification purposes.
I’m not very good with C/C++ so please correct me, isn’t that what’s called a “race condition”? Parallelism can cause non-determinism but not in the same sense LLMs generate non-deterministic output. Compilers are not statistical machines.
You don’t need parallelism to have a race condition, just not handling an event with expected timing can cause one - like when two keys are pressed within one polling cycle and you depend on one being pressed before the other for some logic like up and right arrow for a diagonal but they register as right and up so the diagonal movement doesn’t trigger
Compiler optimisation strategies sometimes use statistical machines and link time optimisation does use random number generators for producing output
An LLM is not inherently non-deterministic though - if you don’t randomly sample and instead have a fixed rule (which is what the recent fingerprint embedding approach does), if applied in all cases the output is deterministic, as the neural net at its core is deterministic function. A lot of the randomness beyond that is due to optimizations [source].
LLMs are however unreliable at ‘compiling’. Whether or not it will be able to complete the requested task (translate human language into code) correctly it not guaranteed - at least nowhere near the compilers we use.
There is a parameter in llms called temperature. If you reduce it down to zero it will become deterministic. And probably even worse.
If you do that it will basically produce the most average possible output, given the context. I don’t know if that’s going to be useful in a compiler context…
Exactly this. If you made an LLM that had deterministic code output then I’m all up for saying “AI is a compiler for human language”. But until then AI is most definitely not a compiler.
Setting the temperature parameter to 0.00 makes an LLM deterministic.