There are indeed some Bitnet LUT implementations, already.
It’s both smaller and faster though; the models are drastically smaller than FP16 at the same parameter count, as they’re just 1.5 bits instead of 16.
How much is lost from the precision drop is an open question, though, and the answer seems to be complex and finicky. I think it’s why big labs are hesitant to pursue it; they want to try safer, lower hanging fruit first. Also, it’s not particularly efficient on hardware (like AI GPUs) where most of the silicon is dedicated to matrix multiply performance anyway.
Thank you! Ah yes that makes sense somewhat I guess. Trading cpu usage for memory usage, an old classic 😁!
Nibble * nibble look up tables could probably be fast too, it all depends on where you want the complexity I guess.
There are indeed some Bitnet LUT implementations, already.
It’s both smaller and faster though; the models are drastically smaller than FP16 at the same parameter count, as they’re just 1.5 bits instead of 16.
How much is lost from the precision drop is an open question, though, and the answer seems to be complex and finicky. I think it’s why big labs are hesitant to pursue it; they want to try safer, lower hanging fruit first. Also, it’s not particularly efficient on hardware (like AI GPUs) where most of the silicon is dedicated to matrix multiply performance anyway.