“It’s a simple question for any enterprise. Would you like something that’s 10 percent of the cost if it’s 90 percent as good as the best, or do you want to pay that 10 times higher cost?”
That really depends…
90% is disasterous. It may sound good because it’s “just” 10% off 100%, but it’s awful. Two iterations reduces that to ~0.01%< but that’s still pretty bad. Obviously the more complex, the higher the risk and the more demand and time per iteration. The machine also has no concept of criticality, so how much is being saved with three or even four iterations to assure failure is basically zero?
“Then the question is: how many tasks require that? I would tell you some do, but the vast majority, summarizing an email, examining a contract, answering a customer call, composing an email, all of those can be done as well with the much cheaper answer.”
Imagine 10,000 emails and hour through your enterprise an hour and 1,000 times It fucks up. Anything from missing info to misinterpreting to just straight up modifying/deleting.
I thought of it as grading as in an academic setting. If the top commercial models are putting out essays that are graded at 100%. The on prem models are putting out results that are graded at 90%. In the case of support emails, the results won’t be much different.
It doesn’t literally mean “90% failure rate vs 100%”
In the HLE benchmark, for example, we may be talking about 45% test score for MiMo Pro or GLM or something, vs 55% for Sonnet and 61% for maxed out Claude.
But further complicating that, those particular open weight models do better than Claude in a hallucination benchmark.
By “90%,” he really means “it’s so close the better model depends on the particular application, workload, harness” and such.
It’s his examples that I’m going off. But maybe he doesn’t know what he’s talking about and an IBM engineer is losing their shit at upper management right now. I used to work for them a long time ago under Virginia and that was standard… Though, so has been every other place since.
I give IBM credit because so many of the people there are ridiculously smart. Not their business people.
Don’t assume the expensive model is 100% accurate. The cheap model is 90% as accurate as the expensive model, which is already less than 100% accurate.
That really depends…
90% is disasterous. It may sound good because it’s “just” 10% off 100%, but it’s awful. Two iterations reduces that to ~0.01%< but that’s still pretty bad. Obviously the more complex, the higher the risk and the more demand and time per iteration. The machine also has no concept of criticality, so how much is being saved with three or even four iterations to assure failure is basically zero?
Imagine 10,000 emails and hour through your enterprise an hour and 1,000 times It fucks up. Anything from missing info to misinterpreting to just straight up modifying/deleting.
I thought of it as grading as in an academic setting. If the top commercial models are putting out essays that are graded at 100%. The on prem models are putting out results that are graded at 90%. In the case of support emails, the results won’t be much different.
It doesn’t literally mean “90% failure rate vs 100%”
In the HLE benchmark, for example, we may be talking about 45% test score for MiMo Pro or GLM or something, vs 55% for Sonnet and 61% for maxed out Claude.
But further complicating that, those particular open weight models do better than Claude in a hallucination benchmark.
By “90%,” he really means “it’s so close the better model depends on the particular application, workload, harness” and such.
No one said 90% failure rate. You brought that up. Reread the post.
You might be lost, Shillsville is 20 miles that way.
You said:
So I interpreted that as an implication of “90% accuracy.”
I’m sorry I misinterpreted you, but I’m not sure why you have to be so abrasive calling that out? “Shillsville?” ???
It’s his examples that I’m going off. But maybe he doesn’t know what he’s talking about and an IBM engineer is losing their shit at upper management right now. I used to work for them a long time ago under Virginia and that was standard… Though, so has been every other place since.
I give IBM credit because so many of the people there are ridiculously smart. Not their business people.
I’m willing to bet that “90%” and “100%” have absolutely no concrete meaning here.
“90% as good for a fraction of the cost” is a set/premade phrase and so the numbers mean nothing, since the person who said it didn’t come up with it.
Like, he extrapolates off of it but I’m willing to bet the number is from the expression.
Don’t assume the expensive model is 100% accurate. The cheap model is 90% as accurate as the expensive model, which is already less than 100% accurate.