The author seems to be confusing user scalability with performance scaling:
The problem with generative AI, in the industry’s own jargon, is that it does not scale. The cost of growing from, say, a thousand users to a million is a key factor that venture capitalists examine when they evaluate start-ups.
This is a question of whether openai can handle 1 million users asking chatgpt to write a basic html website. That can be scaled horizontally and is just a matter of building more data centers.
The author then goes on to conflate this user scaling with performance scaling:
Yet the returns are diminishing. The bigger an AI model is, the less it improves with each added parameter, and so it must be made bigger at a faster rate just to sustain steady progress. I asked a few AI researchers whether they could name any other real-world software that scales so poorly. None of them could think of any. Even outside the world of software, it’s hard to find a comparable example, given that economy of scale is the principle that has made light bulbs, cars, and clothing so affordable. By economic and engineering measures, generative AI might be the worst technology ever deployed.
This is a question of whether chatgpt can generate a full complex web app. For this there may be a limit to this bigger model approach but this is common to most technologies, performance sometimes has hard limits. You aren’t going to get a car to go 300 mph by making the engine bigger and adding more cylinders, there’s diminishing returns, that doesn’t make cars the worst technology ever deployed… maybe they are but for other reasons.
Economies of scale also isn’t about performance scaling, it’s about capacity scaling. Capacity scaling for AI does reflect economies of scale, that’s why you have these large AI companies building large data centers.
Ok, remove the just then, the point still stands that it is a solvable problem. We know how to make data centers, it may not be easy or cheap but it’s possible just like we know how to build car factories.
Yeah and the point is that model improvements so far have meant making huge increases in size, which offsets the datacenters scale out.
The whole point is that this is futile because we will always be playing catch-up to model sizes, to our ultimate downfall. The tech needs to be smarter not larger. That’s why the whole cloud AI business is shit and not going to work. as anyone with a brain has been saying from the beginning.
Jesus Christ man, people’s homes are being sized with eminent domain for this shit. It ain’t worth it.
This is only true if everyone is always using the top line model, which most people don’t. Both because most people just use the default, which is a low or mid tier model, and because it’s expensive. The top line models are becoming increasingly niche.
The tech needs to be smarter not larger.
I agree, that’s why more focus is being put on the harness and agent orchestration these days. You can achieve better results by having a large model orchestrate a bunch of smaller model agents to do simpler tasks then trying to have the large model one shot it. This doesn’t mean the whole cloud AI business is bullshit, they’re still going to need to build out a lot of capacity for these smaller models and still going to need large models to handle the planning and orchestration, it just means the call count for these larger models are going to be lower.
So it’s probably not going to be 1 million calls to a small model turns into 1 million calls to a larger model and the capacity never catches up, it’s going to be 1 million calls to a small model and 1,000 to a large one which is more feasible to build out.
people’s homes are being sized with eminent domain for this shit. It ain’t worth it.
I don’t agree with how the data centers are being rolled out, they can and should be built out with renewable energy and consent from the community which isn’t happening. I disagree that data centers shouldn’t be built at all or that it will be an unachievable Sisyphusian task to build them out.
I wouldn’t separate performance scalability and user scalability as they ultimately go hand in hand together.
Ok think of them as different scaling factors then, maybe n for number of requests and s for size of requests and c for complexity of requests. Scaling for n can be done horizontally by building more data centers which is possible. Scaling for s or c requires building bigger models which has diminishing returns.
Scaling for n is required to make the software business model work, like the article says. Scaling for s or c though isn’t required as long as your average user keeps those constant, which is possible.
LLMs are inefficient by design.
They are less efficient when compared to what traditional computing can already do, eg. Arithmetic, structured data analysis etc. There are things that traditional computing can’t do, eg. writing an essay, that can only be compared to the human brain which is hard to do. So you can say AI is inefficient at calculating 2 + 2 , but it’s a hard case to make that it’s inefficient at writing an essay.
The author seems to be confusing user scalability with performance scaling:
This is a question of whether openai can handle 1 million users asking chatgpt to write a basic html website. That can be scaled horizontally and is just a matter of building more data centers.
The author then goes on to conflate this user scaling with performance scaling:
This is a question of whether chatgpt can generate a full complex web app. For this there may be a limit to this bigger model approach but this is common to most technologies, performance sometimes has hard limits. You aren’t going to get a car to go 300 mph by making the engine bigger and adding more cylinders, there’s diminishing returns, that doesn’t make cars the worst technology ever deployed… maybe they are but for other reasons.
Economies of scale also isn’t about performance scaling, it’s about capacity scaling. Capacity scaling for AI does reflect economies of scale, that’s why you have these large AI companies building large data centers.
At one of my old jobs “just” was considered a bad word
One does not simply walk into Mordor.
Ok, remove the just then, the point still stands that it is a solvable problem. We know how to make data centers, it may not be easy or cheap but it’s possible just like we know how to build car factories.
Yeah and the point is that model improvements so far have meant making huge increases in size, which offsets the datacenters scale out.
The whole point is that this is futile because we will always be playing catch-up to model sizes, to our ultimate downfall. The tech needs to be smarter not larger. That’s why the whole cloud AI business is shit and not going to work. as anyone with a brain has been saying from the beginning.
Jesus Christ man, people’s homes are being sized with eminent domain for this shit. It ain’t worth it.
This is only true if everyone is always using the top line model, which most people don’t. Both because most people just use the default, which is a low or mid tier model, and because it’s expensive. The top line models are becoming increasingly niche.
I agree, that’s why more focus is being put on the harness and agent orchestration these days. You can achieve better results by having a large model orchestrate a bunch of smaller model agents to do simpler tasks then trying to have the large model one shot it. This doesn’t mean the whole cloud AI business is bullshit, they’re still going to need to build out a lot of capacity for these smaller models and still going to need large models to handle the planning and orchestration, it just means the call count for these larger models are going to be lower.
So it’s probably not going to be 1 million calls to a small model turns into 1 million calls to a larger model and the capacity never catches up, it’s going to be 1 million calls to a small model and 1,000 to a large one which is more feasible to build out.
I don’t agree with how the data centers are being rolled out, they can and should be built out with renewable energy and consent from the community which isn’t happening. I disagree that data centers shouldn’t be built at all or that it will be an unachievable Sisyphusian task to build them out.
The performance per parameter has been improving steadily though. Gemma 4 is ~4o level at a fraction of the parameters.
Bubble sort is also a good algorithm if we “solve” its inefficiency by using more powerful hardware. It may not be easy or cheap, but…
I wouldn’t separate performance scalability and user scalability as they ultimately go hand in hand together. LLMs are inefficient by design.
Ok think of them as different scaling factors then, maybe n for number of requests and s for size of requests and c for complexity of requests. Scaling for n can be done horizontally by building more data centers which is possible. Scaling for s or c requires building bigger models which has diminishing returns.
Scaling for n is required to make the software business model work, like the article says. Scaling for s or c though isn’t required as long as your average user keeps those constant, which is possible.
They are less efficient when compared to what traditional computing can already do, eg. Arithmetic, structured data analysis etc. There are things that traditional computing can’t do, eg. writing an essay, that can only be compared to the human brain which is hard to do. So you can say AI is inefficient at calculating 2 + 2 , but it’s a hard case to make that it’s inefficient at writing an essay.