Elon Musk Clamors That We’re Running Out Of Data For Advancing AI LLMs But Let’s Not Overlook Squeezing Out More Juice From Data That We Already Have

In today’s column, I examine a high-profile assertion that advancements in generative AI and large language models (LLMs) will be hampered or possibly curtailed by our soon-to-be dilemma of running out of available data. Elon Musk for example made such a statement recently and many other AI luminaries have repeatedly and loudly raised that very same concern.

Will all AI improvements come to a grinding halt? Are we up a creek without a paddle? Does the grandiose aspiration for artificial general intelligence (AGI) and artificial superintelligence (ASI) fall apart such that the modern-day AI we’ve got in hand today will be the best AI that there will ever be?

Let’s talk about it.

This analysis of an innovative AI breakthrough is part of my ongoing Forbes column coverage on the latest in AI including identifying and explaining various impactful AI complexities (see the link here).

The Vocal Claim About Peak Data

If you perchance follow any of the major AI makers or AI commentators, you would undoubtedly have seen or heard an increasingly panicky remark. It goes like this.

First, to craft generative AI and LLMs, you need to vastly pattern-match on gobs and gobs of data, typically done by scanning the Internet. All kinds of essays, narratives, stories, encyclopedias, poems, and the like are scrutinized in a mathematical and computational fashion to ferret out the nature of human writing. Based on that data training, modern-era AI does a relatively remarkable job of mimicking how humans interact and can carry on conversations. The use of ChatGPT, GPT-4o, o1, o3, Claude, Gemini, Llama, CoPilot, or any of the mainstay generative AI apps provides an impressive indication of the seeming level of fluency that this computationally attains.

All appears to be well and good.

Except for the problem that we supposedly are running out of data that hasn’t already been scanned. In other words, if we’ve already scanned just about all available viable data, and if it takes even more data to make further progress with AI, things are nearing a bleak dead-end. Some say that we are reaching so-called “peak data” of which, afterward, the spigot will be dry.

No more data, no more advancement in AI.

Yikes, that’s bad.

Really bad.

It means that the AI makers that have made bold and fearless promises of where AI is headed will not be able to fulfill their solemn pledges. It means that the valuation of those AI makers, which currently bake solidly a presumed belief that they can get AI to rise to AGI or ASI levels, well, those shaky stool legs are about to get kicked out from underneath.

Another sobering consideration would be that if we hope to use AI to solve crucial humankind issues such as curing cancer or overcoming world hunger, we aren’t going to get there unless we can push AI to greater heights. I’ve discussed how the prevailing aspirational aims are to use AI to solve the United Nations SDGs (sustainability development goals), see the link here.

Kiss those revered ambitions goodbye if AI is indeed hitting the wall and can’t undergo further progress.

Examining The Data Dearth Contention

The clamoring about a dearth or scarcity of available additional data for AI is starting to ring mighty large alarm bells.

During a recent interview, Elon Musk emphatically stated that “we’ve now exhausted, all of the, basically, the cumulative sum of human knowledge has been exhausted in AI training” (stated in a Las Vega CES interview conducted by Mark Penn, posted online on X, January 8, 2025).

Similarly, in an AI research article published in the journal Nature last month, this same point was jarringly noted: “The Internet is a vast ocean of human knowledge, but it isn’t infinite. And artificial intelligence (AI) researchers have nearly sucked it dry” (per the article entitled “The AI Revolution Is Running Out Of Data: What Can Researchers Do?” by Nicola Jones, Nature, December 12, 2024).

Is it true that we’ve used up all available viable data when it comes to training AI?

Maybe, kind of.

Let’s consider a variety of noteworthy caveats.

The Data That Isn’t Being Counted

One aspect is whether the data exhaustion claim is distinguishing between said-to-be freely available public data versus privately held or pay-to-play data. The rub goes this way. AI makers scan the Internet for data that isn’t sequestered behind paywalls. They want presumably free stuff, so they don’t have to pay through the nose to get the data.

You might be aware that a tremendous legal battle is underway as to whether the bevy of AI makers legitimately have or are scanning and utilizing such data or whether it is a form of Intellectual Property (IP) rights taking or rip-off, see my coverage at the link here. If the courts decide that the AI makers must pay up for all that data usage, bam, drop the mic, because the unpaid bill could be beyond imagination.

The gist is that there is data that costs money to access, and not all the AI makers are necessarily willing to pay to do so. There has been a slew of here-and-there deals being forged between data hoarders such as major publishers and social media sites, wherein they now realize the data gold they’ve been sitting on all this time. Might as well monetize the data while the iron is still hot.

Something else to consider is the part of the Internet that is the forbidden zone.

I’ve discussed how the dark side of the Internet, consisting of websites you can’t normally access through conventional browsers, contains tons and tons of potential data, see my analysis at the link here. The issue, of course, is that much of that data is atrocious. It contains foulness, ugliness, and untoward content. If you opt to data train AI on that kind of data, the results are unlikely to be usable for everyday generative AI.

Some AI makers won’t touch the dark web with a ten-foot pole. On the other hand, maybe there is stuff in there that is worth salvaging. One eye-brow-raising perspective is that you can’t truly have a full humankind pattern-matching unless you also include the underworld stuff. Perhaps the right form of filtering could catch hold of the right stuff and avoid the bad stuff.

Get More Data However We Can

The data drought question is fuzzy depending upon whether you are counting solely readily available public data or also including data that costs dough to access, and whether you are willing to scour parts of the Internet that might be construed as perilous and will require a lot more work to divine useable elements.

All in all, let’s assume for the sake of discussion that the Internet is indeed nearly all used up. It is just about as dry as a bone.

What else can we do?

The usual suspects are to consider these three major possibilities:

• (1) Find offline data and bring it online.
• (2) Have humans create more data from scratch.
• (3) Use generative AI to produce synthetic data.

The first use case is that there is undoubtedly a lot of data that isn’t yet electronically available and exists only in paper-based form. Such data is probably not on the Internet. A concerted effort could be made to uncover useful paper-based data and bring it into a digital format.

Anyone who does this is bound to incur costs to do so, thus, the question is whether they will make it freely available online or opt to require payment. Some non-profits and foundations have been bearing the effort and making such data available for free, but this is still likely a drop in the bucket of what might be out there. Look at your desk drawers, attics, and closets since you might have droves of silver ingots in those old hand-written letters, diaries, and the like.

Next is the idea of having humans create more data.

How so?

You could hire legions of people, perhaps via crowdsourcing, and get them to write whatever they might be able to compose. It could be fictional; it could be non-fiction. It could be lengthy; it could be brief. Churn out stories, essays, and poems, until they’ve worn out their typing fingers.

Once again, the issue of cost comes to the fore. Who will pay these people? Will they cheat and submit AI-produced data, skipping the human writing process? And can this be done at scale, in the sense of the number of people and the pace at which people can write, well, it seems a limiting factor as to how much data you can get via this path.

Finally, the perhaps most obvious and seemingly easiest route is to use generative AI to produce data that could then be used to further train other generative AI. This is a no-brainer. You merely tell AI to start spewing out millions upon millions of stories, poems, etc. Let the computer run and do this until the end of time. It is an endless supply of data, coined as synthetic data due to being created via AI and not by human hand.

As with most things in life, there is a potential catch.

The catch is that there is an ongoing and quite heated debate about how useful AI-produced data is. One argument is that the data is essentially a clone of sorts, being weaker and less potent. If you feed that into generative AI for training purposes, the concern is that the AI is going to become watered down. For my in-depth review of this qualm, see the link here.

The draconian view is that if AI makers go the path of relying on AI-generated data, the result will be a catastrophic model collapse, i.e., LLMs will fall apart at the seams and be utterly useless.

Sad face.

The Analogy To Oil Is A Bit Misplaced

A popular saying these days is that data is like oil.

Oil is what makes our world turn and we depend upon it in so many vital ways. The analogy to data is that data is what makes AI become a reality. Without data, the AI isn’t going to be doing much of anything useful. Data has become as valuable as oil, in the sense that they are both precious commodities and are extremely significant to modern society.

I get that and can see why it is an appealing comparison.

But what I remind people during my AI talks is that oil is essentially non-renewable. Once you use up a gallon of oil to ultimately run something like a car or a factory machine, that particular gallon of oil is gone. It is no more. It came and went.

Data doesn’t do that.

This takes us to the primary topic that I wanted to get to in this discussion.

Data is still available even after you’ve scanned it. The act of scanning the data doesn’t cause it to somehow disintegrate. The data is still there. The data can be further utilized. You can presumably reuse that data over and over again. Each use doesn’t pare down the data. The data is still intact.

I realize that smarmy trolls will say that you can delete data. Yes, that’s obvious. You can digitally delete data, and it is no longer wherever it once was. Sure. Got it. I am saying that the act of scanning data, which is what gets the value out of it, doesn’t consume it. The data remains.

Now, if someone scans data and deliberately decides to delete it or accidentally does so, that’s a different story. I’d respectfully ask that we be reasonable and acknowledge that for all fair and sensible considerations, today’s data is not deleted or undercut by the act of using the data.

Squeezing More Juice From Data Is An Option

Why am I making such a big deal out of the fact that data remains intact upon usage?

Here’s the reason.

The unspoken premise and the handwringing of nearing the end of available viable data are that we are already getting as much bang for the buck as we can. The belief is that the data has been fully exploited. Each morsel of data has given its all. Nothing else can be garnered.

I think you can readily see that hidden assumption. The basis for seeking more data is that we are assuming that existing data has been suitably and completely unpacked. Since that seems to be the case, the only other recourse is to find or make more data. Period, end of story.

Not everyone agrees that we’ve gotten all the juice from existing or mined data. There is more in there to be found. Take that data, give it more scrutiny, and squeeze it for every ounce that you can get. Keep squeezing until you are absolutely sure that there isn’t anything else to be discerned.

I’ll share with you in a moment some of the innovative thoughts on getting more juice from available data.

Before we get there, a quick comment about why that doesn’t seem to come up in these conversations about the peak data conundrum. One possibility is that there is a prevailing belief that we are already squeezing data for what it has to offer. We are already optimized about this. Thus, it is a given that we aren’t going to get any further without finding more data.

Another angle is that it is a form of groupthink. If everyone else is clamoring for more data, you will be spurred to do likewise. You want what they want. It is just as simple as that.

Yet an additional perspective is that it doesn’t hurt to be looking for more data. In essence, behind the scenes, you are trying to pull a rabbit out of a hat by making existing data more fruitful. Maybe that will work out, maybe not. In the meantime, save your bacon by looking at the high hills for more data. The two pursuits are not mutually exclusive. You can do both at the same time.

Consider An Example Of Getting More Juice

At this juncture, you are potentially sold on or at least intrigued by the idea that we should be trying every which way to get as much out of data as we can — specifically as it pertains to the data training of AI.

I’ll give you a quick example of how this might be undertaken and then get into several others. These are all innovative AI-driven thought experiments or AI research pursuits, and we don’t know yet which if any approaches will be notably beneficial. Each squeeze requires a cost, and you must weigh the upside value of the additional juice versus the cost to undertake the squeeze. It’s an ROI thing.

This first example has to do with a topic that I recently covered regarding the emergence of large concept models (LCMs), see the link here. Allow me to bring you briefly up to speed.

Right now, most LLMs and generative AI process each word that you enter, one word at a time. In contrast, the precept of LCMs is that rather than focusing on individual words when devising generative AI, it might be interesting and productive to focus on sentences. You enter a sentence, and the generative AI works by crunching the sentence as a whole, rather than parsing each word in the sentence.

To accomplish this, an LCM seeks to identify what concepts exist within a given sentence. A sentence that a user enters is examined to computationally ferret out the unstated concepts that the sentence embodies. Depending upon the sentence, there can be just a few concepts within it or there can be a myriad of concepts.

This takes us to the juice squeezing.

Suppose that a state-of-the-art LCM can garner on average some N number of concepts per sentence. Great. Assume we are happy with that. I’ll make up a totally fictitious number to make this less abstract, let’s say it is 5 concepts per sentence on average.

But we take another look since we want to squeeze as much juice from the data as we can find. After doing some heads-down roll-up-the-sleeves work, we discovered a computational means of getting M number of concepts per sentence, whereby M is greater than N. Pretend for ease of illustration that we are now at 10 concepts per sentence on average.

This suggests that with each sentence input, we are doubling the number of discovered concepts. Assuming that the more concepts derived the better off we are, this is a boon to the LCM approach. We are getting more bang for the buck out of the data that we have.

Extrapolating further, imagine that we could do this for all sentences at scale. If we have millions or billions of sentences, we are hopefully going to see huge gains by having doubled how much we can derive from each sentence. Nice.

Just a reminder that this is a thought experiment, and you should not run out and start proclaiming we’ve achieved some breakthrough in data expansion or elicitation.

More Examples To Whet Your Appetite

I’ll liken this situation once again to oil since it might be helpfully illustrative.

I’ve already noted that the oil analogy has its failings, but one could say that this is somewhat akin to squeezing more out of each gallon of oil. Again, this is a weak comparison because the oil is being destroyed or consumed. Anyway, you get the general conception, namely that if we could stretch out what energy we can derive from each gallon of oil, we wouldn’t necessarily need to find as much additional oil, all else being equal. This doesn’t mean we would stop looking for more oil. We would be perhaps less panicky and more measured.

The same goes for squeezing more out of existing data.

I don’t have space in this discussion to go into depth on the numerous possibilities, so I’ll just quickly run through a handful. I’ll be covering each of these in full discussion in upcoming posts thus be on the look if this topic interests you.

• (1) Use of dynamic contextualization: Prevailing approaches tend to have the model make use of a fixed context window. If this was smartly dynamically adjusted, the chances are that more could be computationally discerned from the data. There is a possibility that long-range dependencies that weren’t found by existing methods might be discovered.
• (2) Use of cross-domain integration. Suppose we slice data into domains and then have seemingly unrelated domains be cross-domain integrated to potentially uncover creative breakthroughs. You can see this happen with humans whereby a person who knows about music can find new value in another domain such as architecture. I’m not equating human facilities with AI and only noting the upsides of cross-domain considerations.
• (3) Use of data remixing. Somewhat like cross-domain integration, this additional approach seeks to remix existing datasets in unconventional ways. If we overlay one type of data into another, new aspects might arise. See for example my discussion about large geospatial models (LGMs) at the link here.
• (4) Use of temporal decomposition. Take data and categorize it by time, such as how data and the underlying patterns of the data change because of time. This is a variation of time-series analysis, though on the basis of textual corpora, see my discussion at the link here.
• (5) Use of quantum-inspired pattern matching. Quantum computing is going to shake up computers and AI, that’s pretty much a given. In this instance of squeezing data, we might be able to use quantum algorithms to examine datasets and find entangled relationships that heretofore were not uncovered. It’s a bit of a reach but worthy of being on the list of possibilities. For my prior coverage of quantum computing, see the link here.

Those are some approaches that seem especially promising. Others exist and I’ll be gradually covering as many as are reasonably sound.

Twists And Turns Aplenty

I’ve got another twist for you.

One of the points earlier was that generative AI tends to produce data that is unsuitable for feeding back into AI for additional data training. Given that concern, we ought to categorically find ways to improve the generation of synthetic data. That would be a home run, for sure.

I don’t want to lean too heavily into the oil analogy, but it comes up handily here. I mentioned that oil is a non-renewable resource. That isn’t quite accurate. It just takes nature a long time to produce oil and therefore it is expandable, just not in our preferred timeline. Suppose we could produce oil, real oil, rather than waiting for nature to do so.

The same goes for data.

Generative AI and LLMs at this time seem to produce a semblance of shall we say inferior data that isn’t good enough to resupply and train AI. Let’s fix that. We can then produce as much additional data as we might ever want. This admittedly doesn’t have to do with squeezing more juice from data. At the same time, if we could get high-quality data consistently via AI, being able to squeeze the heck out of that data would admirably be icing on the cake.

Doing What We Can With What We Have

Final thoughts for now.

A common refrain is that we are addicted to data, just as we are addicted to oil. R. James Woolsey, Jr. famously said this: “We aren't addicted to oil, but our cars are.” The point is that AI is right now considered addicted to data (not in any sentient manner). That is the culprit of focus here.

You see, we have insidiously boxed ourselves in. We have devised a prevailing overarching approach to LLMs and generative AI that places us into the strictly addicting box of needing data. Maybe we have shot our own feet.

Is that the only way to attain advanced AI?

Some argue that we are actually heading toward topping out anyway due to the prevailing approaches. This is not because of a lack of data but due to a chosen approach that is self-limiting. See my remarks about this and the predicted next step involving neuro-symbolic AI at the link here.

The concluding commentary goes to the immortal words of Theodore Roosevelt: “Do what you can, with what you have, where you are.”

Wise words — let’s get to it.