The Google Research team at Alphabet $GOOG might have just realized the power of social media firsthand on March 2026.
When the Google team published their whitepaper on TurboQuant and the massive promised cost savings on AI inference earlier on March 2026, there was radio silence in the markets. But when the same findings were published at mid March 2026 on X, markets went into a tailspin, panicking on the impact for memory stocks and possibly giving investors in companies like Micron $MU and SanDisk $SNDK .
Despite being just a whitepaper, the Pragmatic Optimist newsletter believe that the TurboQuant breakthrough by Google is a paradigm-shifting moment that fundamentally elevates the vantage point for the memory bottleneck trade to a whole new level. This doesn’t just impact memory stocks but at least 3 others industry categories of the semiconductor industry.
There are obviously memory companies, the WFE companies that allow memory companies to do what they do, certain CPU companies, and a handful of AI connectivity companies that should begin to benefit from this paradigm-shifting moment that Google’s research team has caused.
Why Memory (not GPUs) Became Scarce
One of the main reasons that the post-GPT era of AI chatbots immediately impressed the world is because these AI chatbots sounded just like humans. The LLMs powering these AI chatbots were trained on exabytes of training data facilitated via AI training workloads. These AI training workloads needed access to almost unlimited volumes of GPU compute, making GPUs a scarcity and thus shooting up the value of Nvidia’s market cap in 2023-24. That was the honeymoon era for AI, at least in the eyes of markets.
Over the past year, AI companies figured out that if their LLMs had to make money, they had to expand their capabilities from being just a “chatbot” to working seamlessly alongside humans, just like coworkers and agents. To make that leap to agentic AI, models now had to remember & reason just like humans to help them execute tasks. This led to an exponential surge in the generation of tokens (the language, currency, and basic units of AI) and rapidly expanded an AI model’s context windows (the working memory of an AI model where it actively thinks, reasons, and solves problems).
The current architectural setup in data centers was designed for an AI model’s context windows to be loaded in memory chip products like DRAM and HBM. This allowed for context windows (a model’s working memory) to perform efficiently but in due process, exorbitantly increased the demand for memory & storage products since the size of context windows dramatically increased.
These were completely new demand/supply dynamics for the memory industry, resulting in significantly skewing the scale of demand away from GPUs towards memory/storage and products, making companies like Micron and SanDisk winners in the inference leg of the AI trade.
But the volume of tokens being generated has exploded over the last 9 months as we enter the agentic era, ballooning the sizes of the context windows of AI models.
And this is where, the investors see the conversation shifting towards KV cache (key-value cache).
AI Has Two Words For You : “KV Cache”
The big draw about AI inference is that models would not be required to be trained constantly. It’s as easy as that. Else GPUs would continue to be a scarcity.
In order for models to “infer,” they need to recall prior information that they were trained on, and models lean on using “a high-speed digital cheat sheet” called KV cache to assist in inferencing.
The KV cache, a form of digital memory, usually sits in the HBM chips, which in turn, are embedded on GPUs. But depending on the size/volume of tokens being recalled by the model, trade-off decisions are made by the system, and the KV cache can be ‘offloaded’ or extended out to other storage systems like NAND-based SSDs, HDDs, CPUs, etc. (as Nvidia explains here.)
We’ve altered a prior diagram from a previous TPO article to show you what the architectural setup is today and how the industry has designed for KV cache offloading if KV cache limits are reached in HBM chips.
The real short version is when a user asks AI for help on a problem, the underlying LLM model will begin by recalling prior information into the KV cache that gets loaded into HBMs or SSDs while computing any new incremental tokens it has learned from this new user request, further utilizing the KV cache. This is why KV cache management and memory utilization have become such critical topics for leading AI players.
There is a reason why we bolded a few words at the start of this section because they translate into critical investment themes that reflect significant enthusiasm for AI companies sitting at “the bottleneck.” And “the bottleneck” just expanded beyond your typical memory stocks like Micron and Sandisk.
In conclusion, according to the Pragmatic Optimist newsletter, KV cache is of extreme importance to Google, as they noted in their TurboQuant paper, and every company from Nvidia & AMD $AMD , to SK Hynix and Micron is racing towards optimizing the usage of KV cache.
