Custom chips drive AI’s future

These days, it seems like nearly every tech giant is developing its own custom silicon chips, or eyeing existing chipmakers to acquire. Why? As the number of chips needed to power a dizzying array of AI applications skyrockets, producing chips in-house can reduce costs and improve the performance of AI systems, says Shobhit Varshney, a VP and Senior Partner at IBM Consulting.

“When you optimize the architecture of the hardware to work with the architecture of the software, that makes magic,” he says in a recent episode of IBM's Mixture of Experts podcast. “It decreases the total cost and latency while increasing the throughput.”

By producing chips in-house, companies also reduce their reliance on the chipmaker NVIDIA, which controls an estimated 70 to 95% of the AI chip market. This isn’t the whole story, however, says Kaoutar El Maghraoui, a Principal Research Scientist and Manager at IBM Research, in an interview with IBM Think. Reducing dependence on NVIDIA merely shifts “the center of power moves from one giant to another,” she says.

Even as companies take greater control of the design process, which NVIDIA has dominated in recent years, companies will still depend heavily on the Taiwan Semiconductor Manufacturing Company Limited (TSMC), Varshney says. TSMC fabricates the majority of AI chips globally for a wide range of applications, from smartphones to military equipment.

TSMC is the “100-pound gorilla,” says Varshney. “Everybody's designing the chips, but TSMC is the heart of the entire industry at this point.”

The chip race in Silicon Valley has been underway since long before generative AI applications supercharged tech companies’ appetite for them. In 2015, Google’s AI system AlphaGo, powered by a Google-designed chip known as a tensor processing unit (TPU), beat a professional human player at the ancient Chinese game Go. Since then, Google has unveiled a series of chips it designed in-house to power AI systems in its data centers. Most recently, in December 2024, Google announced a new AI chip for quantum computing named Willow. The company says Willow can complete a standard benchmark computation in under 5 minutes—one that would take one of today’s fastest supercomputers 10 septillion, or 10²⁵, years.

Around the same time that Google was launching AlphaGo, researchers at IBM started investigating building AI hardware, too. By 2021, IBM had opened its AI Hardware Center in Albany, New York, to create a wider AI hardware-software ecosystem, and by 2022, IBM’s new Telum microprocessor chip had brought AI inferencing to IBM Z, the mainframes that run roughly 70% of the world’s transactions by value. In late 2024, IBM announced a new Spyre accelerator chip, which brought generative AI to IBM Z mainframes for enterprise users.

Meanwhile, AWS has been working on its own computer chips for AI projects since at least 2018. Fast forward to the 2024 AWS annual event, where Amazon announced its latest custom Trainium3 AI chip, which it is bringing to customers paired with partner Anthropic’s large language models. Many companies have snapped up AWS’s AI chips, including Apple, which drew attention at AWS 2024 as it was a rare moment where Apple discussed one of its vendors.

Not to be outdone, Microsoft, which has made chips to power its gaming functions for years, announced its own custom AI chips in 2023, around the same time tech giant Meta announced its own silicon chip plans. OpenAI is the latest to join the custom silicon party, though it hasn’t made any official announcements yet. While no details have been shared publicly, Reuters reported earlier this month that OpenAI was finalizing its chip designs, with plans to start fabricating them via TSMC in 2025.

Why has the chip race intensified recently? IBM’s Varshney says that when companies can customize chips to specific language models for the use cases they need, they can cut costs, improve latency or speed up the movement of data from one network to another. He points to an example: historically, when companies were doing fraud detection and examining incoming invoices, they used classical computing techniques because the volume was high, and they needed very quick latency. “They also had to do this a million times a day, so the cost would add up really quickly,” Varshney says.

Now that companies can optimize their chips for specific models, the cost of high-volume use cases goes down, and it becomes more cost-effective to use these solutions in production at scale. “So from an enterprise perspective, the use cases don't change,” Varshney says. “But now we start to go after the high-volume ones where earlier the ROI didn't exist.”

Given that more and more tech companies are designing chips in-house, how does TSMC maintain its edge over competitors? For one, the company’s approach from the get-go has been different from other tech companies. They are a dedicated foundry, meaning they don’t design any chips but only manufacture chips for other companies, says IBM’s El Maghraoui. Over the years, an increasing number of companies have outsourced their chip manufacturing to TSMC because of the growing cost of making an increasingly diverse array of chips.

At the same time, the cost of manufacturing individual chips skyrocketed, said Dylan Patel, founder of SemiAnalysis, a semiconductor research and analysis company, in a recent podcast interview. TSMC has been laser-focused on chip fabrication and has made it very easy for their enterprise customers, says Patel.

“Semiconductor manufacturing is very antiquated and difficult,” he explains. “The barrier to entry is much higher [than most tech firms], as the roles are super specialized.” Finally, TSMC employees take great pride in their work, Patel says. “They will work 80 hours a week in a factory, and if anything goes wrong, like an earthquake, they’ll show up in the middle of the night” because they are the only ones who can fix a given piece of equipment, he says.

Another reason why it’s difficult to replicate the TSMC formula is that they have invested billions of dollars in purchasing dozens of highly specialized advanced chipmaking machines created by another superconductor giant, the Dutch company ASML.

ASML pioneered extreme ultraviolet lithography (EUV), which essentially generates incredibly short wavelengths of light in large quantities to print small, complex designs on microchips. ASML’s chipmaking machines cost between USD 183 million and 380 million each. One of the first EUV machines in the world was installed in 2014 at the Albany Nanotech Complex, which is owned and operated by NY CREATES and of which IBM Research is an anchor partner.

Since then, IBM Research and its partners have built an ecosystem to develop and optimize EUV lithography, which has allowed IBM and others to shrink the size of transistors down to just a few nanometers, tens of thousands of times thinner than a strand of hair.

While ASML and partners like IBM keep working on printing smaller and smaller chips, TSMC will remain in the equation as the chip fabricator. Patel from SemiAnalysis doesn’t see TSMC’s involvement as a problem. “I don’t think it’s necessarily breaking the reliance,” he says. “I think it’s getting TSMC to build in the US.” TSMC announced in 2020 that it was going to build chip fabs in the US, and since then, the manufacturing giant has invested USD 65 billion to develop three chip fabs in Arizona. In the fourth quarter of 2024, chip production began at the first TSMC chip fab in the United States.

Experts like El Maghraoui see promise for companies in using AI to find new materials for chips to power AI. To that end, new models from IBM and Meta may help researchers discover new materials for chip fabrication that could level the playing field in the future.

“When we open source these models, we accelerate innovation, foster collaboration and promote the development of entirely new semiconductors,” says El Maghraoui.

For example, a team within IBM Research’s foundation models for materials (FM4M) project is using AI to design new chips that can achieve the same or better performance but with a smaller environmental footprint, says Jed Pitera, IBM’s Strategy Co-Lead for Sustainable Materials. “If I have a fab, and I am making ten different types of chips, what is the total footprint of each chip?” he says, referring to the power and water used, and emissions created making a given chip.

“When we know the total footprint of chip A, we can start to think about changing how we make the chips, so they still achieve the same performance but with a minimized environmental impact,” says Pitera. “When the environmental footprint goes down, the cost moves in the same direction.”