Beyond AI Hype (Part 3): Silicon

Photo by Ryan on Unsplash

Switching Transistors

Every digital abstraction eventually falls through the floorboards and lands on the same unforgiving reality: binary code switching transistors inside a piece of silicon. Intelligence may appear fluid at the application layer — words, images, decisions — but beneath that surface lies the oldest logic in computing: electrons moving across a semiconductor lattice at the edge of thermodynamic tolerance. Software can change overnight; matter cannot. The future of AI is being negotiated not only in research papers and training runs, but in the physics of the chips themselves.

A modern high-end GPU contains between 50 and 90billion transistors, each measuring roughly five to fifteen nanometers across, with the industry aiming for two-nanometer class devices by 2027 or 2028. Manufacturing such a device requires more than a thousand process steps, several months of sequential fabrication, and the synchronized contributions of hundreds of global suppliers. In practice, a single accelerator is the culmination of one of the most complex production processes ever devised by industry.

Modern computing relies on two foundational devices: the CPU and the GPU. The CPU is the generalist, optimized for sequential control flow and orchestration; it is the conductor of the system, not the orchestra. The GPU, by contrast, is a parallel machine built around thousands of compute cores and tens of thousands of concurrent threads, performing linear algebra operations at teraflop and now even petaflop scales per rack. GPUs excel at matrix multiplications not only because of their raw performance but because they are programmable, enabling the same piece of silicon to handle deep learning, graphics, genomics, molecular simulation, and financial modeling. This flexibility is what allowed GPUs to become the default engine for AI scaling.

General Parallel Machine

Long before transformers, high-performance computing embraced the GPU. Weather models used them to solve fluid dynamics across massive grids. Astrophysics simulations propagated particle interactions across billions of nodes. Quantitative finance used them to accelerate Monte Carlo pricing by an order of magnitude or more compared to CPUs. These were narrow but already powerful forms of machine intelligence .

When the transformer architecture appeared in 2017, demanding precisely the type of matrix multiplication GPUs were engineered to accelerate, the alignment was immediate. Training a frontier model is little more than performing linear algebra at massive scale, and GPUs, long accustomed to physics and finance, simply absorbed the workload. Today a single H100-class GPU typically costs between $25,000 and $40,000, with production lead times of three to six months and constrained HBM3 memory availability. Installing racks of such systems requires specialized handling crews, vibration-controlled mounting, redesigned power distribution, and advanced cooling systems.

Behind these devices lies one of the most fragile, globally distributed supply chains in the modern economy. The United States excels at chip design but does not fabricate the most advanced chips. Almost all leading-edge logic is produced at TSMC in Taiwan, which itself relies on EUV lithography machines from ASML in the Netherlands. These machines are costing well over one $180 million each and requiring 12 to 18 months to deliver. Those EUV tools in turn use Zeiss optics from Germany, which take years to calibrate. Meanwhile, Japanese and South Korean firms supply the photoresists, wafers, and high-bandwidth memory stacks that make modern accelerators viable at all. A single GPU embodies contributions from five or six countries and hundreds of companies, none of which can be easily substituted. In a world drifting toward de-globalization, fragility is now priced directly into the cost of compute.

NVIDIA stands not merely as a vendor but as an institution. Its dominance rests on a bet made more than a decade ago: that GPUs could become general-purpose accelerators given the right software environment. CUDA was that environment. By binding its silicon to a unified programming model, NVIDIA fused its hardware to its software and created lock-in that pervades the industry. Every major leap in large-scale AI, from convolutional networks to GPT-4, was trained on NVIDIA hardware.

The numbers reflect this dominance. More than 90% of frontier model training globally uses NVIDIA accelerators. An H100 delivers multiple petaflops of FP8 performance, with each DGX rack drawing ten to fifteen kilowatts and multi-rack pods easily exceeding 100 kilowatts. The company’s weakness is obvious — deep reliance on TSMC and the limited supply of HBM — but its strength is overwhelming: every competitor must not only match NVIDIA’s silicon but also recreate a decade of developer tooling and ecosystem maturity.

AMD approaches the market differently. It cannot match CUDA’s gravitational pull, but the MI300 series has become materially competitive. The MI300X offers more memory than many NVIDIA equivalents, with 190+ gigabytes of HBM3, and delivers training throughput that sits within 10% to 20% of H100 clusters on many real-world workloads. ROCm, AMD’s once-immature software stack, has matured enough that hyperscalers now deploy AMD accelerators at meaningful scale. The cloud providers motivation is clear: they need a second source. AMD does not need to win the entire market; it needs only to be credible enough to give hyperscalers bargaining leverage and hedge supply risk. In moments of scarcity, optionality becomes a strategy.

Application Specific Circuit

If GPUs represent flexibility, ASICs represent commitment. An application-specific integrated circuit is hardened for a model’s core mathematical operations, trading versatility for efficiency. ASICs often achieve two- to five-times improvement in energy efficiency relative to comparable GPUs and produce less heat per teraFLOP. But the economics are brutal. Designing a frontier-class ASIC can require $300 to $600 million in non-recurring engineering, followed by fixed fabrication volume commitments. When model architectures evolve every six to nine months, an ASIC risks becoming obsolete before its cost is amortized.

Google’s TPU program illustrates both the promise and the complexity of this approach. TPU v5p pods scale to tens of thousands of chips operating as a unified cluster, achieving meaningful reductions in energy per training step compared to GPU-based systems. Google controls the entire vertical stack (silicon design, system layout, networking, and software) which allows it to optimize both cost structure and reliability.

Amazon’s Trainium and Inferentia chips reflect similar logic but with a different goal: to stabilize AWS cloud margins by reducing reliance on NVIDIA and improving cost per inference and cost per training step. Meta, seeing the fragility of depending on NVIDIA for its open-source roadmap, has accelerated its own inference accelerator programs. Microsoft and OpenAI, now tightly coupled, are pursuing custom silicon as well. Once training cycles cost hundreds of millions or more, owning the hardware layer becomes not an option but an inevitability.

Packaging

If silicon is the brain of modern AI, packaging is the circulatory system. At frontier scale, bottlenecks rarely originate inside compute cores; they emerge in the spaces between them, in memory interfaces, interposers, copper traces, thermal pathways, and optical links.

High-bandwidth memory (HBM)is the heart of this system. HBM3 and HBM3e supply between three and five terabytes per second of bandwidth per GPU, but each stack consumes five to fifteen watts and produces enough thermal density that liquid cooling becomes mandatory. Without HBM, the arithmetic units in a GPU would starve; with it, the device becomes viable for models with enormous context windows and parameter counts.

CoWoS, or chip-on-wafer-on-substrate packaging, enables the GPU die and its HBM stacks to sit on a shared silicon interposer, allowing hundreds or thousands of parallel data channels. But CoWoS is constrained by physics and manufacturing capacity. TSMC’s CoWoS lines are oversubscribed, with packaging lead times stretching from six to twelve months and per-unit costs that often exceed several thousand dollars. As HBM stacks grow taller and interposers grow larger, heat density climbs, tolerances shrink, and the entire structure approaches the edge of what current materials can sustain. NVIDIA, AMD, Google, and Amazon all converge on similar packaging architectures not because they collaborate, but because physics collapses the design space.

Networking

Networking is the other half of the system, and perhaps the more unforgiving one. A frontier-scale model cannot be trained on a single device; it must be distributed across tens of thousands of accelerators acting in unison. The interconnect determines whether these accelerators behave like one machine or like a fractured cluster. Latency, bandwidth, and congestion control are not abstractions; they define the wall-clock duration of a training run and the degree to which capital expenditure can be amortized before the next generation of models arrives.

InfiniBand, controlled by NVIDIA through its Mellanox acquisition, remains the gold standard for frontier training. It offers microsecond-scale latency, extremely high bandwidth, and adaptive routing optimized for collective operations such as all-reduce, the core primitive in distributed training. The cost of these switches often ranges from tens of thousands of dollars to far more depending on port count and speed, and each device adds heat and power overhead. A high-end AI cluster may devote 20% to 30% of its total power budget to networking alone, and the cost of optical transceivers and fiber — often hundreds of dollars per connection — accumulates quickly.

Ethernet-based designs allow Amazon, Google, and others to exploit commodity optics and custom data-plane processors. Google’s Jupiter fabric achieves petabit-scale throughput, while Amazon’s SRD protocol narrows the performance gap with InfiniBand while retaining the economic advantages of Ethernet. AMD’s accelerators integrate more naturally into these Ethernet-centric ecosystems than NVIDIA’s tightly bound InfiniBand stack. Regardless of the vendor, the requirement is the same: networks must operate as if they were copper wires at the speed of light, even when stretched across hundreds of meters of fiber.

Interconnect costs are immense. In some clusters, the optical backplane and switch fabric rival the cost of GPUs themselves. A trillion-parameter training run may require multiple hierarchical layers of routing, photonic links for long-range paths, and per-device cooling loops. Computation is determined less by math than by the movement of data and heat.

Fixed Costs

All of these architectural decisions — the choice of silicon, the memory topology, the packaging constraints, the networking fabric, the thermal envelope — are not tweaks but irrevocable, long-term commitments. Once a company chooses a chip family, a packaging method, or a network stack, it is binding itself to a capital plan measured in billions and a time horizon measured in years. These choices create enormous fixed costs and enormous operational leverage.

A GPU cluster that has already been built, installed, cooled, powered, and interconnected cannot be reconfigured cheaply. Its economics harden into place. The marginal cost of a training step approaches zero; the marginal cost of unused capacity approaches infinity. Like an airline seat or a hotel room, a GPU hour is perishable. If it goes unsold today, it cannot be reclaimed tomorrow. This forces hyperscalers and model labs into the logic of revenue management — dynamic pricing, long-term contracts, preferential access, demand shaping, and overprovisioning strategies designed to keep the machine full. Financial leverage enables this strategy, but also tightens the trap: debt and multi-year commitments amplify exposure. Once the infrastructure is built, they cannot unbuild it. They must grow into it.

These decisions do not exist in a vacuum. They anchor themselves in the physical world — in power grids, water systems, land use, zoning boards, and municipal politics. What begins as a compute cluster becomes a regional energy footprint, a thermal plume, a water draw, a noise profile, and a new industrial burden placed upon communities who receive none of the operational leverage but all of the environmental externalities. A 500-megawatt or 1-gigawatt AI campus is not merely a financial asset; it is an infrastructural liability that persists for decades. The fixed costs are not only on private balance sheets — they are socialized into the electrical grid, the environment, and the political fabric of the region hosting the facility.

And this is where Part 4 shall begin. The moment silicon leaves the cleanroom, it enters the real world — a world of transmission lines, water rights, cooling systems, transformer farms, and public scrutiny. AI factories do not simply compute; they consume. They reshape land, drain aquifers, congest substations, and test the patience of regulators and communities who must bear the long-term consequences of infrastructure built for models that may be obsolete in 18 months. The financial leverage of Part 3 thus transforms seamlessly into the physical and political liabilities of Part 4. Intelligence may be abstract, but its footprint is not. The next battleground is not just compute efficiency — it is environmental feasibility, public tolerance, and the ability of hyperscalers to justify a new class of industrial installations in neighborhoods that never asked for them.