Azure Maia 300 Explained: Microsoft’s AI Chip System Built for Agentic Inference
Let’s talk about the Azure Maia 300
Imagine asking an AI assistant to fix a bug in a huge codebase. It reads dozens of files, searches the web, runs a test, reads the error, tries again, and finally gives you a working fix. That takes many steps, and every step needs a lot of memory and a lot of fast data movement behind the scenes. Azure Maia 300 is Microsoft’s answer to the hardware problem behind this kind of work.
It is the third generation of Microsoft’s own AI chip family, and it is designed for one job: running trained AI models quickly, cheaply and with less electricity.
In this guide, I will explain the Maia 300 architecture in plain words, show you why it matters with a real-world style example, and tell you what it could mean for the AI tools you use. No hardware degree needed.
What Is Azure Maia 300 and Why Should You Care?
Training an AI model is like going to school. Inference is the job you do after graduating. Every time you ask Microsoft Copilot a question, get a code suggestion from GitHub Copilot, or use a model through Microsoft Foundry, an inference chip does the work.
Training happens once in a while. Inference happens billions of times a day. So even a small saving per answer becomes huge at scale. That is why Microsoft builds its own Microsoft AI accelerator instead of relying only on off-the-shelf parts. When you control the chip, the rack, the network and the datacenter together, you can remove waste at every layer.
Microsoft says the goal of Azure Maia 300 is to deliver more useful tokens per watt and more useful tokens per dollar. A token is a small piece of text that a model reads or writes, roughly a short word or part of a word. More tokens per watt simply means more answers for the same electricity bill.
Why Agentic AI Inference Needs a Different Kind of Chip
A simple chatbot reply is short. You ask, it answers, done. Agentic AI inference is different. An AI agent plans, uses tools, reads long documents, checks results and tries again. This changes what the hardware must be good at.
Let us make it real with an example.
Meet Ananya and her team. They run engineering at a mid-size company that sells delivery software. They build a coding helper that does the following when a developer reports a bug:
- Reads the bug report and about 40 related files from a large codebase.
- Searches the internal docs for similar issues.
- Writes a fix and runs the test suite through a tool.
- Reads the failed test output and corrects the fix.
- Repeats until the tests pass, then writes a summary.
Nothing about this is exotic. But look at what the hardware has to do. The model must hold a very long conversation in memory (this is called long-context work). It must write many tokens one after another (this is called decoding). And it must do this for thousands of developers at the same time.
Three things start to hurt:
- Memory capacity: Each session needs its own memory of the conversation so far. Long sessions fill the chip’s memory fast.
- Memory speed: The chip must read that memory again and again for every new token. If reading is slow, the powerful math units sit idle and wait.
- Moving data around: Shuffling data between chips, between memory and compute, and between racks costs time and electricity.
Microsoft describes the needs in a similar way: keep data close to the compute, give the compute enough memory capacity and bandwidth, move less data, scale communication well, and fit as much power as possible inside a datacenter’s fixed power and cooling limits. Azure Maia 300 is built around exactly those needs.
The Big Idea Behind the Maia 300 Architecture: Move Less Data
Think of a busy restaurant kitchen. A great chef is useless if every ingredient is stored in a warehouse across town. The best kitchens put the ingredients you need right beside the stove.
That is the heart of the Maia 300 architecture. Microsoft’s design is called SDLA, short for Software Defined Locally Accessed Dataflow Architecture. It first appeared in Maia 200. The idea is that moving, storing and converting data is one of the hardest parts of modern AI. So instead of letting hardware guess, the software tells the chip exactly where data should sit, when it should move and when different parts should sync up.
In the kitchen picture, the software is a head chef who plans every ingredient delivery in advance, so nobody waits and nobody walks further than needed.
Maia 300 improves this idea in three main ways:
- Smarter memory layout. Maia 200 treated memory as one even pool. Maia 300 uses a locality-aware layout, known as NUMA, across chiplets. Each compute area has memory nearby, and software can place data right next to the part that uses it.
- Smaller number formats. New low-precision formats shrink the data, so fewer bytes travel for every token.
- A bigger connected system. The rack doubles to 96 accelerators, and the connected group can grow to 12,288 accelerators.
Inside the Maia 300 Chip: Chiplets, Memory and Smaller Numbers
Chiplets: Use the Right Tool for Each Job
Instead of one giant slab of silicon, Maia 300 uses several smaller pieces called chiplets. The compute part is built on TSMC’s N2P process, an advanced manufacturing technology that gives the best performance per watt. Other jobs, such as memory control and input and output, sit on separate dies that are tuned for bandwidth, connections and lower cost.
This is like building a house with the best material for each part. You do not need a luxury finish on the garage floor. Microsoft says this approach raises compute FLOPs by 36%, gives 2.5 times the HBM bandwidth, and adds 14% more scale-up bandwidth compared with Maia 200.
Memory: The Part Agents Care About Most
For an AI inference chip serving long sessions, memory is the star. Each Maia 300 accelerator pairs software-managed on-chip SRAM with eight HBM4 memory stacks. That gives 288 GB of capacity and 18 TB/s of bandwidth per accelerator.
Compared with Maia 200, that is 33% more capacity, 2.5 times the memory bandwidth, and about 2 times the memory bandwidth per watt. Back to Ananya’s team: this means each chip can keep more of the running conversation (the KV state, which is the model’s short-term memory) close by, so it can serve more developers at once without constant trips to far away storage.
Smaller Numbers, Smaller Loads
AI models store their numbers in formats of different sizes. Smaller formats mean less data to move. Maia 300 adds improved MXFP4 and storage formats below 3 bits, plus hardware that converts between formats quickly.
Think of packing for a trip. If you can fold your clothes tighter without damaging them, the same suitcase holds more. Smaller formats let the system run bigger models, longer contexts and more sessions in the same power budget.
The vector engine also gets new instructions to speed up common inference jobs such as softmax and low-precision data casting.
One Network for Everything: From Pod to Superpod
A single chip is never enough for large models. Chips must talk to each other, and the talking has to be fast. This is where Azure Maia 300 does something neat: it treats the network as part of the chip design.
Inside the Rack
Each accelerator splits its connections into two types. Direct links join two accelerators within a tray. A switched network joins all 96 accelerators in the rack, forming a pod. All in-rack links are electrical, which saves power and cost.
Across Racks
A second layer of switches connects up to 128 racks into a superpod, using optical links for the longer reach. 128 racks times 96 accelerators gives the 12,288 accelerators in a full superpod.
Why Microsoft Calls It One Fabric
Many systems use one network inside a server and a different one between servers. Maia 300 uses the same way of sending data in both places. An accelerator can send data straight to any other accelerator in the superpod, without going through the host computer or a separate network card.
There are three speed zones:
- Fastest: The directly connected pair of accelerators.
- Fast: Any accelerator in the same pod, one switch hop away.
- Slower: Any accelerator in another pod, three switch hops away.
Software can keep the most delay-sensitive work, such as tensor parallelism (splitting one layer’s math across several chips that must talk constantly), in the fastest zones. Wider parallel work like expert and pipeline parallelism can stretch across racks.
The bandwidth split is also flexible. It is set based on how jobs are packed, not locked in at design time. An accelerator can drive up to 750 GB/s to any peer in the superpod when only a few accelerators are talking across racks.
Three Practical Wins
- Split the work between reading and writing. Inference has two phases. Prefill reads your whole prompt. Decode writes the answer token by token. Maia 300 lets you size and place these workers separately, and the KV state moves between them over the same fabric, without leaving through a slow host path.
- Fewer stranded chips. Jobs can use capacity across rack boundaries, so odd-sized jobs do not leave expensive accelerators sitting empty.
- Better resilience. If an accelerator fails, a hot spare in another rack can take over without the software kernels noticing. It costs a bit more delay, but the service stays up.
The Rack: 96 Accelerators Within 167 kW
A data center has a fixed budget for power and cooling. So the real question is how much useful AI you can run inside that budget. The Maia 300 rack is designed to answer this.
Dense layout: 96 accelerators sit in 24 trays of four, inside a 167 kW rack. That is double the density of Maia 200. Each tray carries its own host processor, memory, storage and cooling, so separate host trays are not needed. That frees space and lets power and heat be managed as one system.
Fewer switch boxes: Six switch chips live in three liquid-cooled switch trays, giving every accelerator its in-rack connection with little extra space and power.
Power built for uptime: Dual power feeds, high-capacity power shelves and a 50 V busbar spread power around. An automatic transfer switch keeps the rack running even when one feed is down for maintenance.
Flexible cooling: Liquid cooling handles more than 80% of the heat from accelerators, voltage regulators and switch chips. Air handles memory, storage and smaller parts. The rack can connect directly to facility water or use a heat-exchanger unit in air-cooled buildings, so it can be placed in more kinds of datacenters.
Live telemetry: Rack and row level power data feeds into datacenter operations. This helps allocate power, place workloads and set deployment density, which raises tokens delivered per watt.
Microsoft says these choices together give up to 2.5 times more performance per watt than Maia 200 at the rack level.
What This Means for Developers, Businesses and Everyday Users
Developers and AI teams: Expect cloud models and agent platforms to get cheaper and faster over time as hardware like this rolls out. Designing for long context and many tool calls becomes more practical.
Businesses: Efficiency gains in the datacenter can help control the cost of running AI features at scale.
Everyday users: Tools such as Copilot may feel quicker and handle longer tasks, though you will not see which chip is doing the work.
The industry: It shows a clear trend. The winners in AI will not just have a strong chip. They will tune chip, network, rack, cooling and software together.
Frequently Asked Questions About Azure Maia 300
What is Azure Maia 300?
Azure Maia 300 is Microsoft’s third-generation custom AI accelerator system, designed to run AI models efficiently for large-scale inference, especially agent-style workloads.
What is inference in AI?
Inference is the stage where a trained AI model is used to answer real requests, such as writing text or code. It is different from training, where the model learns.
How much memory does each Maia 300 accelerator have?
Each accelerator has 288 GB of HBM4 memory with 18 TB/s of bandwidth, according to Microsoft.
How many accelerators are in a Maia 300 rack?
A rack holds 96 accelerators across 24 trays of four, within a 167 kW power envelope.
What is a Maia 300 superpod?
A superpod links up to 128 racks, for up to 12,288 accelerators, using a two-tier network so they work like one system.
How is Maia 300 better than Maia 200?
Microsoft claims up to 7 times more rack-scale token throughput and more than 2.5 times the performance per watt, along with more memory, more bandwidth and double the accelerators per rack.
What is SDLA?
SDLA stands for Software Defined Locally Accessed Dataflow Architecture. It lets software control exactly where data is stored and moved across memory, compute and network.
Can I rent Maia 300 directly?
Microsoft has not stated pricing or direct access terms in this announcement. It says the chip will power its own products and its wider AI portfolio. Check Microsoft’s official channels for availability.
Azure Maia 300 is a good example of where AI hardware is heading. The goal is not only to build bigger chips. It is to move less data, store data in smaller formats, connect chips with one simple network, and pack more of them into the same power and cooling budget.
If you are building or buying AI services, the practical takeaway is that agent-style workloads are shaping how the chips under them are designed.
Microsoft also says it will share more at Microsoft Ignite 2026 in San Francisco and online, so more details may follow.
What would you build if long, tool-using AI sessions became cheap and fast? Share your ideas in the comments.
