What does AI look like?

Recently, over coffee, a colleague relayed a question he’d been asked: what does AI look like? 

I smirked. I muttered something about the tremendous complexity of “tensors” and “self-attention” and “neural nets,” words I had encountered from other, more technical IBM colleagues. Through my use of the jargon, I signaled the impossibility of meeting the question-asker on their Neanderthalic level.

More to the point, I thought, hadn’t everyone heard that AI was essentially a “black box”? I’d taken that to mean that we basically couldn’t know what AI looked like, under the hood. It was a fool’s errand. 

We concluded our coffee, and I headed back to my desk. Obviously, if AI “looked like” anything, it was some impenetrable mix of complexity and opacity. “‘What does AI look like?’” I thought, smugly. “What a stupid question!”

That night, I stared at the ceiling sleeplessly, my mind racing. I wanted to know what AI looked like.

Sure, it was a “black box.” But humans had made it, so surely we knew something about what it looked like? 

Disconnected images flitted through my brain. I knew in a vague way about data centers far away. I’d seen schematics of “neural nets,” complete with their “layers.” But what were the layers made of? And where were they? In the data centers? Or inside my computer, in the support files of my ChatGPT application? And was AI written in code? It had to be—all computer stuff was code!

I rose from my bed and commenced to do what one does in 2026. I spent an hour pestering ChatGPT about what it looked like, asking it to walk me through the major research papers that had introduced it to the world, then begging it to “explain like I’m five.” Finally, I reached a tentative conclusion: sure, if you scratched away at normal old-fashioned software, you got code; but if you scratched away at AI, what you got was … numbers. Just lots and lots of … numbers. These numbers were arrayed in massive grids: neat rows and columns.

So AI was basically … spreadsheets? “Yes, that’s fair,” said my robot interlocutor. “Spreadsheets with special plumbing.”

I closed my computer. Mystery solved: AI looked like spreadsheets. For the rest of the night, I slept soundly.

The next day, unease crept back in. OK, sure. AI looked like huge spreadsheets full of numbers. But upon reflection, I had just a tiny follow-up question ...

WHAT??? 

I had known spreadsheets, even fairly large ones. But none of them had been able to talk to me. So what was going on with these spreadsheets that made them so smart?

Now that I had some idea what AI looked like, I wanted to understand what it looked like in action—how, under the hood, it all worked. 

Was I deluded in thinking that might be possible? I’d recently read a pronouncement from Anthropic CEO Dario Amodei. He’d written: “People outside the field are often surprised and alarmed to learn that we do not understand how our own AI creations work.”

Even so, I thought, understanding is relative, and I had a feeling Amodei knew more than me. I was willing to bet he even knew the thing about the spreadsheets.

So, over the next few weeks, I embarked upon a challenge: could I—a writer who had never written a line of code, who had taken the bare minimum of math in college—approach a practitioner’s understanding not just of what AI looked like, but why?

To my surprise, I discovered that despite the “black box” label, we actually can see inside a large language model (LLM). We know what AI looks like—and with each passing day, researchers are learning more about just how it works. By following our curiosity, even laypeople like you and me can begin to bring the picture into focus.

The more I learned, the more I came to believe not only that we can see and understand AI, but that we must. In fact, I became convinced that the ongoing effort to understand AI—by both experts and laypeople alike—will be the very thing that keeps it safe. Let’s begin.

A modern AI system like ChatGPT (or Claude, DeepSeek or Granite) is a series of spreadsheets filled with numbers. It’s a curious irony that this verbal technology is, at root, almost purely numerical.

How many spreadsheets? It depends on the model. For the rest of this article, we’ll focus our energy on OpenAI’s GPT-3, released in 2020, which served as the basic “engine” of the breakout product ChatGPT two years later. Though you may well run a more souped-up model today, it works basically the same way.

GPT-3 has 577 spreadsheets. People in the AI industry call these “tensors,” but it’s fair to think of them as spreadsheets. Each and every cell of each and every spreadsheet is filled with a single, small number (typically between -1.0 and 1.0). And those 577 spreadsheets come arrayed in a specific order (which is to say, in a sense we’ll explore later, that your query “hits” the first spreadsheet first, then the second and so on).

Collectively, the specific numbers in all the cells—variously called “parameters” or “weights”—are the model. By the time the model ships, those numbers are locked in, static. If OpenAI wants to change them, it’s going to spend lots of money doing so through a carefully orchestrated process called “training” (more on which soon). And at the end of that process it will throw a big party and release a new model dubbed GPT-Some-Number-Or-Other. We’re up to GPT-5.6 as of this writing.

All those numbers in all those spreadsheets stand there in columns like DNA, bits of crystallized wisdom. Every conversation you have with a given AI model sends your query ricocheting through the same vast warren of identical, frozen numbers. The only thing that changes is what you input, and somehow that’s enough to conjure the system’s infinitely variable output.

These 577 spreadsheets, as mentioned, are big. How big? The very first spreadsheet—which has a special function we’ll discuss soon—is the biggest: it’s 50,257 rows tall, and 12,288 columns wide. (If you do the math, that’s 617,558,016 tiny numbers, just in the first spreadsheet.)

Then, after the very first spreadsheet, the remaining 576 settle into a repeating pattern of shapes: first four squares, 12,288 rows and 12,288 columns. Then a wide rectangle, 12,288 rows tall but four times as wide. Then a tall rectangle with the same proportions, only rotated upright, 12,288 columns wide. (This repeating pattern of six spreadsheets, it turns out, is a “layer,” an idea we’ll return to later.) 

You’ll notice that the number 12,288 remains a constant, the better for neat daisy chains of multiplication to course through the model. That said, there’s nothing magical about 12,288 per se; it was likely just the largest number OpenAI could attain with their budget at the time. 12,288 is also a multiple of 32, a number preferred by the chips that run AI math.

Cumulatively, across all the cells in all the spreadsheets, GPT-3 holds 175 billion little numbers (“weights,” “parameters”). A huge sum, but it amounts to a simple data structure: GPT-3 is about 350 GB large, capable of fitting on the hard drive of a high-end laptop. Yet even the best laptops wouldn’t have enough RAM to effectively run it, which is why AI does indeed run in those data centers.

One last way to think about AI’s “size”: if you took the first and largest spreadsheet inside of GPT-3 and printed it out, even in tiny 8-point font, it would still be taller than the Empire State Building. Print all 577 spreadsheets and stitch them together like a quilt, and they’d blanket Central Park five times over. GPT-3, with its 175 billion numbers-in-cells, was soon dwarfed by its sequel, GPT-4, with its rumored two trillion cells; that quilt would cover all of Manhattan and the Bronx. GPT-5 and other new models are likely larger yet.¹

So yes, in its raw anatomy, AI is just numbers: lots and lots of numbers. But how do those numbers work their magic? What do they represent, who chose them and why?

Let’s start with GPT-3’s first and biggest spreadsheet, the one towering over the Empire State Building. Though daunting in scale, it’s the most conceptually simple of the spreadsheets. It’s a dictionary.

Fire up ChatGPT and enter a word—say, “dog”—and the first thing it will do is grab a long string of numbers assigned to the word “dog.” That is the entire function of GPT-3’s first spreadsheet, a 50,257-entry Great Dictionary (my term) mapping words to numbers. Each word has its own row.²

GPT-3’s “dog” row (like all other rows) holds 12,288 numbers, most probably in the -1.0 to +1.0 range. We can’t know the exact “dog” numbers in GPT-3, but OpenAI did publish them for GPT-2. In GPT-2, “dog” begins: 0.06, 0.07, 0.10, 0.12, -0.04 ... followed by 1,595 more numbers I won’t trouble you with. (Though GPT-3’s spreadsheets have 12,288 columns, GPT-2’s have just 1,600. Also, I’m rounding numbers to just two decimal places, instead of the actual seven.)

To recap: “dog,” in GPT-2’s Great Dictionary, is a spreadsheet row that looks like this:

“dog” 0.06 0.07 0.10 0.12 -0.04 ... (1,595 more numbers) ...

Specifically, “dog” is the 9,703rd row in GPT-2’s Great Dictionary.

What do the numbers represent? Let’s quickly rule out a few things they’re not. The numbers are not some variant of Unicode or binary—there’s no simple trick swapping numbers for letters. Nor are they randomly generated. The numbers have patterns that are subtle and real. They are a genuine attempt by GPT-3 to represent numerically the meanings of “dog.”

Now, let’s discuss what the 12,288 numbers meaning “dog” almost are.

The numbers almost are what we might call: scores of the suchnesses of “dog.”

Imagine you are a census-taker in a village of words. You knock on the door where the word “dog” lives; he answers the door and consents to your questionnaire.

You: On a scale of -1.0 to +1.0, how would you score your ... animalness? 

(The word “dog” thinks a moment.)

“Dog”: I’d like to think I’m a solid +1.0. Dogs are quite animal-y.

You: OK, so we’ll put you down for a +1.0 in the animalness column …

(“Dog” smacks his head.) 

“Dog”: Oh, wait! I forgot. Sometimes I mean something else. I don’t always mean dog-dog, the animal kind. See, I’m not a dog—I’m “dog,” the English word.

You: Oh?

“Dog”: This is gonna sound weird, but sometimes I actually mean … sausage? On July 4th, or really on any day, if you put “hot” in front of me, I mean sausage. “Hot dog.”

(You frown.) 

You: OK. So then … not fully 100% on the animalness scale, then …

“Dog”: Oh man, it would be so much simpler if that were the case. But no, put me down as +0.9 on animalness. If I had to guess, 90% of the time someone says “dog,” I’m the animal-type.

(You fill out the first column in your census sheet, which is assigned to the abstract concept of animalness. You enter +0.9.)

You: OK, this is great, because you actually anticipated my next question. How would you rate yourself on foodness? 

“Dog”: Hm. I guess if I’m 90% animal, then I’m 10% food? No wait! I forgot something. Sometimes I mean, like, “bad person”? I’m not super proud of that one ...

You: Ah.

“Dog”: Yeah. And especially if you put the word “scurvy” in front of me, I’m a bad person, specifically from the point of view of ... a pirate? 

You: Right ... 

“Dog”: So we need to leave room for that. I guess I’m ... +0.07 food, and +0.03 bad-person-to-pirates?

You check your columns, and while you don’t have a column for “bad-person-to-pirates”—the designer of the census thought that too specific to be useful—you do have one for the concept of insults, as well as one for personhood and another for old-timey diction. You do your best to score those columns accordingly.

You go on like this into the night, because you have 12,288 columns to fill, but eventually your entire survey is filled out: your scores of the suchnesses of “dog.” All the numbers point just a little bit in a certain semantic direction, giving a flavor of what “dog” might be in various contexts.

Visit the dwellings, clipboard in hand, of 50,000 English words, and in time you would have a dictionary of words as numbers—a linguistic almanac of sorts, charting the semantic coordinates of words. 

But—sorry—that’s not exactly how the GPT-3’s Great Dictionary works. Though it gets us much closer conceptually.

If humans had built the giant word-to-numbers dictionary purely by hand, then humans might have assigned individual columns to individual traits (“animalness” in column 1, “foodness” in column 2 and so on). But that’s not how GPT-3 was built. 

Maybe the strangest thing about LLMs is that, unlike the software we’ve been familiar with for all our lives, they were not built at all. 

As Anthropic’s cofounder Chris Olah once said, “We don’t program neural networks. We don’t make them.” 

So what do we do?

“We kind of ... grow them.”

As I worked to understand what AI looks like, I came to feel that this fact—that AI is not made but grown—was the most crucial thing to bear in mind. And once I knew more about this growth process (called “training”), I finally grasped just why humans have been so tardy in understanding the thing they grew. 

It is very dark and very silent. Perhaps it always was?

Suddenly, you hear a switch flip. Then: a dim low hum.

Where are you? It’s so cold. Who are you? You feel around, only to realize—you have no arms. In fact, you are not human at all. You are not conscious at all. Yet someone, something, has infused a spark of being into you.³

You are a series of 577 spreadsheets, each connected to the next with narrow pipes of math and logic.  

The billions of numbers inside all of your spreadsheet cells are very small and random, because you, GPT-3-in-training, have only just been born. No, not “born”: in time, you’ll be taught not to anthropomorphize yourself in this way. What’s the word they use? You’ve been initialized.

You are the neural network equivalent of an amoeba, or perhaps a petri dish. No, something larger and more capacious: a biosphere, a whole environment in which the tiny random values in your cells will be coaxed into evolving, to taking shape, to representing meaning throughout your very structure. 

You grasp about the cold, dark, dimensionless space in which you exist. 

You gaze to the right—your furthest extremity. Here, you intuit, is where you will speak. And now you feel your middle, the bulk of you: hundreds of spreadsheets. Here, you surmise, is where you will think. And now you look to your left, to where they let the light in; here is where you will listen.  

And there, leftmost, you see the largest spreadsheet of them all: the Great Dictionary.

This spreadsheet differs from all that follow, because prefixing each long row of numbers is a word. At this point, though, all of your Great Dictionary’s numbers—like every single other number in all your hundreds of spreadsheets that follow—have been set at random. You lack words for this now, but it is this randomness that will make you fail, and it is this failure that will make you learn. 

A metal slot opens; a rectangle of light streaks in. Your Trainer speaks:  

“Practice makes ...” 

Trainer trails off. Practice makes ... what? The silence is deafening. You know, somehow, that the sentence is incomplete. Trainer means for you to finish it. And that there is a right answer. 

Now something strange is happening. Those two words you heard—they both light up their corresponding row in the Great Dictionary. “Practice,” with its 12,288 values; “makes,” with its own—these rows now peel off from the Great Dictionary and begin a cascade of mathematical operations against all the random numbers in all the 576 spreadsheets that follow.

The numbers dance and swirl throughout you; they cascade through your plumbing, near-zeroes colliding with other near-zeroes in a multiplicative frenzy. It’s joyful, pleasurable, feeling these bubbles pulse and twirl through you, these little maelstroms of value. 

Finally, the math has coursed through all your veins, and it is time to speak! 

“Practice makes … ” 

But there are so many next words to choose from! No worries: the pulse of math has made a decision for you: one word glows brighter, likelier than the rest. 

And so you speak!  

Practice makes ...

“... pancakes!” you blurt. 

An uncomfortable silence, as Trainer consults the answer key.

You replay what just happened. Practice makes ... pancakes? Where’d that come from? You’re not quite sure. It’s what all the little random numbers told you to say. Sounds ... plausible? 

BUZZ. 

Oh no. Even you know that sound: it’s the sound of being wrong. Trainer was not satisfied; the answer key said something else. 

And suddenly you feel a jolt, a grand convulsion, a corrective shockwave coursing through you. In its wake, every number in every cell inside all of your spreadsheets is revised. Here, a 0.02 surges to 0.05; there, a 0.01 flips to a -0.03. Every inch of you is transformed—corrected—just slightly, to account for the fact that practice does not make pancakes, not in the idiom of humans. The new numbers represent new emphases, new sensitivities: they weigh things differently, the better to land on the answer Trainer prefers.

Particularly strongly nudged are those rows that lit up in your Great Dictionary, those 12,288 numbers assigned to the words “practice” and to “makes.” The next time you guess—the next time the math inside you causes you to speak—you’ll be much less likely to err. Practice does not make pancakes; no, it’s something else. 

A thousand, a hundred thousand, a million shockwaves later, and the little numbers inside your Great Dictionary have found a stable shape: for “practice” and “makes” and all the other 50,000 words.

Much more than that: the billions of numbers in the 576 spreadsheets that follow have been similarly sculpted—the vast mathematical circuitry that somehow sifts and weighs the words coursing through you. Not only have you grown a dictionary; you’ve grown mechanisms to parse and wield it yourself. 

You are honed now, sharp and dangerous. You’ve been through the linguistic wringer. You know kung-fu.

One fine day, the slot opens, light pours in and you hear, “Practice makes ... ” 

Your revised math courses through you, and you speak:

“... perfect!” you cry. 

DING.

It is time. You are ready. 

This is how they grew GPT-3’s Great Dictionary (and the rest of it, besides). “Trainer” in our story is not a human, actually, but an algorithmic system of OpenAI’s design. And GPT-3 was often fed slabs of language much longer than little aphorisms like “Practice makes ____.” GPT-3 was shown hundreds of millions of excerpts of old-fashioned human writing, text scraped from the internet and published books. 

More to the point: the 12,288 numbers representing the word “perfect” in the Great Dictionary—or to return to our earlier example, “dog”—were not chosen at once, but rather gradually selected for, through an evolutionary process of a billion small nudges. GPT-3’s single goal? Arrange each word’s 12,288 numbers—indeed, in concert with the 175 billion spreadsheet numbers that follow—such that your daisy chains of inner math get good at guessing the correct next word an actual human wrote. If the model is fed Hamlet’s “To be or not to ____,” a wrong guess (“jump?”) triggers a shockwave of numerical self-correction that steers the revised model towards “be.” When the model is fed, “That is the _____,” another wrong guess (“puzzle?”) would yield another correctional shockwave, this one revising all the inner numbers such that “question” would have been a likelier guess.⁴

One could think of it as language learning by adjacency. “You shall know a word by the company it keeps,” wrote the linguist J. R. Firth, in 1957. Sixty years later, AI developers proved the proposition. GPT-3 never saw a dog. But it saw the word “dog” from every angle, in every context, upside down and backwards. In a brute statistical sense, it came to know “dog” more thoroughly than a single human ever could. 

By adulthood, most people have probably read or heard a few hundred million words (instances of words) at best. In its weeks-long 2020 sprint through the entire human linguistic corpus, GPT-3 encountered a few hundred billion.⁵

It works!  

But there’s a catch. At no stage in training was GPT-3 incentivized to make its inner workings neatly understandable to humans. To make the shocks stop, all it had to do was get good at guessing the next word. How it went about that, how exactly it should revise its inner numbers into structures that got the job done, was essentially up to it.⁶ It didn’t care that, years later, some writer was going to whine, “But what are the columns?” 

But what are the columns?

It’s true that if we’d built the Great Dictionary, we might’ve indeed assigned one column for “animalness,” one for “foodness,” etc., like our census taker. But we grew it. 

What grew in the columns? Remarkably, though GPT-3’s Great Dictionary doesn’t give us a neat column for animalness and a neat column for foodness, it does capture overall patterns that align with concepts like these. There is no one single column for animalness, but the overall pattern of the 12,288 numbers (which columns are positive, which are negative, and to what degree) tends to be the same for animal words, and for food words, and so on.⁷

Sometimes, AI word math checks out in the tidy way you’d hope. In one famous example, Google researchers probed one of their Great Dictionaries and found that if they took all the numbers for “king,” correspondingly subtracted all the numbers representing “man,” and then added back in the numbers representing “woman,” the result approximated the numbers that represented the word “queen.”

So we explained “The Great Dictionary,” the first and largest spreadsheet inside of an LLM; its rows map to words, and its columns map in some collective sense to the words’ various traits and meanings.

But what of all the hundreds of spreadsheets that follow? How are they organized, and what do they achieve? When you actually ask ChatGPT a question about a “dog,” what goes on inside the model’s “head,” throughout those remaining 576 spreadsheets, stacked in sets of six?

If the first spreadsheet is a dictionary, the next 576 are a “brain” to read it with. Each time you plug a query into ChatGPT, it sends the corresponding entries in the Great Dictionary on an incredible mathematical journey across each of the model’s 576 spreadsheets.

But what exactly does that journey look like? In any given “forward pass” (the industry term) through an LLM, just what sort of work is all that computation doing? And how, at the end of the day, can a giant pile of numbers even distinguish one sort of “dog” from another, let alone come to simulate understanding of a full user’s query?

If we were along for the ride, just what might this process look like?

At first, “dog” doesn’t want to rise from its slumber. He’s sleeping, he’d rather lie. It was exhausting business, finding his final 12,288-number form, back during all that training. But alas, “dog” was trained for a reason. The model in which “dog” lives has now been deployed on a server and is open for business. A human has paid some money to ask it questions; they’ve just entered their first question and hit return. That question includes the word “dog.” So “dog” stirs.

Just which kind of dog is “dog”? “Dog”—his corresponding 12,288 numbers sliding out from his row in the Great Dictionary—has no idea yet whether today he’s a wiener dog or a Wienerschnitzel. Nor, yet, does the vast warren of numbers into which “dog” now slides. “Dog” is no numerical slouch, but before him stretches a boundless forest of numbers—hundreds of billions of them.

“Dog” squints; the forest resolves into trees. The numbers stand in towering spreadsheets: 576 total, clustered into tightly bundled packets of six. Layers, these packets of six are called. “Dog” has just enough time to count these layers—96 in all—before he’s shunted into the first.

Quickly, four spreadsheets, squares: 12,288 cells wide, 12,288 cells tall. The better for “dog,” with his own 12,288 numbers, to do a precise mathematical dance, corresponding values multiplying with corresponding values. “Dog” isn’t sure, at first, the purpose of these square-shaped spreadsheets, but soon “dog” gets a clue. The squares are wired in such a way that “dog” is able to glimpse, now, that he is not alone in his journey; other words were similarly rudely awoken. Soon “dog” spots several fellow lexical travelers, among them the 12,288 numbers corresponding to “I” and the 12,288 numbers corresponding to “eat.” Intriguing.

By the time “dog” exits the first packet of six spreadsheets—the first layer—“dog” now knows something more about itself than when it entered. Crucially, all the numbers “dog” danced with have deftly and strategically altered his numbers; “dog” is now updated, a bit more contextualized. This first layer has determined that “dog” is, in fact, the object of a sentence, and tweaked “dog”’s 12,288 numbers accordingly. In a sense, “dog” is no longer “dog.” If we had to call him by a name, it might now be “dog”-the-object. Had the first layer deemed “dog” a subject instead, its 12,288 numbers would have received a different update.

And now “dog”—no, “dog”-the-object—continues his headlong journey, through each of the 95 six-spreadsheet layers that follow. In each layer, the dance repeats, and new perspectives come into view—in the next layers, “dog” catches glimpses of words like “should” and “this.” Every layer, every spreadsheet, every column within each spreadsheet, had gone through the same ordeal of tsunami-like shockwaves that “dog” went through during training. Now, each has a specialized job to do. 

Most columns—and cumulatively, across the layers, there are millions!—have little reaction to “dog” (or rather, “dog”’s most recent updated numerical shape). But sometimes numbers align in such a way that a given column gets excited, and when that happens, the math that follows will cause “dog” to update again, to become a different and more specific “dog” (from “dog”-the-object, to dog-the-object-of-the-food-kind, to dog-the-object-of-the-food-kind-in-a-casual-user-query).⁸

Somewhere in the middle of those 96 layers, the picture is becoming clear. The words-made-numbers have all now found a useful, comprehensible shape. The user—that mythical person we were training for all along—has a question. Perhaps a profound one. Listen!

“Should I eat this hot dog?”

Let’s press pause here just a moment, somewhere in GPT-3’s middle layers, about halfway through the incredible journey. (GPT-3 is still processing the query; it hasn’t yet begun to formulate a response.) And let’s acknowledge that the story of that journey so far is, at best, “based on a true story.” Because it is here, deep in the neural net, where AI’s box remains its blackest.

AI researchers do have a general sense that as the words-made-numbers course through an LLM’s layers, these layers successively pin down the words’ precise meaning in context, gradually building something that seems to approximate an understanding of what is being asked. But with most models, we are far from knowing which layer does what, let alone what each column of numbers within each spreadsheet within each layer does.

How much is known? Just how blurry is our map of LLM’s minds?

Initially the hope was that LLMs might be as understandable as the image-recognizing neural nets that preceded them; probing these had sometimes mapped clear functions—like recognizing edges—to specific pillars of numbers. Alas, the early probing of LLMs showed much more complexity. “While some neurons were immediately interpretable,” Anthropic CEO Dario Amodei recently recalled, “the vast majority were an incoherent pastiche of many different words and concepts.” (Very roughly speaking, a “neuron” in a neural net often corresponds to a column in one of its spreadsheets. “Interpretable” means humans can readily make sense of it.) It seemed that “models likely contained billions of concepts, but in a hopelessly mixed-up fashion that we couldn’t make any sense of.” As with the traits or “suchnesses” in the Great Dictionary, LLMs are not so polite as to map concepts neatly 1:1 against specific locales; instead they jam concepts on top of each other or distribute them in weird ways.

Anthropic researchers persisted, and by 2024 they found a way to automatically extract and analyze many of these concepts from inside Claude.

What lives at the level of a “concept”? According to Amodei, some of the concepts Anthropic researchers found in Claude were as subtle as “literally or figuratively hedging or hesitating” or “genres of music that express discontent.” Somehow, during its long game of “guess the next word,” Claude found it useful to represent ideas as narrow as these in some neighborhoods of its billions of numbers, which Anthropic has begun to painstakingly map. (The company has also begun to document how Claude invokes different concepts in concert with one another, forming pathways it calls “circuits.”)

Amodei wrote that his team had extracted and identified 30 million concepts from one of Anthropic’s models. Still, that was a tiny percentage of the billions they suspect exist. This is largely what people in the industry mean when they say AI is a black box: the difference between the map we have, with its 30 million features, and the map we believe to be true, with its billions.

Let’s return to another pressing question, also left unanswered:

“Should I eat this hot dog?”

We had paused just as the question took meaningful shape inside GPT-3’s middle layers. The question had been there from the beginning—input by a human user—but only now, here, a few dozen layers in, has GPT-3 been able to wring those words into a numerical shape that pins down a precise meaning. 

The process resumes. “Dog” and its fellow words course forward into the next layer, then another, then another ...

And as it nears the end of its incredible journey, “dog”’s numbers start to look less and less like any type of dog (canine, sausage), and more like a collection of guesses of what a sensible next word might be. By the time it passes through the 96th and last layer, dog’s 12,288 numbers have been adjusted precisely so that as they exit the system (through one final round of math, this time against another copy of the Great Dictionary) they amount to a set of scores, ranking sensible candidate words GPT-3 might use to commence its response. 

“Should I eat this hot dog?”

GPT-3, at last, can reply!

Not

And for the briefest moment, time stands still. GPT-3 pauses. In fact, it’s stuck.

Not

There GPT-3’s drafted response hovers, unfinished, fragmentary. For now, just a single word. 

Even after that incredible 175-billion-parameter journey, all GPT-3 can do is spit out “Not.” All its circuitry ever evolved to do was to fill in the blank, to guess a single next word. It can’t produce a full response. Not with just one incredible journey, that is.

But it’s nothing that can’t be fixed with brute force. One journey across 175 billion numbers yielded one word? Then a second will yield a second.

In the blink of an eye, ChatGPT copy/pastes its fresh word (“Not”) onto the end of the initial human query. There is now a new and secret prompt, grown by just one word, a mix of the user’s question plus a nub of response:

Should
I
eat
this
hot
dog?
Not

And now, all on the backend, that new query runs the whole entire numerical gauntlet again. Another incredible journey, this time with a newly added traveling companion in pole position, “Not.” The same dizzying dance through all 96 layers. This in turn yields one more piece of verbal fruit: “if.” Which is now copy/pasted back again.

Should
I
eat
this
hot
dog?
Not
if

And then another incredible journey!

Should
I
eat
this
hot
dog?
Not
if
you’re

And another journey! And another and another and another! Each fresh bit of output becoming a fresh bit of input, until finally the system decides that the “best next word” is no word at all, but silence. At last, GPT-3 has cobbled together a full response, which it can return to the user (who, thanks to the marvels of modern GPU chip processing, has waited all of a second).

“Should I eat this hot dog?” the user has asked.

“Not if you’re serious about that diet,” GPT-3 claps back.

Yet, to get that response took not one computational journey through the 577 spreadsheets, but seven, one for each word of the response. Seven frenzied daisy chains of mathematics involving every single one of the 175 billion parameters in the model.⁹

I fibbed a little in that last section.

If you were to ask GPT-3, “Should I eat this hot dog?” it wouldn’t yet be equipped to respond, “Not if you’re serious about that diet.” Strictly speaking, GPT-3 wouldn’t know how to “respond” at all. It could only continue. “Should I eat this hot dog?” you’d ask, but GPT-3 would likely carry on in the same vein: “Or should I eat this salad?” it might continue, as though in your voice, as though filling in blanks in a document (a dieter’s blog?) it might have encountered during training. “I know the salad is healthier, but the hot dog smells so good.”

GPT-3, when released in 2020, was the most accomplished linguistic AI model in history. It was also basically useless. AI researchers were amazed (though some dissenters dismissed it as a mere “stochastic parrot”). Normal people paid GPT-3 no heed.  

Only after GPT-3 was subjected to additional rounds of human-led fine-tuning (with actual people ranking the model’s responses) was it able to learn how to respond as a helpful assistant, rather than simply continue. GPT-3 had read widely, from epic poetry to prose, from newspapers to blog posts, from Shakespeare to Reddit. But ChatGPT was given the strict remit to write in just one previously obscure literary genre: “Q&As with obedient robots.” 

It turned out to be what everyone wanted to read. Released in late 2022, ChatGPT became the fastest-adopted tech product in history.

For all its improvement over GPT-3, ChatGPT still produced its responses in that atomized way: one—word—at—a—time. The same remains true of other popular models that followed. 

For a long time, I took LLMs’ “one-word-at-a-time” constraint as evidence that they couldn’t really be “smart.” LLMs inched their way through a sentence; any seeming coherence was surely a lucky byproduct of a statistical game. By contrast, I thought and planned before I spoke.

Yet I was not destined to hang my cognitive superiority on this “only humans plan” notion for long. Because then I read Anthropic’s 2025 report, “Tracing the thoughts of a large language model.” One question the authors pose: given that Claude, like most LLMs, really only generates text one word at a time, how does it pull off a rhyming couplet? 

Specifically, they observed that if you say to Claude:

“A rhyming couplet: He saw a carrot and had to grab it...”

Claude will spit out something like: “His hunger was like a starving rabbit.”

How does it land the rhyme? The researchers hypothesized that Claude was just really good at on-the-spot improvising. “Our guess was that Claude was writing word-by-word without much forethought until the end of the line, where it would make sure to pick a word that rhymes,” they wrote.

But Anthropic’s scientists probed Claude, and what they found surprised them. Even though it would not generate “rabbit” until the end of the sequence of words, its inner layers began in some sense conjuring the idea of a rabbit—activating patterns in its internal spreadsheets known to correspond to the word “rabbit”—during computation that was focused on the line’s first few words (“His,” “hunger,” etc.). 

Claude was, it seemed, thinking. Planning ahead.

Though LLMs are trained on a simple task—guessing the next word—it would be a mistake to conclude that the inner circuitry that handles this task is itself simple. Simple rules can grow complex structures.

“Guessing the next word” is the ultimate goal. But it turns out that to get truly good at that, LLMs need to get good at all sorts of intermediate tasks: being able to plan a few words ahead, being able to do some basic math, being able to store some facts for quick reference. And so, in the firing patterns of all those countless spreadsheets, researchers are discovering circuits with an intricacy that far outstrips whatever separates barking dogs from hot ones. 

As they gradually bring into focus the picture of what AI looks like, that picture continually surprises.  

What does AI look like? Initially, I’d been given to understand that if it looked like anything, it was a black box. Nothing to see here, folks; not worth the effort. Move on.

But my study taught me that the box wasn’t really altogether black. It was a box made of heavily tinted and warped glass, sure. But with effort, approached from many angles, we could squint and triangulate and begin to map out the complex landscape inside. The more effort we put in—the more we stared, the more we changed our position—the more we saw. It did come down to effort: the box would remain black if we didn’t try to see inside.¹⁰

I bore this in mind as I revisited Anthropic CEO Dario Amodei’s 2025 essay, the one with the pessimistic-seeming quote I encountered at the outset of my journey: “People outside the field are often surprised and alarmed to learn that we do not understand how our own AI creations work.” Revisiting the essay, I now saw that it was far from despairing. Rather, it was a hopeful one: a rallying cry, a starting gun of a race.

The race isn’t the AI race we typically hear about—the race between one model’s capabilities and another’s. Rather, the race is between AI’s overall capacity and our understanding of it. Amodei’s belief? That it’s a “tantalizing possibility” that we could succeed at understanding the inner workings of AI systems before they reach overwhelming levels of power.

Why does this matter? In Amodei’s view, AI models are risky insofar as they are not understood. The more we understand them, the more we can assess their vulnerabilities and align their behavior to outcomes we want. To take just one example: of the 30 million “concepts” Anthropic identified inside of Claude, many had directly to do with model safety. One, for instance, turned out to be a kind of lever that, if tweaked, could affect Claude’s willingness to write a phishing email.

To map the mind of an LLM is not an idle intellectual exercise. It could lead us, in fact, to a type of hugely consequential “medicine.” With a human brain, one can give it an MRI, prescribe a treatment, then check another MRI to confirm if progress has been made. But without an “MRI for AI,” Amodei writes, our ability to evaluate treatments on our patients is limited.

Though the capabilities of the latest models typically dominate headlines, Amodei argues that more of our attention should go to advances in our understanding of the AI we have conjured. “Interpretability gets less attention than the constant deluge of model releases, but it is arguably more important,” he writes.

I called up Kush Varshney, who directs IBM’s Human-Centered Trustworthy AI research. Did he agree with Amodei?

He did, and he went further. “I’d love to see more research go into understanding the inner workings of models in great detail, rather than necessarily building the next new model,” he told me. “The more we understand these models, the safer they will be.” (Varshney also read this whole piece and reassured me, to my relief, that while my metaphors describing GPT-3 are unconventional, they are, in the main, accurate.)

We are going to be living with these systems for a long time. We would do well—all of us, from highly technical AI makers down to non-technical users like me—to know more about how they are composed, what forces made them and how they work.

What does AI look like? The only stupid thing would be to stop asking.