In 2023, Tigges et al. described a pattern they called the summarization motif: as a language model reads a clause, some of its attention heads gather up the gist of it and store it on the punctuation mark at the end, so that later text can consult the comma instead of re-reading every word. That made me wonder what a model does when the commas aren’t there at all. There is a practical angle too, since dictation tools produce text with odd punctuation, and some (Wispr Flow, in my experience) capitalize whichever word follows a pause, mid-sentence or not. I was interested in whether this would impact model performance.
Anecdotally, it seemed unlikely in frontier models because they don’t seem to care much about missing punctuation, typos, etc. Still, it seemed conceptually interesting to imagine how they might summarize with punctuation.
This is a small project, using synthetic text and eight open base models from 124M to 8B parameters, run on my M5 Pro Mac and some rented time on an H100. In particular, it looks inside small models.
TL;DR
With punctuation present, models do use it as a summary point for information spread over several words (a mood set by three adjectives), though not for a fact carried by one word (the country a city is in). Six of the eight models I tested have a punctuation “hub” like this.
When I removed the punctuation (and sentence-start capitals), no single word took over its job. The models mostly read the original words directly instead, although in the larger ones a thin route spread across the passage partly makes up for it. A simple classifier (a probe) can still find the information almost anywhere in the passage; the model just doesn’t read it from there.
Over a few sentences, removing punctuation cost little, and how much depended on how the question was asked. Over 900–1,500 tokens it cost the larger models more, and Llama-3.1-8B more than half its mood margin, about twice what a word-swap control cost. The models hurt most are also the ones that keep the punctuation hub at that range, but they are the largest ones too, so I can’t say the hub is why.
Setup
Each passage has a first sentence that sets a property, 1 to 16 sentences of neutral filler, and a readout at the end that asks for the property. I used two properties:
Country, carried by one word: the first sentence mentions a French or Japanese city (“Our flight landed in Bordeaux right on time…”), and the model has to answer France or Japan.
Mood, spread over three words: the first sentence has three adjectives that are all positive or all negative (“At the bookstore the selection was delightful on the shelves, the clerk was impressive at the counter, and the reading corner was solid in the back.”), and the readout “and overall the whole experience was” is scored on “ great” versus “ terrible.”
I rendered every passage five ways:
normal text;
with no punctuation, and with sentence-initial words lowercased, since a capital is a boundary cue too;
with the punctuation put back in the wrong places;
with a few random dictation-style capitals;
and a control with some words swapped for random ones, tuned to be about as surprising to the model as the unpunctuated text.
(The argument being that models do worse on any strange text, so if removing punctuation costs more than the control does, the punctuation itself is doing something.)
The score is the margin, the difference in logits between the right and wrong answer; a margin of +1.67 means the model favors the right answer by about 5 to 1. To look inside, I used three tools. Attention knockout stops the readout’s tokens from attending to chosen positions, at every layer, while the rest of the passage can still read them. So I can ask how much of the margin survives when the readout can’t see the first sentence’s words, and how much more is lost when it can’t see that sentence’s punctuation either. Activation patching copies the internal state at one position, or a set of positions, at one layer, between two passages that differ only in the property, and measures how much of the answer moves with it. Linear probes check where the property can be read out at all. I built everything on GPT-2 small to validate the method, and then ran it on Pythia (160M to 6.9B), GPT-Neo-125M and Llama-3.1-8B.
Results
Punctuation is a summary hub, for some information
For country, the model mostly just looks back at the city. Blocking GPT-2’s readout from the first sentence’s words removes about 95% of its margin, and the city’s own position carries about 80% of the effect under patching. That is unsurprising, since models are known to attach facts like this to the word itself (Geva et al., 2023).
Mood is different. In GPT-2, blocking the readout from every word of the first sentence still leaves about half the margin, and blocking its two commas and period as well removes most of the rest, so those three marks carry about 38% of the margin by this measure. Patching agrees on where the route is: copying just those three marks recovers 15–20% of the answer, while all sixteen filler sentences together recover only about 6%. The two numbers measure different things, so I don’t think either is the true share. Six of the eight models have a hub like this (all but GPT-Neo and Pythia-160M). In those cases, it carries 20–42% of the mood margin by the knockout measure.
Without punctuation, no word takes its place
In the unpunctuated text, only 9% of GPT-2’s mood margin survives when the readout can’t see the first sentence’s words, compared with about half with punctuation. Copying everything between the adjectives and the readout recovers only 3–7% of the answer, and no single position recovers more than 6%. The adjectives themselves matter more instead, their patching recovery rising to 52% at sixteen filler sentences from 33% with punctuation. In other words, GPT-2 doesn’t move the summary somewhere else; it goes back and reads the adjectives. Tigges et al. had more or less anticipated this, since they froze attention in some experiments precisely so the model couldn’t route around the commas. As far as I could find, earlier work either interfered with punctuation’s activations while leaving the marks in the text (Tigges et al.; Chauhan et al., 2026) or removed punctuation and measured only accuracy (Razzhigaev et al., 2025), and I didn’t find anyone who had removed it and looked for where the information goes.
The larger models are less tidy. In none of them does a single word take over: without punctuation, the best single position recovers at most 4% of the answer, while the period alone recovers 6–20% with punctuation in every model but Pythia-2.8B (more on that below). In Llama and Pythia-2.8B, the words just after the missing commas do pick up one to three percentage points, which is the only relocation-like signal I found. But in Pythia-2.8B and 6.9B, at sixteen filler sentences, the passage as a whole carries about as much without punctuation as with it (around 10% of the answer), spread thinly over many positions. That is something like the diffusion I had guessed at going in, and it seems to appear only in the larger models over longer passages.
The probes, meanwhile, found the mood decodable in about 90% of the positions and layers I checked, with or without punctuation. The causal methods say the readout uses only a few of those places.
The behavioral cost is small, except over long passages
Over short passages, removing punctuation didn’t hurt the mood task. In fact, the margin went up slightly in seven of the eight models, which may partly be because the mood readout reads a little oddly right after a period. On the country task it cost little. In GPT-2 that cost was clearly about the punctuation itself, seven times what the control cost on the question format (twice on the continuation). In the other models it depended heavily on how the question was asked. In Pythia-6.9B, for instance, removing punctuation raised the margin when the country was asked as a question on a new line, and lowered it when the passage was continued (“and all of this was in the country of”), on the same passages. I have a few ideas about why, but none that fits every model.
Long passages are a different story. With about 900 or 1,500 tokens of book text (from PG19) between the first sentence and the readout, the larger the model, the more removing punctuation hurt. At about 900 tokens the change in the mood margin runs steadily from +0.16 in Pythia-160M to −0.18 in Pythia-6.9B (a third of its margin), and Llama’s falls by more than half, from 1.70 to 0.76, while the word-swap control only brings it down to 1.24. In Llama and Pythia-6.9B, putting the punctuation in the wrong places hurts exactly as much as removing it, while stray capitals cost nothing. Those two are also the only models that keep the punctuation hub at this range (the others’ hubs mostly disappear, and most of them barely do the mood task at all). Pythia-2.8B, whose hub is gone at this range, also loses a quarter of its margin.
Other findings
On the country task, asked as a question, misplaced punctuation was the most damaging condition in seven of the eight models. When they’re all the readout can see, misplaced marks carry some of the mood (as much as correct ones in GPT-2, less than half as much in the larger models), but in the intact model the readout mostly ignores them.
Stray capitals were mostly harmless. They’re about as surprising to GPT-2 as removing all punctuation, but the hub survives them in every model. Keeping real sentence-start capitals in unpunctuated text, on the other hand, partly stands in for the punctuation in the larger models.
Hub strength isn’t a simple function of size or data. At about 125M parameters, GPT-2 (trained on WebText) has a strong hub, while GPT-Neo and Pythia-160M (both trained on the Pile) barely have one, which points toward training rather than tokenizer or architecture. Pythia-410M has a hub on the same data, though, and within Pythia, where every model saw the same data in the same order, hub strength goes up and down with size.
Pythia-2.8B, the model Tigges et al. studied, has the weakest punctuation hub of the larger models, apparently because it keeps its boundary summary one word later, on the first word of the next sentence rather than on the period.
Caveats
This is a narrow setup. The text is templated, there are only two properties, and the models are small base models. The mood results rest on one readout, which reads slightly oddly after a period and may flatter the unpunctuated text, as may the fact that unpunctuated passages are 10–15% shorter. The knockout percentages are measured with the main route already blocked, so they aren’t shares of the intact model’s margin that add up, and my patching works one layer at a time, so it can miss routes spread over several layers. The long-range link between the hub and behavior is a comparison across models that also tracks size; I didn’t block only the punctuation within a model.
Conclusion
I suspect models generally have more than one route to the same information, so that removing one mostly shifts the work onto another. Over short passages the punctuation route seems to be a convenience; over long ones, at least in the larger models, it looks more important. Before running the larger models I wrote down a prior that they would need punctuation less. At long range they seemed to need it more. I cannot extend these results to the largest models, but that is the question I’d most like to see answered in frontier models, along with whether stray dictation capitals matter in the models people actually use.
This was a lot of fun, and a good education in methods.
The code, every per-model result and a dated log of every design decision are on GitHub if you want to do your own analysis.
Disclaimer: I conducted this research a few weeks ago, but have been lazy about writing it up. Accordingly, I relied on Claude heavily, much more than usual, to draft this post.