The Cousin Mysteries

Or: A Brief History of Inferring What We Cannot See

The first thing worth knowing is that most of the room you are reading this in has gone missing. Not the chair, not the laptop, not the half-finished glass of water; those are accounted for. The trouble is that whatever does the accounting (the mass, the gravitational pull on the floor, the total energy budget of the cubic meter you occupy) does not add up. Roughly five percent of the cosmos is the stuff you can touch and stub your toe on. The other ninety-five we have agreed, by some combination of convention and resignation, to call dark.

The word is doing a lot of work. There are two darks, technically, and they are different. Dark matter pulls galaxies together. Dark energy pushes them apart. Both are invisible to the instruments we have, and both have been given the same adjective despite doing opposite things, the way you might call two cousins by the same nickname because you cannot keep them straight. Together they constitute about ninety-five percent of everything that is, which means our entire physical account of the universe is anchored in a five percent minority. This is the situation. Nobody really knows how to feel about it.

I think we should feel weird about it, and I think part of the weirdness is that the cousins do not arrive in our heads as raw observations. They arrive, like everything else in cosmology, through a chain of inferences, instruments, calibrations, and choices, and the choices have fingerprints. The names "dark matter" and "dark energy" feel like answers, but they are placeholders sitting at the end of a long lineage of decisions. We will come back to this. For now, just notice that the cubic meter you occupy is mostly something we have inferred rather than seen.

A short note on the vocabulary, because the vocabulary turns out to be telling. Dark matter is the term Fritz Zwicky used in 1933, in German (dunkle Materie), when he looked at the Coma cluster and concluded that there had to be roughly five times more matter holding it together than was visible. Dark energy is much newer, coined in 1998 to describe whatever was making the universe expand faster than it should be. In between, physicists have proposed an entire bestiary of unseen things, with names that read, on inspection, like the contents page of a poorly translated metaphysics text. Quintessence, from the medieval fifth element. Phantom dark energy, which is exactly what it sounds like and which the literature treats as a serious technical term.¹ The cosmological constant problem, a problem so old it has its own honorific. Cosmologists have been reaching, for roughly a century, for words that already meant ghost or spirit or placeholder, and we should probably notice that this is not an accident. When a field runs out of nouns, it starts borrowing from theology.

The Man Who First Said It

The man who first proposed dark matter had a difficult personality. Fritz Zwicky was Swiss-Bulgarian, worked at Caltech, and published roughly a thousand papers across a career in which he proposed (correctly) that supernovae are powered by neutron star formation, (correctly) that gravitational lensing should be observable, and (correctly) that a substantial fraction of the universe is invisible to telescopes. He also proposed several things that turned out to be wrong, but the more relevant fact about him, at least for the social history of cosmology, is that he was famously unpleasant. He coined the term spherical bastard to describe a colleague who was a bastard from any angle you looked at him, which is the kind of thing one says about other people when one has decided not to be nice.²

In 1933 Zwicky measured the velocities of galaxies in the Coma cluster, did the math, and found that the cluster was held together by gravity considerably stronger than its visible matter could supply. He concluded that there was dunkle Materie, dark matter, doing the heavy lifting. The result was published in a Swiss journal, and the field, by and large, ignored it for forty years.

It is hard to say how much of the ignoring was about the result and how much was about Zwicky being an ass. Probably both. He was abrasive, the result was uncomfortable, the math was not airtight (he had the wrong value for the Hubble constant, which threw his numbers off by a factor that other astronomers found suspicious), and the broader culture of physics in the 1930s was not especially eager to take cluster dynamics seriously. The combination meant that one of the most important observations in twentieth century astronomy sat in the literature for four decades, periodically rediscovered, periodically forgotten, like a book on a shelf that everyone walks past.

The person who eventually forced the field to listen was Vera Rubin. In the 1960s and 70s, working with Kent Ford and a high-resolution spectrograph, Rubin measured the rotation curves of spiral galaxies, which is to say she measured how fast the stars in different parts of a galaxy were going around the galactic center. Newton predicts that the stars on the outside should be moving slower than the ones on the inside, the way the outer planets in the solar system orbit more slowly than the inner ones. Rubin found that the rotation curves were flat. The stars on the outside of a spiral galaxy were going just as fast as the stars closer in.

There are exactly two ways to explain this. Either Newton's law of gravity is wrong at galactic scales, or there is much more mass in those galaxies than we can see, distributed in a way that keeps the outer stars moving fast. Rubin's measurements, taken across many galaxies, made the second explanation hard to avoid.

She did this work without access to the Palomar Observatory, which was, by policy, closed to women until 1965.³ She had to lobby for time on lesser instruments. She published carefully, repeatedly, with overwhelming data. By the early 1980s, the consensus had shifted: dark matter was real, or something a great deal like it was, and Zwicky had been right almost half a century before any major instrument was available to confirm him. Rubin never won the Nobel Prize. She died in 2016, at eighty-eight, with the question of what the dark matter actually is still open.

The Year Nobody Believed Their Own Data

The other cousin appeared, less expected, in 1998. Two competing teams of astronomers had spent years measuring Type Ia supernovae, which are useful because they are very bright and very consistent in their intrinsic luminosity. (When a particular kind of white dwarf accretes enough matter to cross a critical threshold, it explodes in a way that produces almost the same total light output every time. This makes it a standard candle, and standard candles let you measure cosmic distances, which is one of the few things in cosmology that does what its name says.) The supernova distances themselves were calibrated, several handoffs back, against Cepheid distances, which were calibrated against Henrietta Leavitt's period-luminosity relation, which she had worked out at thirty cents an hour. Every result either team published was, in this sense, a result of hers too. Both teams expected to find that the expansion of the universe was slowing down, because gravity should be putting the brakes on the universe just as it puts the brakes on a thrown ball.

What both teams found, independently, was that the universe was speeding up.

Adam Riess, then a young researcher on the High-Z Supernova Search Team, has talked about going home from the office with this result and not being able to sleep. He went back to the data the next morning and the morning after, hunting for the error he was sure he had made. He sent emails to colleagues asking them to talk him out of his own findings. Saul Perlmutter's team, working separately, found the same thing. Two groups of people, who did not particularly like each other, had independently spent the better part of a year recalculating in an effort to find a mistake, and had failed.

This is the moment dark energy enters the literature. The universe is not slowing down. The universe is accelerating. There is something pushing it apart, a pressure or a substance or a property of space itself, and we cannot see what is doing the pushing. The Nobel Prize for the discovery went to Riess, Perlmutter, and Brian Schmidt in 2011. None of them, then or now, can tell you what dark energy actually is.

The Chain of Custody

It is worth understanding, at least roughly, how anyone measures the universe well enough to be wrong about it. The technique is a chain of custody. To know how fast a galaxy is receding, you need its redshift (easy, you read it off the spectrum) and its distance (very hard, because nothing in the sky carries a label). The distance has to be inferred, and the inference is built handoff by handoff, each handoff documented, each handler adding their fingerprints.

The first handoff is parallax: the apparent shift of nearby stars as the Earth moves around the Sun. This works for stars within a few hundred light years and is the only direct measurement we have. Beyond that, you need a candle.

Leavitt was the handoff that turned the supernova distances into anything you could trust. In 1908, working as a computer at the Harvard College Observatory (the term meant a person, mostly women, hired to do astronomical calculations at low pay), she noticed that a particular kind of pulsing star, a Cepheid variable, had a precise relationship between how slowly it pulsed and how bright it actually was. The slower the pulse, the brighter the star. This was an extraordinary result, because it meant you could look at a Cepheid in some distant galaxy, time its pulse, calculate how bright it really was, compare that to how bright it looked from here, and get its distance. Cepheids became the second handoff in the chain.

The chain continues outward. Cepheids let you calibrate Type Ia supernovae, which let you calibrate the largest redshifts, which let you calibrate the cosmic microwave background, which lets you calibrate, in a manner of speaking, the universe. Every single measurement of dark energy is a measurement made along this chain, and every handoff carries assumptions, some of them disputed, none of them perfect. Break any link and everything downstream of it wobbles. Henrietta Leavitt was paid roughly thirty cents an hour and is sitting, even now, at the bottom of every cosmological calculation anyone has ever published, not because she is lowest but because everything else is downstream of her. She was, among other things, the lowest-paid shoulder the giants ever stood on.

A small detour, while we are doing the history. The first person to propose what we now call the Big Bang was a Belgian Catholic priest named Georges Lemaître, who in 1927 suggested, on the basis of Einstein's equations and some early redshift data, that the universe was expanding from an initial dense state he called the primeval atom. Lemaître was building on, and partly rediscovering, work done five years earlier by a Russian physicist named Alexander Friedmann, who had derived from general relativity the result that a homogeneous universe could not, mathematically, sit still: it had to be either expanding or contracting.⁴ Einstein was unconvinced (of Lemaître, and earlier of Friedmann), and is reported to have told Lemaître, "Your calculations are correct, but your physics is abominable." Lemaître, by all accounts, took this in stride. He was, after all, a priest; he was used to people not finding his cosmology persuasive.

Lemaître's hypothesis came with a prediction. If the universe had really emerged from a hot dense state, there ought to be a faint glow of leftover radiation, cooled by billions of years of expansion into the microwave portion of the spectrum, suffusing the sky in every direction. The prediction was ignored for the better part of forty years, on the grounds that it had come from a priest, until in 1964 two engineers at Bell Labs named Arno Penzias and Robert Wilson picked up a persistent microwave hiss in their radio antenna that they could not get rid of.

They assumed it was caused by pigeon droppings on the dish, cleaned the dish, and found the noise was still there. What they had been listening to, accidentally, was the afterglow of the Big Bang. We now call this the cosmic microwave background; it is the oldest light in the universe and the single most precise dataset cosmology has.⁵ Lemaître was informed of the discovery shortly before his death in 1966. He is reported to have been pleased.

for the first 380,000 years, a hot, opaque plasma fills the universe

The pattern that emerges, walking the history forward, is that the people who first proposed our current cosmological furniture were rarely the ones the field wanted to listen to. Zwicky was ignored partly because he was abrasive. Rubin was excluded from the relevant telescope because of her sex. Lemaître was tolerated as a curiosity because he was a clergyman with equations, which most physicists found a little theological. Friedmann was told by Einstein, in print, that he had made a mistake, and died of typhoid before being vindicated. Leavitt was paid as a Harvard computer at thirty cents an hour. The cosmological history of the cousins is not a clean march of credentialed insight. It is mostly people the field did not want to listen to, telling the field something it did not want to hear, being right about it, often before any major instrument was available to confirm them.

The Vacuum Is the Most Full Thing

Step back from the history for a moment and consider what dark energy actually is, conceptually. The simplest version of the theory says that empty space itself has a kind of latent energy, a constant pressure that pushes outward, present everywhere even where there is no matter at all. The technical name is vacuum energy. The popular name is the cosmological constant, the letter Λ that Einstein originally added to his equations to keep the universe static and then removed when Hubble showed it was expanding. (Einstein supposedly called this his greatest blunder, though the quote is probably apocryphal, attributed to him by Gamow.) The constant has since been put back, because the supernova data require it, which means Einstein was wrong twice in opposite directions about the same letter. Few physicists have managed this. Fewer have done it on purpose.

But sit with the picture for a moment. The current best theory says that empty space has, by volume, the largest energy budget of anything in the cosmos. The vacuum is not a nothing through which things move; it is a something that constitutes most of what there is.

This is, philosophically, an inversion of about twenty-five hundred years of Western thought. Aristotle argued that voids could not exist; later scholastics summarized this as nature abhorring a vacuum. Pascal wrote about the silence of infinite spaces, which terrified him. The history of empty space, in the West, is the history of a thing that was either impossible or terrifying or both. We are now telling each other, with straight faces, that empty space is the most full thing in the universe. Most of what is, is the apparent absence of anything.

It is the kind of fact that sounds technical until you stop to imagine it, at which point it stops sounding technical and starts sounding like something a mystic would say after a long retreat. The vacuum is full. The void contains. The empty is the most.

We Have Done This Before

There is one more thing the history teaches, and it is uncomfortable. The current cosmology is not the first time physics has confidently inferred an invisible substance.

In the nineteenth century, physicists were certain that light required a medium to travel through. Sound waves need air; water waves need water; light waves, by parallel reasoning, must need an ether, a luminiferous something filling all of space and providing the substrate for electromagnetic waves. Maxwell described its mechanical properties. The brightest minds in the field worked out what it should look like. In 1887, Albert Michelson and Edward Morley designed an experiment to detect the Earth's motion through the ether, and found, to their considerable confusion, that no motion through any ether could be detected. The ether did not exist. It had never existed. Almost a century of confident theorizing about the properties of an invisible medium turned out to be theorizing about a thing that wasn't there.

The relevant parallel here is not exactly that dark matter and dark energy might evaporate the way the ether did. The evidence for the cousins is, by most measures, stronger and more diverse than the evidence for the ether ever was. The relevant parallel is that physics, as a discipline, has done this before. We have inferred invisible things, given them names, written equations for their properties, built careers on their existence, and then discovered, occasionally, that we were wrong. Sometimes the invisible things are real (atoms, neutrinos, the Higgs). Sometimes they are not (the ether, phlogiston, caloric).⁶ And the awkward fact is that we cannot always tell, while we are doing the inferring, which kind of invisible thing we are dealing with.

The cosmologists currently studying dark matter and dark energy, which, again, constitutes 95% of the universe, are aware of this. Most of them will admit it if you ask. Few of them lead with it.

The Largest Embarrassment in Physics

Now I should tell you about the largest unresolved problem in physics, which most popular accounts of cosmology politely decline to mention.

If you take the simplest theoretical model of dark energy (vacuum energy from quantum field theory, a calculation any graduate student can do in a weekend) and compute what its value should be, and then compare that value to the value we actually measure, the two numbers differ by a factor of approximately ten to the one hundred twentieth.⁷ That is, the prediction is off by a one followed by a hundred and twenty zeroes. It is not just the largest discrepancy in physics. It is the largest discrepancy in physics by such a wide margin that it has lapped the second-place finisher. There is no other theoretical-experimental disagreement in any branch of science that comes within fifty orders of magnitude of it. We are not close.

Physicists have known this since the early 1980s. The collective response has been, broadly, to keep working on other problems. There are proposed solutions (anthropic reasoning, the multiverse, supersymmetric cancellations that mostly didn't pan out, modifications to gravity that mostly don't work either), but none of them is convincing enough to count as a consensus. The current best guess is that we are missing something fundamental, and the current best plan is to keep collecting data and hoping the missing thing reveals itself. This is, for a discipline that prides itself on precision, an unusually relaxed approach to being wrong by ten to the one hundred twentieth.

I find this funny in a particular bleak way, but it is also genuinely instructive. Physics is not a clean march from confidence to deeper confidence. Physics is mostly working in the gap between what the model predicts and what reality is doing, and sometimes that gap is small and gets fixed by lunchtime, and sometimes the gap is the whole sky, and the sky stays the way it is no matter how long you stare at it.

The Method Speaks

The current generation of cosmologists is doing what most generations of scientists do: refining the methods. New instruments (the Dark Energy Spectroscopic Instrument, the Vera C. Rubin Observatory⁸, the Euclid satellite) are collecting data that previous generations could only dream of. New mathematical tools (neural networks, Bayesian model comparison, simulation-based inference) are being deployed. The hope is that better data and better methods will eventually reveal what dark matter and dark energy actually are.

I should disclose, here, that I have been doing some of this work myself. I am not a professional cosmologist. I am a person with a laptop, a public dataset, and an interest in whether dark energy is constant or changing. I trained a neural network to reconstruct the equation of state of dark energy from current measurements, with the dim hope that the network would be able to see something the simpler models could not.⁹ It found a faint signal: a possible dip in the equation of state somewhere around redshift one, which would, if real, suggest that dark energy was slightly stronger several billion years ago than it is now. The signal was, by my own calibration, marginal. About one sigma in the most charitable accounting, and consistent with zero in the less charitable one.

What the work taught me, more than anything about dark energy itself, was how much of any cosmological result is the method speaking. The location of the dip in my reconstruction (around redshift one) turned out to be partly a consequence of the smoothness regularizer I had built into the network; a single anomalous measurement at redshift 0.706 was being smeared, by the math, into a broader feature located somewhere else. When I removed the regularizer and asked the data alone, the dip moved. When I dropped the anomalous measurement, the dip got smaller. The method was not lying, exactly. The method was speaking. It was telling me something about the data, but also something about itself, and it was up to me to figure out which was which.

Worse, in a way that took me longer to understand: when I asked what kind of "discovery" my method would invent from data with no signal in it at all, I found it would invent a three-sigma one about ten percent of the time, just from the noise of the optimizer landing in different local minima.

When I tried the analysis with three different supernova compilations, all three agreed that whatever was happening was in the same direction; they disagreed about how strongly, and the disagreement was about the size of the noise. Direction robust, depth at the floor.

This experience generalizes. Almost every cosmological result of the last thirty years carries, somewhere in its fine print, a section about how much of the answer comes from the data and how much comes from the choices the analyst made about how to look at the data. The honest version of every paper is partly a confession, an inventory of the assumptions that, if changed, would have produced a different number. The number we end up with is the answer the universe gave to a particular question framed in a particular way using a particular set of tools. The universe is not obligated to give us a stable answer. The universe is mostly indifferent to our framing.

This is not a counsel of despair. It is a counsel of humility. The cousins are real (we have very strong reasons to believe so), but our access to them is mediated, in every detail, by the chain of inferences and the assumptions and the choices that go into looking. Every number you read about dark energy is a number with a lineage, and the lineage matters about as much as the number does.

The Cousins, Still Unknown

Walking back, finally, to the room you are reading this in. Five percent of the cubic meter you occupy is matter you can name. The rest is, in some sense, the cousins, distributed across all the empty space of all the universe, present everywhere, including under your chair, including inside your body, including in the space between your eyes and this screen. There is no telescope you can point at this. There is no experiment that can pull a sample out of the room and put it on a table for inspection. Most of what is, is here, and most of what is, is hidden, and we have written equations for it that are wrong by factors so large that we cannot even bring ourselves to discuss them in polite company.

We have given the unknown things names (dark matter, dark energy, the cousins, whatever) and the names have done the work of making the unknown feel known. Naming is a powerful thing. Naming is also, sometimes, a way of pretending we have an answer when what we have is a placeholder.

The future will tell us more. The next decade of data, from telescopes that are being built or have just begun observing, will sharpen the picture; the picture may also confound us further, the way the supernova data confounded the people who collected it in 1998. Either is possible. Both have happened before. The cousins will, eventually, become less mysterious or more mysterious, and either is interesting.

In the meantime, while you have been reading this, dark energy has been making the space between your eyes and the screen very slightly larger. The effect is too small to perceive at any scale you have access to; on cosmic scales it is becoming the only thing that matters. The cousins have been doing this the whole time. They were doing it before Henrietta Leavitt sat down at the Harvard Observatory in 1908; they were doing it before Friedmann wrote down his equation in 1922; they were doing it before there was a Harvard. They will keep doing it after we have all stopped reading and stopped writing and stopped, eventually, paying attention. Whatever they turn out to be, they were here first, and the room you are sitting in is mostly them, and now you know it.

Footnotes

1. There is a lovely sub-genre of cosmology paper in which the authors propose a model called something like "ghost dark energy" or "phantom-tachyon hybrid quintessence" and treat the name with complete sincerity throughout, equations and all. The papers are not joking. Reading them feels like being handed a perfectly serious blueprint titled Werewolf Containment Strategy. ↩

2. Or liked. It is likely the main reason he's largely unknown to the public despite his accomplishments. He was a dick. ↩

3. The exclusion was sometimes formal policy and sometimes the kind of unwritten institutional fact that operates the same way (the distinction does not matter much to the person being excluded). The 200-inch Hale Telescope at Palomar had been operating since 1948; Rubin was, by most accounts, the first woman granted observing time there, in 1965. The official reason for the exclusion, when one was offered, was that the dome did not have a women's restroom. Rubin is reported to have addressed this on her first visit by cutting a skirt out of paper and taping it onto the figure on the men's-room door. ↩

4. Friedmann, born in St. Petersburg in 1888, derived in 1922 from Einstein's general relativity that a homogeneous universe could not, mathematically, sit still. He did this seven years before Hubble measured the expansion and five years before Lemaître proposed the primeval atom. Einstein read the paper, decided Friedmann had made an error, published a note saying so, was politely corrected by Friedmann (who wrote back showing the math again), and eventually published a retraction agreeing that Friedmann had been right all along. Friedmann died of typhoid fever in 1925, at thirty-seven, having never lived to see his equations confirmed by an actual telescope. The neural network in my paper is doing nothing more than learning what to put on the right-hand side of Friedmann's century-old equation, given what current telescopes see, and then integrating that equation forward to produce the distances those telescopes measure. He would, I suspect, find this charming. ↩

5. The CMB is the thermal radiation released when the universe was about 380,000 years old, the moment when it had cooled enough for protons and electrons to combine into neutral hydrogen and the cosmos became transparent to light for the first time. That light has been traveling, and stretching with the expansion of space, for the 13.8 billion years since; it now reaches us as microwave radiation at a temperature of 2.725 Kelvin, about three degrees above absolute zero. It comes from every direction in the sky, almost perfectly uniform, with tiny temperature variations of roughly one part in a hundred thousand that turn out to be the seeds of every galaxy and cluster that came later. The Planck satellite mapped these variations in extraordinary detail between 2009 and 2013. Almost everything we believe we know about the age, geometry, and composition of the universe (including the proportions of dark matter and dark energy in the present-day cosmos) is, in some downstream sense, derived from analyzing those variations. ↩

6. Phlogiston was the substance eighteenth-century chemists believed was released during combustion; things burned, on this theory, by giving off phlogiston into the air, which is the opposite of what is actually happening (they are taking on oxygen), and the theory survived for nearly a century partly because it explained the observations almost as well as the correct theory did, and partly because its leading proponent, Georg Stahl, was a forceful personality who did not enjoy being contradicted. Caloric was the imagined fluid that flowed from hot bodies to cold ones, carrying heat the way water carries water; this lasted into the nineteenth century and was killed, more or less single-handedly, by a Bavarian-born American spy and military engineer named Benjamin Thompson, who noticed while boring cannons in Munich that the friction produced apparently unlimited heat, which a finite caloric fluid could not. The ether you have already met. The pattern, if there is one, is that each of these substances was proposed to fill an explanatory gap, was given precise mechanical properties, generated entire research programs, and then turned out not to exist. None of which, the cosmologists would like me to remind you, is necessarily what is happening with the cousins. It is just what has happened, every previous time, with everything else. ↩

7. The number ten to the one hundred twentieth has the property of being too large to compare to anything ordinary. It is larger than the number of atoms in the observable universe (around ten to the eightieth). It is larger than the number of Planck volumes in a human body multiplied by the age of the universe in Planck times. There is no concrete object you can point at to convey it. You have to take it on faith that the disagreement is, in a literal mathematical sense, the worst disagreement there has ever been about anything in the history of trying to be right. ↩

8. Boss. ↩

9. For the curious, the technical write-up is here and the code is on GitHub. The short version: a physics-informed neural ordinary differential equation that makes the Friedmann equation the forward pass of the computational graph, applied to current DESI baryon acoustic oscillation data, three supernova compilations, the Planck cosmic microwave background distance priors, and thirty-six cosmic chronometer measurements. The paper includes a table watching the reported significance fall from 3.8σ to 1.3σ as the analysis pipeline is progressively corrected; each row is a real mistake that was actually made before being caught. I tried, in writing it, to be transparent about every assumption that could be questioned. The number of such assumptions turned out to be larger than I had expected when I started. ↩