From Prime Numbers to Nuclear Physics and Beyond
In early April 1972, Hugh Montgomery, who had been a Member in the School of Mathematics the previous year, stopped by the Institute to share a new result with Atle Selberg, a Professor in the School. The discussion between Montgomery and Selberg involved Montgomery’s work on the zeros of the Riemann zeta function, which is connected to the pattern of the prime numbers in number theory. Generations of mathematicians at the Institute and elsewhere have tried to prove the Riemann Hypothesis, which conjectures that the non-trivial zeros (those that are not easy to find) of the Riemann zeta function lie on the critical line with real part equal to 1⁄2.
Montgomery had found that the statistical distribution of the zeros on the critical line of the Riemann zeta function has a certain property, now called Montgomery’s pair correlation conjecture. He explained that the zeros tend to repel between neighboring levels. At teatime, Montgomery mentioned his result to Freeman Dyson, Professor in the School of Natural Sciences.
In the 1960s, Dyson had worked on random matrix theory, which was proposed by physicist Eugene Wigner in 1951 to describe nuclear physics. The quantum mechanics of a heavy nucleus is complex and poorly understood. Wigner made a bold conjecture that the statistics of the energy levels could be captured by random matrices. Because of Dyson’s work on random matrices, the distribution or the statistical behavior of the eigenvalues of these matrices has been understood since the 1960s.
Dyson immediately saw that the statistical distribution found by Montgomery appeared to be the same as the pair correlation distribution for the eigenvalues of a random Hermitian matrix that he had discovered a decade earlier. “His result was the same as mine. They were coming from completely different directions and you get the same answer,” says Dyson. “It shows that there is a lot there that we don’t understand, and when we do understand it, it will probably be obvious. But at the moment, it is just a miracle.”
The unexpected discovery by Montgomery and Dyson at teatime in the 1970s opened a tantalizing connection between prime numbers and mathematical physics that remains strange and mysterious today. Prime numbers are the building blocks of all numbers and have been studied for more than two thousand years, beginning with the ancient Greeks, who proved that there are infinitely many primes and that they are irregularly spaced.
More than forty years after the teatime conversation between Dyson and Montgomery, the answer to the question of why the same laws of distribution seem to govern the zeros of the Riemann zeta function and the eigenvalues of random matrices remains elusive, but the hunt for an explanation has prompted active research at the intersection of number theory, mathematical physics, probability, and statistics. The search is producing a much better understanding of zeta functions, prime numbers, and random matrices from a variety of angles, including analyzing various systems to see if they reflect Wigner’s prediction that the energy levels of large complex quantum systems exhibit a universal statistical behavior, a delicate balance between chaos and order defined by a precise formula.
Wigner’s universality conjecture is somewhat analogous with the classical central limit theorem of probability theory, which explains why many distributions in nature tend to be close to the normal distribution of the Gaussian bell curve. The British polymath Sir Francis Galton described the central limit theorem as:
“I know of scarcely anything so apt to impress the imagination as the wonderful form of cosmic order expressed by the ‘Law of Frequency of Error.’ The law would have been personified by the Greeks and deified, if they had known of it. It reigns with serenity and in complete self-effacement, amidst the wildest confusion. The huger the mob, and the greater the apparent anarchy, the more perfect is its sway. It is the supreme law of Unreason. Whenever a large sample of chaotic elements are taken in hand and marshaled in the order of their magnitude, an unsuspected and most beautiful form of regularity proves to have been latent all along.”
For centuries, probability theory has been used to model uncorrelated or weakly correlated systems. There is strong evidence that random matrix statistics plays a similar fundamental role for complicated correlated systems, among them the energy levels of the uranium nucleus, the zeros of the zeta function, and the spacing pattern of a decentralized bus system in the city of Cuernavaca, Mexico, which two physicists, Milan Krbálek and Petr Šeba, studied in the late 1990s. The bus system had no central authority or timetables to govern the arrivals and departures of the buses, which were individually owned by the drivers. In order to maximize their incomes, the drivers adjusted their speeds based on information obtained by bystanders about the departure times of the buses in front of them. Krbálek and Šeba recorded the actual departure times of the buses at various stops and found that the spacings between buses match the statistical behavior found in random matrix theory.
According to Percy Deift (frequent Member in the School), “The list of such problems is varied, long, and growing, and points to the emergence of what one might call ‘macroscopic mathematics.’” Just as physicists discern physical laws from the emergence of universal behavior in macroscopic systems, Deift observes, mathematicians are beginning to study the universality of a wide variety of mathematical problems. The widespread phenomenon, in which a whole class of seemingly unrelated problems or physical situations gives rise to the same statistical histogram, is being examined for insights that it might provide about various chaotic systems in the world around us.
“There are lots of deterministic things that behave chaotically,” says Thomas Spencer, Professor in the School of Mathematics. “In general, that is a very hard thing to understand. Even the zeros of the Riemann zeta function, which are given by a formula, behave chaotically. Prime numbers behave chaotically. We don’t understand why they do, and we would like to understand to what extent they do. What is amazing is that we can describe these characteristics in terms of random matrices to some extent.”
In the mid-nineteenth century, mathematicians began to focus on conceptual properties rather than formulas and on understanding abstract concepts and relationships rather than calculation. Bernhard Riemann was a champion of the subject that was invented around that time called complex analysis. He explained that if you want to understand a complex analytic function—which is a rule relating complex numbers—you need to understand the locations of its zeros.
The zeta function was discovered by Leonhard Euler in the eighteenth century. Euler defined the function and showed that it has a deep and profound connection with the pattern of the primes. Riemann, after whom the zeta function is named (you don’t have to be the first to study an object in mathematics in order to get it named after you), wrote about the zeta function and his related hypothesis in his memoir published in 1859—his only paper on the zeta function. Riemann’s zeta function and other zeta functions similar to it known as L-functions (which came out of Lejeune Dirichlet’s study of prime numbers in 1836) appear in number theory, the theory of dynamical systems, geometry, function theory, and physics. In his memoir, Riemann was trying to explain how to obtain an exact simple analytical formula for counting the number of primes up to a given limit.
“The failure of the Riemann Hypothesis would create havoc in the distribution of prime numbers,” writes Enrico Bombieri, Professor Emeritus in the School, in the official problem description on the website of the Clay Mathematics Institute where it is listed as one of the Millennium Problems worth $1 million to any individual who can solve it. “This fact alone singles out the Riemann Hypothesis as the main open question of prime number theory.”
The late Atle Selberg was an authority on the Riemann Hypothesis and developed many tools that allow mathematicians to go around it. “There have probably been very few attempts at proving the Riemann Hypothesis,” said Selberg, “because, simply, no one has had any really good idea for how to go about it.” Bombieri thinks Selberg may have meant to include “good” in his statement (as in “very few good attempts”) in light of the numerous efforts that Bombieri receives from amateur mathematicians and physicists. “From time to time,” says Bombieri. “I also receive crank mail from amateur mathematicians who think that they have disproved the Riemann Hypothesis.”
In his memoir, Riemann gave a hint that he had computed the first few zeros. “Certainly, it is desirable to have a rigorous proof of this,” wrote Riemann. “I have put aside this research for the time being after some fleeting superficial attempts, because it is not immediately needed for the sequel of my investigation.” When his notes were reviewed by the late Carl Ludwig Siegel (Member and Professor at the Institute in the 1940s) about seventy years after Riemann’s death, it was confirmed that before making his conjecture, Riemann had checked by hand that the first few zeros all lie on the critical line. With one of the earliest computers, Alan Turing computed the first one thousand zeros of the Riemann zeta function. Today, the hypothesis has been confirmed into the trillions of zeros.
“The Riemann zeta function remains one of the mysteries of modern mathematics. It is a function that we understand a lot about except for the most important question,” says Peter Sarnak, Professor in the School of Mathematics. “It connects the theory of prime numbers, or encodes deep information about the theory of prime numbers, with the zeros. It controls the prime numbers in a way that nothing else we know does. While understanding prime numbers is an important problem, it is the generalizations of the Riemann zeta function and the objects associated with these that make it more significant.”
Generalizations are extensions of a mathematical problem in an analogous, less-specific setting. Trying to prove the Riemann Hypothesis has produced answers to complicated questions with deep and extensive reach into many diverse areas of mathematics and physics. The Riemann Hypothesis and its generalizations have “analogues that are true all over the show,” says Sarnak. “There is no question about its universal truth for all zeta functions that appear in the world. Its importance is amplified in that there are hundreds of theorems that say if the Riemann Hypothesis is true, or some generalization of it, then the following is true, and the following can be stunning. And most of these consequences are still not known, although many have been proven without proving the Riemann Hypothesis. So it allows you to get where you want to go quite quickly, and since we haven’t been able to solve this problem, we find very complicated substitutes. In fact, if anybody proved this Riemann Hypothesis and these generalizations, you could throw away a number of books and many papers in libraries, which are there primarily to sidestep not knowing it.”
For many years there was no proof of the prime number theorem without using Riemann’s zeta function. One of the great achievements in 1948 took place at the Institute when Selberg and Paul Erdös (Member, 1938–40) gave an elementary proof of the prime number theorem without using the Riemann zeta function. “Some people naively thought there was no elementary proof,” says Sarnak. “And there was a further hope that once you gave an elementary proof of the theorem it might give a better viewpoint on the Riemann Hypothesis, but that hasn’t materialized yet.”
When asked what would be the first thing he would do if he were brought to life again after five hundred years, David Hilbert, who with George Pólya proposed looking for a quantum mechanical system with eigenvalues given by the Riemann zeta-function zeros, replied, “I would ask whether the Riemann Hypothesis has been proved.” André Weil, late Professor in the School of Mathematics, put a tremendous effort into trying to solve the problem later in his life. “In the past it sometimes occurred to me that if I could prove the Riemann Hypothesis, which was formulated in 1859, I would keep it secret in order to be able to reveal it only on the occasion of its centenary in 1959,” said Weil in a 1979 interview. “Since 1959, I have felt that I am quite far from it; I have gradually given up, not without regret.”
In 1941, Weil proved the analogue of the Riemann Hypothesis for all one-variable function fields over finite fields, the focus of his work while in a prison in Rouen for, in his words, “having a disagreement with the French authorities on my military ‘obligations.’” Weil introduced new geometric points of view that allow the translation of the problem to one in algebraic geometry. In 1973, Pierre Deligne used Alexander Grothendieck’s cohomology theory, a tool that linearized the problem, to prove the Riemann Hypothesis for the zeta functions of complete nonsingular projective varieties over finite fields of any dimension. (For this work and other seminal contributions, Deligne, Professor Emeritus in the School of Mathematics, has been awarded the 2013 Abel Prize of the Norwegian Academy of Science and Letters).
The function field analogue, where Weil and Deligne achieved their results, realizes the zeros as the eigenvalues of a matrix. “What is lacking in the case of the Riemann zeta function is an eigenvalue interpretation of the zeros that is useful,” says Sarnak. “From the data, you have this remnant of a real object that you want to resurrect. Before you can start to do that, the first thing you need to know is, is it really coming from such and such an object? Are there some good tests or signatures for that?”
Quantum mechanics is a linear algebraic interpretation of mechanics in which the energy levels of quantum systems correspond to eigenvalues of matrices. It might provide a mathematical tool for using linear algebra to prove the Riemann Hypothesis if, as it appears, the zeros of the Riemann zeta function are behaving like eigenvalues of a matrix.
More than sixty years ago, Wigner was led to ask: If you take a random matrix, what do its eigenvalues look like? Would they look different if you chose the numbers randomly? It turns out that if you choose a matrix at random and look at its eigenvalues, you get a very different behavior than if you choose the numbers at random directly. What the eigenvalues look like became an interesting subject. This was the beginnings of what became random matrix theory—a subject in the theory of probability and linear algebra.
Dyson worked vigorously on random matrix theory for about ten years, roughly from 1962 to 1972, mostly in collaboration with Madan Mehta (Member, 1962–63). In the theory of random matrices, Dyson and Mehta identified three types of matrix ensembles with different correlations: Gaussian orthogonal ensemble (time reversal invariant and integer spin with weakest repulsion between neighboring levels), Gaussian unitary ensemble (no time reversal invariance with medium repulsion), and Gaussian symplectic ensemble (time reversal invariant with half integer spin and strongest repulsion). The ensembles are abbreviated as GOE, GUE, and GSE, respectively. In 1989, Andrew Odlyzko (Member, 1983– 84) computed eight million zeros of the Riemann zeta function near to zero number 1020 and computed their pair-correlation numerically in a deep and very thorough investigation that confirmed the GUE connection.
When Sarnak was a graduate student at Stanford University, Paul Cohen (Member, 1959–61, 67), pointed him to Montgomery’s work on the pair-correlation of the zeros of zeta and its connection to random matrix theory, and asked, why is it so? Sarnak’s efforts to try to answer that question began with a paper with Zeev Rudnick (Member, 2008–10) on the higher correlations for zeros of the zeta function and led eventually to his extension of Montgomery’s work. In the 1990s, Sarnak and his collaborator Nick Katz (frequent Member since 1991) brought techniques from physics and geometry into the function field setting of Deligne’s proof and found that not only the pair-correlation function but all the many-level correlation functions of the zeta-function zeros agree with those of the GUE ensemble. They used techniques of mathematical physics, particularly those of Michel Gaudin, who calculated the distribution of the spacings between nearest-neighbor eigenvalues in 1961, to explain the phenomenon and the symmetry types.
The way Deligne proved the Riemann Hypothesis in general in the function field setting was not by looking at one zeta function at a time. Zeta functions can be collected into families, and with each family, you can ask subtle questions about the distribution of their zeros.
Different families produce different distributions, in each of which the spacing correlation is universal, corresponding to random matrix theory. Central to this study is a group—the monodromy group—associated with the family, which acts linearly on related spaces. Monodromy is the glue—a symmetry connected with the family, which marries the different members. Sarnak’s former students, among them Michael Rubinstein (Member, 2009–10), verified numerically all the conjectures in the classical number theoretic setting that Katz and Sarnak put forward connected with this glue.
“Not only does there appear to be a matrix interpretation, but this glue that is so critical in the proof in the function field setting, is apparently present here too. We know that, we can see it, but we don’t have the object yet,” says Sarnak. “It has led to predictions and theorems because this glue controls much of what happens within a family. We know a bit about it without knowing its real source. But just knowing what we can prove about the glue is already enough to deduce some consequences of the Riemann Hypothesis in general, and hence to solve a number of problems.”
Progress also has been made in using random matrix theory to further explore connections between prime numbers and quantum physics. The even “moments” of the zeta function on the critical line are notoriously difficult to compute and in particular the leading coefficient for their asymptotic behavior. The only known cases, dating from the 1920s, are the second and fourth moments for which the coefficients are 1 and 2. Brian Conrey and Amit Ghosh (frequent Members since the early 1980s) conjectured that the coefficient for the sixth moment is 42. Using random matrix theory, physicists Jon Keating and Nina Snaith of Bristol University confirmed this result and provided a formula to predict all the numbers in the sequence.
Universality of various matrix ensembles will be one of the major themes of the School’s special year program on non-equilibrium dynamics and random matrices that Spencer will be running next year with Horng-Tzer Yau of Harvard University (Member, 1987–88, 2003) who will be the School’s Distinguished Visiting Professor. Yau is the author, along with Laszlo Erdos̈ and Benjamin Schlein, and, independently, Terence Tao (Visitor, 2005) and Van Vu (frequent Member since 1997), of a recent proof that states that the eigenvalue statistics of a matrix do not depend on the distribution of its elements as long as they are independent and have the same distribution.
By investigating the universality phenomenon in random matrices, mathematicians are trying to develop a better sense of what universality is, why it arises, and how it can be used. The principle of universality suggests this is an enormous class, according to Spencer, who works in supersymmetric statistical mechanics and quantum mechanics. Some statistical properties of the random matrices are known and understood and others are conjectural.
Spencer’s interest in random matrix theory stems from his study of the quantum mechanics of an electron moving in a random environment, such as a crystalline lattice with random impurities. Following work done by physicist Philip Anderson on the electronic structure of magnetic and disordered systems, Spencer has been studying Anderson-like matrices that look very different from those of Wigner. In these matrices, the randomness is limited and confined. To understand the quantum excitations of the electrons, Spencer studies the statistical mechanics of supersymmetric spins on a lattice. Physicists Franz Wegner and Konstantin Efetov explained the relation between the energy levels of the electron and supersymmetric statistical mechanics in the early 1980s. In the language of statistical mechanics, the energy level spacing should match those of GUE or GOE at low temperatures in three dimensions when the spins are aligned or ordered.
“If randomness is present, we have a pretty good understanding of universality from the point of view of physics using supersymmetric statistical mechanics,” says Spencer. “But for a purely deterministic system, we have a much poorer understanding of how Wigner-Dyson statistics will emerge.”
Perhaps surprisingly, one of the forebears commonly credited with understanding universality, isn’t rooting for it. “My main contribution was to find these three different classes. It was the opposite of universality, proving that the three types are really very different. My prejudice is the other way,” says Dyson.
“There is this sort of religious belief in universality which I don’t agree with, but certainly there is some evidence for it. The idea is if you prove something for a single example, then it is likely to hold for all kinds of things with the same structure. The behavior is universal. It is the same behavior for a big class of objects. You could say that I have a baby who is very bad tempered, so I assume that that is a universality class, that all babies have bad tempers, which of course is not a valid conclusion. [Laughs] So I would be skeptical about universality. Sometimes it works and sometimes it doesn’t. It is a matter of picking your examples. You can always find examples that behave the same, and you find examples that behave differently. But I don’t like to call it universal unless it really is true everywhere.”
If universality doesn’t explain the mysterious connection, why does the pair correlation of the zeros of the Riemann zeta match that of the eigenvalues of the GUE? “That’s a good question,” says Dyson, “but we don’t know.”
Dyson believes quasicrystals, which he studied with Paul Steinhardt when Steinhardt was a Member in the School of Natural Sciences more than twenty years ago, could provide some clues for solving the Riemann Hypothesis, which would illuminate pathways in many directions including its connection to random matrix theory. “People thought there were only two kinds of matter—the kind that is ordered in perfect crystals and the kind that is disordered and just a jumble of atoms,” says Dyson. “The quasicrystal comes in between. It has long-range order, but it doesn’t have a regular spacing. That was a big surprise. I spent a lot of time trying to understand it. There is still a lot more to be understood.”
Dyson started out as a mathematician, as a student of Harold Davenport’s, a mathematician known for his extensive work in number theory. Dyson also took classes from G. H. Hardy who proved that there are infinitely many zeros of the Riemann zeta function on the critical line. When Dyson addressed an audience at the Mathematical Sciences Research Institute in 2002, he challenged young mathematicians to study quasicrystals. “Like every serious student of pure mathematics, when I was young, I had dreams of proving the Riemann Hypothesis. I had some vague ideas that I thought might lead to a proof, but never pursued them vigorously,” Dyson told the audience. “In recent years, after the discovery of quasicrystals, my ideas became a little less vague. I offer them here for the consideration of any young mathematician who has ambitions to win a Fields Medal.”
Quasicrystals were discovered in 1984 and exist in spaces of one, two, or three dimensions. Dyson suggests mathematicians obtain a complete enumeration and classification of all one-dimensional quasicrystals, the most prevalent type, with the aim of identifying one with a spectrum that corresponds to the Riemann zeta function and one that corresponds to the L-functions that resemble the Riemann zeta function. If it can be proved that a one-dimensional quasicrystal has properties that identify it with the zeros of the Riemann zeta function, then the Riemann Hypothesis will have been proved. “In one dimension, there is no symmetry, and you have an enormous variety of quasicrystals which we have not ever classified,” says Dyson. “There is a huge universe there we haven’t explored. It could be a very deep part of mathematics once you get into it. It’s a wild speculation that it could lead to the Riemann Hypothesis. But I think it’s not completely crazy; it is certainly a possibility.”
