One of my favourite problems in mathematics is the Kakeya family of conjectures. There are many versions of these conjectures, but one of the simplest to state is the following:

Let be a compact subset of which contains a unit line segment in every direction. Then has Hausdorff and Minkowski dimension .

Sets which contain a unit line segment in every direction are known as Kakeya sets. It was observed by Besicovitch that for , Kakeya sets can have arbitrarily small Lebesgue measure; in fact they can have Lebesgue measure zero. This in turn implies that the solution to the Kakeya needle problem (what is the least amount of area in the plane needed to rotate a unit line segment by ?) is that a unit needle can be rotated in arbitrarily small area (see this previous blog post of mine for further discussion).

The conjecture is trivial in one dimension, and also proven in two dimensions (a result of Davies), but remains open in three and higher dimensions. Nevertheless, there are a number of partial results, typically of the form “Kakeya sets in have Hausdorff or Minkowski dimension at least ” for various values of and (with the objective being to get all the way up to “. One can also phrase such results in a largely equivalent discrete fashion, as follows. Let be a small number, and let be a collection of tubes which are oriented in a -separated set of directions (thus if , are oriented in the direction , respectively for some , then ). This -separation hypothesis forces to be (we allow implied constants to depend on ); let us now assume that is close to its maximal value, thus . The Kakeya set conjecture (for the Minkowski dimension) is then equivalent to the assertion that

for all , where denotes the volume of the set , and denotes the estimate for some depending only on . Similarly, a partial result of the form

for all and some would imply (and is basically equivalent to) the assertion that Kakeya sets have (lower) Minkowski dimension at least .

There is also a somewhat stronger Kakeya maximal function conjecture which is also of interest; with the same hypotheses as above, the conjecture asserts that

for all and . This conjecture is trivial for , but the difficulty increases as approaches ; for any fixed , the estimate (2) easily implies (1), and hence that Kakeya sets in have Minkowski dimension at least ; it also can be used to imply that such sets have Hausdorff dimension at least as well. (The terminology “Kakeya maximal function conjecture” comes from the fact that (2) is dual to a certain type bound on the Kakeya maximal function

a variant of the Hardy-Littlewood maximal function which averages over tubes oriented in a direction , rather than on balls, but we will not discuss the maximal function further here.)

The Kakeya conjecture is known to have applications to other fields of mathematics, for instance to Fourier analysis (as observed by Fefferman), to wave equations (as observed by Wolff), to analytic number theory (as observed by Bourgain), to cryptography (as observed also by Bourgain) and to random number generation in computer science (as observed by Dvir and Wigderson). However, the focus of this talk is not on the fields of mathematics impacted by the Kakeya conjecture, but rather on the fields of mathematics that are used to make progress on this conjecture, as there is a striking diversity of mathematical techniques which have been usefully applied to produce a number of non-trivial partial results on the problem. In particular, I wish to discuss how the following areas of mathematics have been used to attack the Kakeya problem:

• Incidence geometry;
• Additive combinatorics;
• Multiscale analysis;
• Heat flows;
• Algebraic geometry;
• Algebraic topology.

— 1. Incidence geometry —

The earliest positive results on the Kakeya problem, when interpreted from a modern perspective, were based primarily on exploiting elementary quantitative results in incidence geometry – the study of how points, lines, planes, and the like (or more precisely, -thickened versions of such concepts, such as balls, tubes, and slabs) intersect each other. The incidence geometry approach was greatly clarified by the introduction by Wolff in 1995 of the finite field model of the Kakeya conjectures, in which the underlying field is replaced by a field of some large but finite order (which of course must be a prime, or a power of a prime). The analogue of the Kakeya set bound (1) is then an estimate of the form

for all , where is a family of lines in that contains one line in every direction (thus, will have cardinality comparable to ), and denotes the cardinality of a set . In a similar vein, the analogue of (2) is the assertion that

under the same hypothesis on . The fact that (4) implies (3) can be easily seen from the identity

and Hölder’s inequality, using the fact that the multiplicity function is supported in .

There is a close analogy between the finite field Kakeya problems and the Euclidean ones; arguments that make progress in one setting can often be adapted to make progress in the other. However, there is no formal correspondence, and there are some arguments that seem to be specific to the finite field setting, and other arguments that are specific to the Euclidean one. Nevertheless, the finite field setting has proven to be an extremely useful toy model to gain intuition and insight into the (more complicated) Euclidean problem.

Very roughly speaking, every incidence geometry fact in classical Euclidean geometry can be converted (at least in principle) via elementary combinatorial arguments into a lower bound on Kakeya sets. Consider for instance the Euclidean axiom that any two lines intersect in at most one point. This can be converted to the bound for any dimension , for any one of the versions of the Kakeya problem mentioned above; this was established by Davies for the Euclidean dimension problem and by Córdoba for the Euclidean maximal function problem. We sketch a heuristic proof for the finite field set problem (3). Suppose for simplicity that has size exactly . There are about lines in , each containing points; if we adopt the heuristic that the points described this way are spread out uniformly in the set , then each point in that set should be incident to about lines. Now, let us count the number of configurations consisting of two distinct lines in intersecting at a single point. On the one hand, since has about lines, and any two lines intersect in at most one point, the number of such configurations is at most ; on the other hand, since there are about points, and each point is incident to about lines, then the number of configurations is at least . Comparing the lower and upper bounds gives the desired bound . (This argument assumed uniform distribution of the multiplicity function, but the general case can be handled similarly by applying the Cauchy-Schwarz inequality.)

There are several other instances of this incidence geometry strategy in action:

• The axiom that two distinct points determine a line can be used to give the bound in any dimension . This was done (implicitly) by Drury for the Euclidean set and maximal problems, and explicitly by by Christ, Duandikoextea, and Rubio de Francia with a refined endpoint estimate at .
• The fact that any three lines in general position determine a regulus (ruled surface), which contains a one-dimensional family of lines incident to all three of the original lines, can be used to give the bound in the three-dimensional case ; this was achieved by Schlag (and the bound was also obtained by a slightly different argument earlier by Bourgain).
• The axiom that any two intersecting lines determine a plane, which contains a one-dimensional family of possible directions of lines, was used by Wolff to give the bound in all dimensions . This argument relied cruically on the fact that the lines pointed in different directions; note that if one allows lines to be parallel, then a plane contains a two-dimensional family of lines rather than a one-dimensional one.
• In four dimensions (and for the finite field set problem), I managed to improve the Wolff bound of in four dimensions slightly to , by a more complicated axiom from incidence geometry, namely that any three reguli in general position determine an algebraic hypersurface, the lines in which only point in a two-dimensional family of directions.

From this sequence, it seems that one needs to imply increasingly “high-degree” facts from incidence geometry to make deeper progress on the Kakeya conjecture, in particular one begins to transition from incidence geometry to algebraic geometry. Indeed, one can view the polynomial method of Dvir (introduced later in this post) as the natural continuation of these methods. However, there is some evidence that the incidence geometry approach, by itself, is not sufficient to establish the full conjecture. For instance, if one heuristically inserts an arbitrary incidence geometry fact involving bounded-degree algebraic varieties into the above method, it appears that one cannot obtain a lower bound on of better than . Also, there are near-counterexamples to the Kakeya conjecture (such as the Heisenberg group when the field admits a non-trivial conjugation ) which only fail to contradict that conjecture due to the parallel nature of some of the lines, so any argument establishing the conjecture must make essential use of the fact that lines point in different directions.

— 2. Additive combinatorics —

In 1998, Bourgain introduced a somewhat different approach to the Kakeya problem, which relied on elementary facts of arithmetic (and in particular, addition and subtraction), rather than that of geometry, to obtain new bounds on the Kakeya problem. This additive-combinatorics method fared better than the geometric method in higher dimensions, basically because the additive structure of high-dimensional spaces was much the same as for low-dimensional spaces, even if the geometric structure becomes much different (and less intuitive).

It is convenient to illustrate the method using the finite field model problem (3), using the method of “slices” introduced by Bourgain (though it is also possible to perform the additive combinatorial method without slicing the set), though the method can also be adapted to the other variants of the Kakeya problem. As before, we argue heuristically in order to simplify the discussion. Let be a finite field Kakeya set, and suppose it has cardinality about . We then write as and consider the three “slices” of , defined as the intersection of with the horizontal hyperplanes , , and respectively (let us assume is odd for the sake of discussion, so that is well-defined). Then we expect to all have cardinality about . On the other hand, contains about lines, each of which connect a point to a point , with the midpoint lying in . Since two points uniquely determine a line, we thus have about many pairs in whose sums are contained in a small set, namely . (This fact alone already leads to the lower bound mentioned earlier.) On the other hand, since the lines all point in different directions, the differences of all these pairs are distinct. The additive combinatorial strategy is then to play off the compressed nature of the sums on one hand, and the dispersed nature of the differences on the other. One of the main ingredients here are elementary additive identities, such as

which suggests that collisions of sums should imply collisions of differences. This identity, by itself, is insufficient to obtain any new bound on the Kakeya problem, because even if the pairs and come from lines in , there is no reason why the pairs or should also. But one can then combine the above identity with further identities, such as

which can be used to convert collisions of some differences to collisions of other differences. Implementing this strategy rigorously using a combinatorial tool now known as the Balog-Szemerédi-Gowers lemma, Bourgain improved the bound to (for the Minkowski dimension in both Euclidean and finite field settings), which was superior to the bounds obtained by incidence geometry methods in high dimensions. By inserting more and more such additive identities into this framework (and taking more and more slices), the constants improved somewhat; for instance, in high dimensions, the best result so far (on the Minkowski Euclidean problem) is , where is the largest root of , a result of Nets Katz and myself; see this survey article of Nets and myself for further discussion.

It may well be that the arithmetic approach could eventually settle the entire Kakeya conjecture (this would correpsond to lowering the exponent appearing in the above results all the way to ); there is a formalisation of this assertion, known as the “arithmetic Kakeya conjecture”, which has some amusing relationships with group theory. However, we have not been able to find additive combinatorial arguments that are so efficient that they do not lose anything in the exponents, and it is not clear whether this conjecture is within reach of known technology. One possible direction to pursue is to move from additive combinatorics (the combinatorics of addition and subtraction) to arithmetic combinatorics (the combinatorics of addition, subtraction, multiplication, and division). For instance, the sum-product phenomenon in finite fields was used by Bourgain, Katz, and myself to improve the bound on three-dimensional Kakeya sets slightly from to for some . Nets Katz and I are exploring this direction further, and I hope to report on our results at some point in the future.

— 3. Multiscale analysis —

The finite field model is considered simpler than the Euclidean model for a number of reasons, but one of the main ones is that the finite field model does not have the infinite number of scales that are present in the Euclidean setting. But one can reverse this viewpoint, and instead look for ways to exploit the multitude of scales available in the Euclidean case. One promising strategy in this direction is the induction on scales strategy, introduced by Bourgain for the closely related restriction and Bochner-Riesz problems, and developed further by Wolff. The basic idea here is to deduce a Kakeya estimate at scale from the corresponding the Kakeya estimate at a coarser scale, such as . We sketch the basic idea as follows. Suppose that the Kakeya conjecture was already established at scale ; roughly speaking, this means that any connection of tubes that point in different directions (i.e. -separated directions) must be essentially disjoint. One can then argue that that this should implies the same assertion for tubes. To justify this, we assume that these “thin” tubes are arranged in a sufficiently “self-similar” (or “sticky”) fashion that they can be organised into “fat” tubes, which themselves point in different (i.e. -separated) directions. By the Kakeya hypotheses, these fat tubes are essentially disjoint. What about the thin tubes inside any given fat tube? Well, if one rescales a fat tube about its axis, dilating its lateral dimensions by so that it becomes a cylinder, then the thin tubes inside that fat tube essentially expand into fat tubes, which by hypothesis are again essentially disjoint. Since the thin tubes inside each fat tube are essentially disjoint, and the fat tubes are themselves essentially disjoint, the entire collection of thin tubes should be essentially disjoint as well.

This argument is remarkably difficult to make rigorous, in part because it is not obvious at all why the tubes should stick together in a self-similar way, but also because the fat tubes can intersect each other in a coplanar fashion, allowing the thin tubes in each fat tube to align up with an unusually high multiplicity. In a 63-page paper, Nets Katz, Izabella Laba, and I managed to implement this idea, in conjunction with much of the incidence geometry and additive combinatorics arguments discussed in previous sections, but only managed to improve the (Euclidean Minkowski dimension) bound for three-dimensional Kakeya sets from to (with similar small improvements in higher dimensions). Nevertheless, the multiscale method did lead to another method which gave further results, namely the heat flow method which we now discuss.

— 4. Heat flow —

It is possible to eliminate the coplanarity and stickiness issues arising in the multiscale approach by replacing a linear Kakeya problem such as (2) with a multilinear variant, namely

whenever for are families of -tubes with -separated directions, pointing close to the cardinal direction . Roughly speaking, the difference between (5) and (2) is that “coplanar” interactions have been abolished from (5) by fiat; the expression inside the norm consists only of products of tubes whose directions are in general position. With these difficulties eliminated, the induction on scales argument comes much closer to working properly. However, the degradation of constants is too poor; when passing from scale to scale , the losses get squared, leading to a net loss of for some constant , which is unacceptable. However, it turns out (as shown by Bennett, Carbery, and myself) that one can avoid all such losses by performing a continuous analogue of the induction on scales procedure, in which the tubes are replaced by distorted Gaussian functions, which are then continuously dilated about their major axis by heat flow, effectively increasing their thickness from towards . One then shows that (a suitable modification of) the expression in (5) increases along this heat flow, allowing one to deduce (5) at scale from (5) at scale . (One still incurs a loss because, for technical reasons, one can only obtain a monotonicity formula if one works with norms rather than norms, requiring an interpolation to finish the job.)

Another way to view the heat flow argument is to take the (gaussian-smoothed) tubes and slide them continuously toward the origin, so that at the final stage one obtains a “bush” of (smoothed) tubes through the origin, for which (5) is easy to establish. One can use essentially the same monotonicity computations as before to show that the norm (or more precisely, a slightly weighted norm) in (5) increases along this sliding procedure, thus providing a sort of variational proof of (5).

The multilinear Kakeya conjecture (5) implies its usual counterpart (2) in two dimensions by a standard angular rescaling argument, but unfortunately it seems to be strictly weaker than the linear Kakeya conjecture in three and higher dimensions, because it says nothing about the coplanar intersections which seem to be a major feature in the higher-dimensional Kakeya problem. (Indeed, (5) suggests, roughly speaking, that if there is a counterexample to the Kakeya conjecture, it will come from “plany” Kakeya sets, in which the lines that pass through a typical point will essentially all lie on a hyperplane.) Thus far we have been unable to find a monotonicity formula that can handle the coplanar case (and perhaps one should not expect to find one, given that the extremisers for the linear maximal function are likely to resemble Besicovitch sets and thus not be amenable to a simple variational argument). Nevertheless (5) is one of the few Kakeya-type estimates we have in higher dimensions whose exponents are sharp (up to epsilons).

— 5. Algebraic geometry —

A striking breakthrough on the finite field side of the Kakeya problem was achieved recently by Dvir, by introducing a high-degree analogue of the low-degree algebraic geometry used in the incidence geometry approaches (see also this blog post for more discussion). Indeed, one can view Dvir’s argument in the incidence geometry framework, with the two key incidence geometry inputs involving high-degree hypersurfaces. The first is the following (a high-degree analogue of Wolff’s observation that a plane contains only a one-dimensional family of lines; see also my paper with Mockenhaupt, or another paper of mine, for similar lemmas):

Lemma 1 If is a degree hypersurface in with , then the number of possible directions of lines in is at most .

Indeed, if one extends the degree hypersurface in to a degree hypersurface at the hyperplane at infinity, then every line in will extend to a point in , representing the direction of . (Here we use the fact that a polynomial of degree cannot vanish at every point of a line unless it is identically zero (in the algebraic sense) on that line (and in particular, on the extension of that line to the plane at infinity.) So the number of possible directions is bounded by the cardinality of , which is by the Schwartz-Zippel lemma.

The second ingredient can be viewed as a high-degree analogue of such facts as “two points determine a line” or “three points determine a plane”, and is as follows:

Lemma 2 If is a set of points in , then is contained inside a hypersurface of degree .

Indeed, to prove this lemma one needs to find a non-trivial polynomial of degree at most for some which vanishes at every point in . But the vector space of polynomials of degree at most has dimension about , and the requirement that vanish at every point in imposes linear constraints, so by linear algebra one can find such a polynomial as soon as the is much larger than , and the claim follows.

Putting the two lemmas together, we see that any Kakeya set (which, by definition, contains lines in directions) cannot be contained in any hypersurface of degree much less than , and thus must have cardinality . (In fact, the optimal size of a Kakeya set is now known to be within a factor of of , a result of Dvir, Kopparty, Saraf, and Sudan.) The argument can also be adapted to establish the maximal function estimate in finite field vector spaces, and also generalises to other families of curves in algebraic varieties, as was done recently by Ellenberg, Oberlin, and myself (see this blog post for further discussion). There are even some tentative indications that these algebraic geometry methods will adapt well to more abstract algebraic settings, such as that of schemes.

— 6. Algebraic topology —

It was thought that the high-degree algebraic geometry methods of Dvir and others were strongly dependent on the discrete nature of the finite field setting, and would not extend to continuous settings such as that of Euclidean space. Very recently, however, Larry Guth managed to partially extend Dvir’s results to this case. One key observation is to replace Lemma 2 by a topological counterpart, namely

Lemma 3 (Polynomial Ham Sandwich theorem) If are a collection of bounded open sets in , then there exists a hypersurface of degree which bisects each of these open sets (thus the sets and have equal volume).

Note that if one shrinks these bounded open sets to a tiny neighbourhood of points , then in the limit one recovers the analogue of Lemma 2 over the reals (note that the proof of Lemma 2 is valid over any field). This allows one to adapt Dvir’s method to continuous settings, basically by replacing points by balls. (There is another ingredient needed, which is an isoperimetric-type inequality that asserts that if a hypersurface bisects a ball, then the intersection of that hypersurface with the ball has a large area; see this blog post for further discussion.)

Unfortunately, the arguments, like the heat flow arguments, so far are restricted to the non-coplanar case, as otherwise certain Jacobian factors begin to enter in an unfavorable fashion in the estimates, so the Euclidean linear Kakeya problem has so far not been in impacted by these results. However, the arguments of Guth are able to recover the multilinear estimate (5), and in fact can also remove the loss, thus obtaining a very sharp estimate. Even more recently, Guth and Katz have managed to solve a discrete analogue of the Euclidean Kakeya problem in which coplanarity has been eliminated by fiat, namely the joints problem of Sharir. Define a joint in to be a configuration of concurrent lines, which do not all lie in a hyperplane. Sharir conjectured that a collection of lines in could form at most joints (this is optimal, as can be seen by looking at lines parallel to the coordinate axes passing through a discrete cube of sidelength and unit spacing); this conjecture has now been verified by Guth and Katz, in the three-dimensional case at least. (The estimate (5) was already observed to be closely related to the joints problem by Bennett, Carbery, and myself, although it only gives a strong result in the case when one has a quantitative lower bound on the non-coplanarity of the joints.)

In summary, there has been an influx of techniques from many different areas of mathematics that have each contributed significant progress on the Kakeya conjecture. One may still need a couple more key ideas before the problem is finally solved in full, and such ideas may come from a quite unexpected source, but with the current rate of progress I am now optimistic that we will continue to see significant advances in this area in the near future.

Lemma 1 (Regularity lemma via random neighbourhoods) Let . Then there exists integers with the following property: whenever be a graph on finitely many vertices, if one selects one of the integers at random from , then selects vertices uniformly from at random, then the vertex cells (some of which can be empty) generated by the vertex neighbourhoods for , will obey the regularity property

with probability at least , where the sum is over all pairs for which is not -regular between and . [Recall that a pair is -regular for if one has

for any and with , where is the density of edges between and .]

The proof was a combinatorial one, based on the standard energy increment argument.

In this post I would like to discuss an alternate approach to the regularity lemma, which is an infinitary approach passing through a graph-theoretic version of the Furstenberg correspondence principle (mentioned briefly in this earlier post of mine). While this approach superficially looks quite different from the combinatorial approach, it in fact uses many of the same ingredients, most notably a reliance on random neighbourhoods to regularise the graph. This approach was introduced by myself back in 2006, and used by Austin and by Austin and myself to establish some property testing results for hypergraphs; more recently, a closely related infinitary hypergraph removal lemma developed in the 2006 paper was also used by Austin to give new proofs of the multidimensional Szemeredi theorem and of the density Hales-Jewett theorem (the latter being a spinoff of the polymath1 project).

For various technical reasons we will not be able to use the correspondence principle to recover Lemma 1 in its full strength; instead, we will establish the following slightly weaker variant.

Lemma 2 (Regularity lemma via random neighbourhoods, weak version) Let . Then there exist an integer with the following property: whenever be a graph on finitely many vertices, there exists such that if one selects vertices uniformly from at random, then the vertex cells generated by the vertex neighbourhoods for , will obey the regularity property (1) with probability at least .

Roughly speaking, Lemma 1 asserts that one can regularise a large graph with high probability by using random neighbourhoods, where is chosen at random from one of a number of choices ; in contrast, the weaker Lemma 2 asserts that one can regularise a large graph with high probability by using some integer from , but the exact choice of depends on , and it is not guaranteed that a randomly chosen will be likely to work. While Lemma 2 is strictly weaker than Lemma 1, it still implies the (weighted) Szemerédi regularity lemma (Lemma 2 from the previous post).

— 1. The graph correspondence principle —

The first key tool in this argument is the graph correspondence principle, which takes a sequence of (increasingly large) graphs and uses random sampling to extract an infinitary limit object, which will turn out to be an infinite but random (and, crucially, exchangeable) graph. This concept of a graph limit is related to (though slightly different from) the “graphons” used as graph limits in the work of Lovasz and Szegedy, or the ultraproducts used in the work of Elek and Szegedy. It also seems to be related to the concept of an elementary limit that I discussed in this post, though this connection is still rather tentative.

The correspondence works as follows. We start with a finite, deterministic graph . We can then form an infinite, random graph from this graph by the following recipe:

• The vertex set of will be the integers .
• For every integer , we randomly select a vertex in , uniformly and independently at random. (Note that there will be many collisions, i.e. integers for which , but these collisions will become asymptotically negligible in the limit .)
• We then define the edge set of by declaring to be an edge on if and only if is an edge in (which in particular requires ).

More succinctly, is the pullback of under a random map from to .

The random graph captures all the “local” information of , while obscuring all the “global” information. For instance, the edge density of is essentially just the probability that a given edge, say , lies in . (There is a small error term due to the presence of collisions, but this goes away in the limit .) Similarly, the triangle density of is essentially the probability that a given triangle, say , lies in . On the other hand, it is difficult to read off global properties of , such as being connected or -colourable, just from .

At first glance, it may seem a poor bargain to trade in a finite deterministic graph for an infinite random graph , which is a more complicated and less elementary object. However, there are three major advantages of working with rather than :

• Exchangeability. The probability distribution of has a powerful symmetry or exchangeability property: if one takes the random graph and interchanges any two vertices in , e.g. and , one obtains a new graph which is not equal to , but nevertheless has the same probability distribution as , basically because the were selected in an iid manner. More generally, given any permutation , the pullback of by has the same probability distribution as ; thus we have a measure-preserving action of the symmetric group , which places us in the general framework of ergodic theory.
• Limits. The space of probability measures on the space of infinite graphs is sequentially compact; given any sequence of infinite random graphs, one can find a subsequence which converges in the vague topology to another infinite random graph. What this means is that given any event on infinite graphs that involve only finitely many of the edges, the probability that obeys converges to the probability that obeys . (Thus, for instance, the probability that contains the triangle will converge to the probability that contains the same triangle.) Note that properties that involve infinitely many edges (e.g. connectedness) need not be preserved under vague limits.
• Factors. The underlying probability space for the random variable is the space of infinite graphs, and it is natural to give this space the Borel -algebra , which is the -algebra generated by the cylinder events “” for . But this -algebra also has a number of useful sub--algebras or factors, representing various partial information on the graph . In particular, given any subset of , one can create the factor , defined as the -algebra generated by the events “” for . Thus for instance, the event that contains the triangle is measurable in , but not in . One can also look at compound factors such as , the factor generated by the union of and . For instance, the event that contains the edges is measurable in , but the event that contains the triangle is not.

The connection between the infinite random graph and partitioning through random neighbourhoods comes when contemplating the relative difference between a factor such as and (say). The latter factor is generated by the former factor, together with the events “” for . But observe if is generated from a finite deterministic graph , then lies in if and only if lies in the vertex neighbourhood of . Thus, if one uses the vertex neighbourhoods of to subdivide the original vertex set into cells of varying sizes, the factor is generated from , together with the random variable that computes which of these cells the random vertex falls into. We will see this connection in more detail later in this post, when we use the correspondence principle to prove Lemma 2.

Combining the exchangeability and limit properties (and noting that the vague limit of exchangeable random graphs is still exchangeable), we obtain

Lemma 3 (Graph correspondence principle) Let be a sequence of finite deterministic graphs, and let be their infinite random counterparts. Then there exists a subsequence such that converges in the vague topology to an exchangeable infinite random graph .

We can illustrate this principle with three main examples, two from opposing extremes of the “dichotomy between structure and randomness”, and one intermediate one.

Example 1 (Random example) Let be a sequence of -regular graphs of edge density , where , , and as . Then any graph limit of this sequence will be an Erdös-Rényi graph , where each edge lies in with an independent probability of .

Example 2 (Structured example) Let be a sequence of complete bipartite graphs, where the two cells of the bipartite graph have vertex density and respectively, with and . Then any graph limit of this sequence will be a random complete bipartite graph, constructed as follows: first, randomly colour each vertex of red with probability and blue with probability , independently for each vertex. Then define to be the complete bipartite graph between the red vertices and the blue vertices.

Example 3 (Random+structured example) Let be a sequence of incomplete bipartite graphs, where the two cells of the bipartite graph have vertex density and respectively, and the graph is -regular between these two cells with edge density , with , , , and . Then any graph limit of this sequence will be a random bipartite graph, constructed as follows: first, randomly colour each vertex of red with probability and blue with probability , independently for each vertex. Then define to be the bipartite graph between the red vertices and the blue vertices, with each edge between red and blue having an independent probability of of lying in .

One can use the graph correspondence principle to prove statements about finite deterministic graphs, by the usual compactness and contradiction approach: argue by contradiction, create a sequence of finite deterministic graph counterexamples, use the correspondence principle to pass to an infinite random exchangeable limit, and obtain the desired contradiction in the infinitary setting. (See this earlier blog post of mine for further discussion.) This will be how we shall approach the proof of Lemma 2.

— 2. The infinitary regularity lemma —

To prove the finitary regularity lemma via the correspondence principle, one must first develop an infinitary counterpart. We will present this infinitary regularity lemma (first introduced in this paper) shortly, but let us motivate it by a discussion based on the three model examples of infinite exchangeable graphs from the previous section.

First, consider the “random” graph from Example 1. Here, we observe that the events “” are jointly independent of each other, thus for instance

More generally, we see that the factors for all distinct are independent, which means that

whenever and the are distinct.

Next, we consider the “structured” graph from Example 2, where we take to avoid degeneracies. In contrast to the preceding example, the events “” are now highly dependent; for instance, if and , then this forces to lie outside of , despite the fact that the events “” each occur with a non-zero probability of . In particular, the factors are not jointly independent.

However, one can recover a conditional independence by introducing some new factors. Specifically, let be the factor generated by the event that the vertex is coloured red. Then we see that the factors now become conditionally jointly independent, relative to the base factor , which means that we have conditional independence identities such as

Indeed, once one fixes (conditions) the information in (i.e. once one knows what colour the vertices are), the events “” for either occur with probability (if have distinct colours) or probability (if have the same colour), and so the conditional independence is trivially true.

A similar phenomenon holds for the “random+structured” graph from Example 3, with . Again, the factors are not jointly independent in an absolute sense, but once one introduces the factors based on the colour of the vertex , we see once again that the become conditionally jointly independent relative to the .

These examples suggest, more generally, that we should be able to regularise the graph (or more precisely, the system of edge factors ) by introducing some single-vertex factors , with respect to which the edge factors become conditionally independent; this is the infinitary analogue of a finite graph becoming -regular relative to a suitably chosen partition of the vertex set into cells.

Now, in Examples 2, 3 we were able to obtain this regularisation because the vertices of the graph were conveniently coloured for us (red or blue). But for general infinite exchangeable graphs , such a vertex colouring is not provided to us, so how is one to generate the vertex factors ?

The key trick – which is the infinitary analogue of using random neighbourhoods to regularise a finitary graph – is to sequester half of the infinite vertices in – e.g. the negative vertices – away as “reference” or “training” vertices, and then and colorise the remaining vertices of the graph based on how that vertex interacts with the reference vertices. More formally, we define for by the formula

We then have

Lemma 4 (Infinitary regularity lemma) Let be a infinite exchangeable random graph. Then the for natural numbers are conditinally jointly independent relative to the . More precisely, if is a set of natural numbers, is a subset of , and is a -measurable event for all , then

Proof: By induction on , it suffices to show that for any , the event and the event are independent relative to .

By relabeling we may take and for some . We use the exchangeability of (and Hilbert’s hotel) to observe that the random variables

and

have the same distribution; in particular, they have the same norm. By Pythagoras’ theorem, they must therefore be equal almost surely; furthermore, for any intermediate -algebra between and , is also equal almost surely to the above two expressions. (The astute reader will observe that we have just run the “energy increment argument”; in the infinitary world, it is somewhat slicker than in the finitary world, due to the convenience of the Hilbert’s hotel trick, and the fact that the existence of orthogonal projections (and in particular, conditional expectation) is itself encoding an energy increment argument.)

As a special case of the above observation, we see that

In particular, this implies that is conditionally independent of every event measurable in , relative to , and the claim follows.

We remark that the same argument also allows one to easily regularise infinite exchangeable hypergraphs; see my paper for details. In fact one can go further and obtain a structural theorem for these hypergraphs generalising de Finetti’s theorem, and also closely related to the graphons of Lovasz and Szegedy; see this paper of Austin for details.

— 3. Proof of finitary regularity lemma —

Having proven the infinitary regularity lemma, we now use the correspondence principle and the compactness and contradiction argument to recover the finitary regularity lemma, Lemma 2.

Suppose this lemma failed. Carefully negating all the quantifiers, this means that there exists , a sequence going to infinity, and a sequence of finite deterministic graphs such that for every , if one selects vertices uniformly from , then the vertex cells generated by the vertex neighbourhoods for , will obey the regularity property (1) with probability less than .

We convert each of the finite deterministic graphs to an infinite random exchangeable graph ; invoking the ocrrespondence principle and passing to a subsequence if necessary, we can assume that this graph converges in the vague topology to an exchangeable limit . Applying the infinitary regularity lemma to this graph, we see that the edge factors for natural numbers are conditionally jointly independent relative to the vertex factors .

Now for any distinct natural numbers , let be the indicator of the event “ lies in “, thus when lies in and otherwise. Clearly is -measurable. We can write

where

and

The exchangeability of ensures that are exchangeable with respect to permutations of the natural numbers, in particular and .

By the infinitary regularity lemma, the are jointly independent relative to the , and also have mean zero relative to these factors, so in particular they are infinitely pseudorandom in the sense that

Meanwhile, the random variable is measurable with respect to the factor , which is the limit of the factors as increases. Thus, given any (to be chosen later), one can find an approximation to , bounded between and , which is -measurable for some , and such that

We can also impose the symmetry condition . Now let be an extremely small number (depending on ) to be chosen later. Then one can find an approximation to , bounded between and , which is -measurable for some , and such that

Again we can impose the symmetry condition . We can then extend by exchangeability, so that

for all distinct natural numbers . By the triangle inequality we then have

and by a separate application of the triangle inequality

The bounds (2), (3) apply to the limiting infinite random graph . On the other hand, all the random variables appearing in (2), (3) involve at most finitely many of the edges of the graph. Thus, by vague convergence, the bounds (2), (3) also apply to the graph for sufficiently large .

Now we unwind the definitions to move back to the finite graphs . Observe that, when applied to the graph , one has

where is a symmetric function which is constant on the pairs of cells generated the vertex neighbourhoods of . Similarly,

for some symmetric function . The estimate (2) can then be converted to a uniformity estimate on

while the estimate (3) can be similarly converted to

If one then repeats the arguments in the preceding blog post, we conclude (if is sufficiently small depending on , and is sufficiently small depending on , , ) that for of the choices for , the partition induced by the corresponding vertex neighbourhoods will obey (1). But this contradicts the construction of the , and the claim follows.

Lemma 1 (Regularity lemma, standard version) Let be a graph on vertices, and let and . Then there exists a partition of the vertices , with bounded below by and above by a quantity depending only on , obeying the following properties:

• (Equitable partition) For any , the cardinalities of and differ by at most .
• (Regularity) For all but at most pairs , the portion of the graph between and is -regular in the sense that one has

  

  for any and with , where is the density of edges between and .

This lemma becomes useful in the regime when is very large compared to or , because all the conclusions of the lemma are uniform in . Very roughly speaking, it says that “up to errors of size “, a large graph can be more or less described completely by a bounded number of quantities . This can be interpreted as saying that the space of all graphs is totally bounded (and hence precompact) in a suitable metric space, thus allowing one to take formal limits of sequences (or subsequences) of graphs; see for instance this paper of Lovasz and Szegedy for a discussion.

For various technical reasons it is easier to work with a slightly weaker version of the lemma, which allows for the cells to have unequal sizes:

Lemma 2 (Regularity lemma, weighted version) Let be a graph on vertices, and let . Then there exists a partition of the vertices , with bounded above by a quantity depending only on , obeying the following properties:

• (Regularity) One has

  

  where the sum is over all pairs for which is not -regular between and .

While Lemma 2 is, strictly speaking, weaker than Lemma 1 in that it does not enforce the equitable size property between the atoms, in practice it seems that the two lemmas are roughly of equal utility; most of the combinatorial consequences of Lemma 1 can also be proven using Lemma 2. The point is that one always has to remember to weight each cell by its density , rather than by giving each cell an equal weight as in Lemma 1. Lemma 2 also has the advantage that one can easily generalise the result from finite vertex sets to other probability spaces (for instance, one could weight with something other than the uniform distribution). For applications to hypergraph regularity, it turns out to be slightly more convenient to have two partitions (coarse and fine) rather than just one; see for instance my own paper on this topic. In any event the arguments below that we give to prove Lemma 2 can be modified to give a proof of Lemma 1 also. The proof of the regularity lemma is usually conducted by a greedy algorithm. Very roughly speaking, one starts with the trivial partition of . If this partition already regularises the graph, we are done; if not, this means that there are some sets and in which there is a significant density fluctuation beyond what has already been detected by the original partition. One then adds these sets to the partition and iterates the argument. Every time a new density fluctuation is incorporated into the partition that models the original graph, this increases a certain “index” or “energy” of the partition. On the other hand, this energy remains bounded no matter how complex the partition, so eventually one must reach a long “energy plateau” in which no further refinement is possible, at which point one can find the regular partition.

One disadvantage of the greedy algorithm is that it is not efficient in the limit , as it requires one to search over all pairs of subsets of a given pair of cells, which is an exponentially long search. There are more algorithmically efficient ways to regularise, for instance a polynomial time algorithm was given by Alon, Duke, Lefmann, Rödl, and Yuster. However, one can do even better, if one is willing to (a) allow cells of unequal size, (b) allow a small probability of failure, (c) have the ability to sample vertices from at random, and (d) allow for the cells to be defined “implicitly” (via their relationships with a fixed set of reference vertices) rather than “explicitly” (as a list of vertices). In that case, one can regularise a graph in a number of operations bounded in . Indeed, one has

Lemma 3 (Regularity lemma via random neighbourhoods) Let . Then there exists integers with the following property: whenever be a graph on finitely many vertices, if one selects one of the integers at random from , then selects vertices uniformly from at random, then the vertex cells (some of which can be empty) generated by the vertex neighbourhoods for , will obey the conclusions of Lemma 2 with probability at least .

Thus, roughly speaking, one can regularise a graph simply by taking a large number of random vertex neighbourhoods, and using the partition (or Venn diagram) generated by these neighbourhoods as the partition. The intuition is that if there is any non-uniformity in the graph (e.g. if the graph exhibits bipartite behaviour), this will bias the random neighbourhoods to seek out the partitions that would regularise that non-uniformity (e.g. vertex neighbourhoods would begin to fill out the two vertex cells associated to the bipartite property); if one takes sufficiently many such random neighbourhoods, the probability that all detectable non-uniformity is captured by the partition should converge to . (It is more complicated than this, because the finer one makes the partition, the finer the types of non-uniformity one can begin to detect, but this is the basic idea.)

This fact seems to be reasonably well-known folklore, discovered independently by many authors; it is for instance quite close to the graph property testing results of Alon and Shapira, and also appears implicitly in a paper of Ishigami, as well as a paper of Austin (and perhaps even more implicitly in a paper of myself). However, in none of these papers is the above lemma stated explicitly. I was asked about this lemma recently, so I decided to provide a proof here.

— 1. Warmup: a weak regularity lemma —

To motivate the idea, let’s first prove a weaker but simpler (and more quantitatively effective) regularity lemma, analogous to that established by Frieze and Kannan:

Lemma 4 (Weak regularity lemma via random neighbourhoods) Let . Then there exists an integer with the following property: whenever be a graph on finitely many vertices, if one selects at random, then selects vertices uniformly from at random, then the vertex cells (some of which can be empty) generated by the vertex neighbourhoods for , obey the following property with probability at least : for any vertex sets , the number of edges connecting and can be approximated by the formula

This weaker lemma only lets us count “macroscopic” edge densities , when are dense subsets of , whereas the full regularity lemma is stronger in that it also controls “microscopic” edge densities where are now dense subsets of the cells . Nevertheless this weaker lemma is easier to prove and already illustrates many of the ideas.

Let’s now prove this lemma. Fix , let be chosen later, let be a graph, and select at random. (There can of course be many vertices selected more than once; this will not bother us.) Let and be as in the above lemma. For notational purposes it is more convenient to work with the (random) -algebra generated by the (i.e. the collection of all sets that can be formed from by boolean operations); this is an atomic -algebra whose atoms are precisely the (non-empty) cells in the partition. Observe that these -algebras are nested: .

We will use the trick of turning sets into functions, and view the graph as a function . One can then form the conditional expectation of this function to the product -algebra , whose value on is simply the average value of on the product set . (When and are different, this is simply the edge density ). One can view more combinatorially, as a weighted graph on such that all edges between two distinct cells , have the same constant weight of .

We give (and ) the uniform probability measure, and define the energy at time to be the (random) quantity

one can interpret this as the mean square of the edge densities , weighted by the size of the cells . From Pythagoras’ theorem we have the identity

for all ; in particular, the are increasing in . This implies that the expectations are also increasing in . On the other hand, these expectations are bounded between and . Thus, if we select at random, expectation of

telescopes to be . Thus, by Markov’s inequality, with probability we can freeze such that we have the conditional expectation bound

Suppose have this property. We split

where

and

We now assert that the partition induced by obeys the conclusions of Lemma 3. For this, we observe various properties on the two components of :

Lemma 5 ( is structured) is constant on each product set .

Proof: This is clear from construction.

Lemma 6 ( is pseudorandom) The expression

is of size .

Proof: The left-hand side can be rewritten as

Observe that the function is measurable with respect to , so we can rewrite this expression as

Applying Cauchy-Schwarz, one can bound this by

But from Pythagoras we have

and so the claim follows from (3) and another application of Cauchy-Schwarz.

Now we can prove Lemma 4. Observe that

Applying Cauchy-Schwarz twice in and using Lemma 6, we see that the RHS is ; choosing we obtain the claim.

— 2. Strong regularity via random neighbourhoods —

We now prove Lemma 3, which of course implies Lemma 2.

Fix and a graph on vertices. We randomly select an infinite sequence of vertices in , drawn uniformly and independently at random. We define , , as before.

Now let be a large number depending on to be chosen later, let be a rapidly growing function (also to be chosen later), and set and for all , thus grows rapidly to infinity. The expected energies are increasing from to , thus if we pick uniformly at random, the expectation of

telescopes to be . Thus, by Markov’s inequality, with probability we will have

Assume that is chosen to obey this. Then, by another application of the pigeonhole principle, we can find such that

Fix this . We have

so by Markov’s inequality, with probability , are such that

and also obey the conditional expectation bound

Assume that this is the case. We split

where

We now assert that the partition induced by obeys the conclusions of Lemma 2. For this, we observe various properties on the three components of :

Lemma 7 ( locally constant) is constant on each product set .

Proof: This is clear from construction.

Lemma 8 ( small) We have .

Proof: This follows from (4) and Pythagoras’ theorem.

Lemma 9 ( uniform) The expression

is of size .

Proof: This follows by repeating the proof of Lemma 6, but using (5) instead of (3).

Now we verify the regularity.

First, we eliminate small atoms: the pairs for which clearly give a net contribution of at most and are acceptable; similarly for those pairs for which . So we may henceforth assume that

Now, let , have densities

then

We divide into the three pieces , , .

The contribution of is exactly .

The contribution of can be bounded using Cauchy-Schwarz as

Using Lemma 8 and Chebyshev’s inequality, we see that the pairs for which this quantity exceeds will contribute at most to (1), which is acceptable if we choose so that . Let us now discard these bad pairs.

Finally, the contribution of can be bounded by two applications of Cauchy-Schwarz and (9) as

which by (6) is bounded by

This can be made by selecting sufficiently rapidly growing depending on . Putting this all together we see that

which (since ) gives the desired regularity.

Remark 1 Of course, this argument gives tower-exponential bounds (as is exponential and needs to be iterated times), which will be familiar to any reader already acquainted with the regularity lemma.

Remark 2 One can take the partition induced by random neighbourhoods here and carve it up further to be both equitable and (mostly) regular, thus recovering a proof of Lemma 1, by following the arguments in this paper of mine. Of course, when one does so, one no longer has a partition created purely from random neighbourhoods, but it is pretty clear that one is not going to be able to make an equitable partition just from boolean operations applied to a few random neighbourhoods.

As an example, if P is the square in with vertices (0,0),(2,0),(0,2),(2,2), then , and the volume of the square in is 4.

Background:

is a d-dimensional polytope if :

1) P is the intersection of finitely many halfspaces.

2) P is bounded

3) P is d-dimensional

Some examples of such polytopes in are cubes, rectangular prisms, simplices, octahedra, dodecahedra, etc. A square in is a polytope but is not 3-dimensional this is not an example of such a .

Theorem 1 ( density Hales-Jewett theorem) Let . Then if is a sufficiently large integer, any subset of the cube of density at least contains at least one combinatorial line , where is a string of s, s, s, and ‘s containing at least one “wildcard” , and is the string formed from by replacing all ‘s with ‘s.

The full density Hales-Jewett theorem is the same statement, but with replaced by for some . (The case is trivial, and the case follows from Sperner’s theorem.)

This theorem was first proven by Furstenberg and Katznelson, by first converting it to a statement in ergodic theory; the original paper of Furstenberg-Katznelson argument was for the case only, and gave only part of the proof in detail, but in a subsequent paper a full proof in general was provided. The remaining components of the original argument were later completed in unpublished notes of McCutcheon. One of the new proofs is essentially a finitary translation of this argument; in principle one could also finitise the significantly more complicated argument of Furstenberg and Katznelson for general , but this has not been properly carried out yet (the other two proofs are likely to generalise much more easily to higher ). The result is considered quite deep; for instance, the general case of the density Hales-Jewett theorem already implies Szemerédi’s theorem, which is a highly non-trivial theorem in its own right, as a special case.

Another of the proofs is based primarily on the density increment method that goes back to Roth, and also incorporates some ideas from a paper of Ajtai and Szemerédi establishing what we have called the “corners theorem” (and which is also implied by the case of the density Hales-Jewett theorem). A key new idea involved studying the correlations of the original set with special subsets of , such as -insensitive sets, or intersections of -insensitive and -insensitive sets. Work is currently ongoing to extend this argument to higher , and it seems that there are no major obstacles in doing so.

This correlations idea inspired a new ergodic proof of the density Hales-Jewett theorem for all values of by Austin, which is in the spirit of the triangle removal lemma (or hypergraph removal lemma) proofs of Roth’s theorem (or the multidimensional Szemerédi theorem). A finitary translation of this argument in the case has been sketched out; I believe it also extends in a relatively straightforward manner to the higher case (in analogy with some proofs of the hypergraph removal lemma ).

— 1. Simpler cases of density Hales-Jewett —

In order to motivate the known proofs of the density Hales-Jewett theorem, it is instructive to consider some simpler theorems which are implied by this theorem. The first is the corners theorem of Ajtai and Szemerédi:

Theorem 2 (Corners theorem) Let . Then if is a sufficiently large integer, any subset of the square of density at least contains at least one right-angled triangle (or “corner”) with .

The density Hales-Jewett theorem implies the corners theorem; this is proven by utilising the map from the cube to the square, defined by mapping a string to a pair , where are the number of s and s respectively in . The key point is that maps combinatorial lines to corners. (Strictly speaking, this mapping only establishes the corners theorem for dense subsets of , but it is not difficult to obtain the general case from this by replacing by and using translation-invariance.)

(The corners theorem is also closely related to the problem of finding dense sets of points in a triangular grid without any equilateral triangles, a problem which we have called Fujimura’s problem.

The corners theorem in turn implies

Theorem 3 (Roth’s theorem) Let . Then if is a sufficiently large integer, any subset of the interval of density at least contains at least one arithmetic progression of length three.

Roth’s theorem can be deduced from the corners theorem by considering the map defined by ; the key point is that maps corners to arithmetic progressions of length three.

There are higher analogues of these implications; the general version of the density Hales-Jewett theorem implies a general version of the corners theorem known as the multidimensional Szemerédi theorem, which in term implies a general version of Roth’s theorem known as Szemerédi’s theorem.

— 2. The density increment argument —

The strategy of the density increment argument, which goes back to Roth’s proof of Theorem 3 in 1956, is to perform a downward induction on the density . Indeed, the theorem is obvious for high enough values of ; for instance, if , then partitioning the cube into lines and applying the pigeonhole principle will already give a combinatorial line. So the idea is to deduce the claim for a fixed density from that of a higher density .

A key concept here is that of an -dimensional combinatorial subspace of – a set of the form , where is a string formed using the base alphabet and wildcards (with each wildcard appearing at least opnce), and is the string formed by substituting for for each . (Thus, for instance, a combinatorial line is a combinatorial subspace of dimension .) The identification between and the combinatorial space maps combinatorial lines to combinatorial lines. Thus, to prove Theorem 1, it suffices to show

Proposition 4 (Lack of lines implies density increment) Let . Then if is a sufficiently large integer, and has density at least and has no combinatorial lines, then there exists an -dimensional subspace of on which has density at least , where depends only on (and is bounded away from zero on any compact range of ), and for some function that goes to infinity as for fixed .

It is easy to see that Proposition 4 implies Theorem 1 (for instance, one could consider the infimum of all for which the theorem holds, and show that having this infimum non-zero would lead to a contradiction).

Now we have to figure out how to get that density increment. The original argument of Roth relied on Fourier analysis, which in turn relies on an underlying translation-invariant structure which is not present in the density Hales-Jewett setting. (Arithmetic progressions are translation-invariant, but combinatorial lines are not.) It turns out that one can proceed instead by adapting a (modification of) an argument of Ajtai and Szemerédi, which gave the first proof of Theorem 2.

The (modified) Ajtai-Szemerédi argument uses the density increment method, assuming that has no right-angled triangles and showing that has an increased density on a subgrid – a product of fairly long arithmetic progressions with the same spacing. The argument proceeds in two stages, which we describe slightly informally (in particular, glossing over some technical details regarding quantitative parameters such as ) as follows:

• Step 1. If is dense but has no right-angled triangles, then has an increased density on a cartesian product of dense sets (which are not necessarily arithmetic progressions).
• Step 2. Any Cartesian product in can be partitioned into reasonably large grids , plus a remainder term of small density.

From Step 1, Step 2 and the pigeonhole principle we obtain the desired density increment of on a grid , and then the density increment argument gives us the corners theorem.

Step 1 is actually quite easy. If is dense, then it must also be dense on some diagonal , by the pigeonhole principle. Let and denote the rows and columns that occupies. Every pair of points in forms the hypotenuse of some corner, whose third vertex lies in . Thus, if has no corners, then must avoid all the points formed by (except for those of the diagonal ). Thus has a significant density decrease on the Cartesian product . Dividing the remainder into three further Cartesian products , , and using the pigeonhole principle we obtain the claim (after redefining appropriately).

Step 2 can be obtained by iterating a one-dimensional version:

• Step 2a. Any set can be partitioned into reasonably long arithmetic progressions , plus a remainder term of small density.

Indeed, from Step 2a, one can partition into products (plus a small remainder), which can be easily repartitioned into grids (plus small remainder). This partitions into sets (plus small remainder). Applying Step 2a again, each can be partitioned further into progressions (plus small remainder), which allows us to partition each into grids (plus small remainder).

So all one has left to do is establish Step 2a. But this can be done by the greedy algorithm: locate one long arithmetic progression in and remove it from , then locate another to remove, and so forth until no further long progressions remain in the set. But Szemerédi’s theorem then tells us the remaining set has low density, and one is done!

This argument has the apparent disadvantage of requiring a deep theorem (Szemerédi’s theorem) in order to complete the proof. However, interestingly enough, when one adapts the argument to the density Hales-Jewett theorem, one gets to replace Szemerédi’s theorem by a more elementary result – one which in fact follows from the (easy) version of the density Hales-Jewett theorem, i.e. Sperner’s theorem.

We first need to understand the analogue of the Cartesian products . Note that is the intersection of a “vertically insensitive set” and a “horizontally insensitive set” . By “vertically insensitive” we mean that membership of a point in that set is unaffected if one moves that point in a vertical direction, and similarly for “horizontally insensitive”. In a similar fashion, define a “-insensitive set” to be a subset of , membership in which is unaffected if one flips a coordinate from a to a or vice versa (e.g. if lies in the set, then so must , , , etc.). Similarly define the notion of a “-insensitive set”. We then define a “complexity set” to be the intersection of a -insensitive set and a -insensitive set ; these are analogous to the Cartesian products .

(For technical reasons, one actually has to deal with local versions of insensitive sets and complexity sets, in which one is only allowed to flip a moderately small number of the coordinates rather than all of them. But to simplify the discussion let me ignore this (important) detail, which is also a major issue to address in the other two proofs of this theorem.)

The analogues of Steps 1, 2 for the density Hales-Jewett theorem are then

• Step 1. If is dense but has no combinatorial lines, then has an increased density on a (local) complexity set .
• Step 2. Any (local) complexity set can be partitioned into moderately large combinatorial subspaces (plus a small remainder).

We can sketch how Step 1 works as follows. Given any , let denote the string formed by replacing all s with s, e.g. . Similarly define . Observe that forms a combinatorial line (except in the rare case when doesn’t contain any s). Thus if we let , , we see that must avoid essentially all of . On the other hand, observe that and are -insensitive and -insensitive sets respectively. Taking complements and using the same sort of pigeonhole argument as before, we obtain the claim. (Actually, this argument doesn’t quite work because , could be very sparse; this problem can be fixed, but requires one to use local complexity sets rather than global ones, and also to introduce the concept of “equal-slices measure”; I will not discuss these issues here.)

Step 2 can be reduced, much as before, to the following analogue of Step 2a:

• Step 2a. Any -insensitive set can be partitioned into moderately large combinatorial subspaces (plus a small remainder).

Identifying the letters and together, one can quotient down to ; the preimages of this projection are precisely the -insensitive sets. Because of this, Step 2a is basically equivalent (modulo some technicalities about measure) to

• Step 2a’. Any can be partitioned into moderately large combinatorial subspaces (plus a small remainder).

By the greedy algorithm, we will be able to accomplish this step if we can show that every dense subset of contains moderately large subspaces. But this turns out to be possible by carefully iterating Sperner’s theorem (which shows that every dense subset of contains combinatorial lines).

This proof of Theorem 1 seems to extend without major difficulty to the case of higher , though details are still being worked out.

— 3. The triangle removal argument —

The triangle removal lemma of Ruzsa and Szemerédi is a graph-theoretic result which implies the corners theorem (and hence Roth’s theorem). It asserts the following:

Lemma 5 (Triangle removal lemma) For every there exists such that if a graph on vertices has fewer than triangles, then the triangles can be deleted entirely by removing at most edges.

Let’s see how the triangle removal lemma implies the corners theorem. A corner is, of course, already a triangle in the geometric sense, but we need to convert it to a triangle in the graph-theoretic sense, as follows. Let be a subset of with no corners; the aim is to show that has small density. Let be the set of all horizontal lines in , the set of vertical lines, and the set of diagonal lines (thus all three sets have size about ). We create a tripartite graph on the vertex sets by joining a horizontal line to a vertical line whenever and intersect at a point in , and similarly connecting or to . Observe that a triangle in corresponds either to a corner in , or to a “degenerate” corner in which the horizontal, vertical, and diagonal line are all concurrent. In particular, there are very few triangle in , which can then be deleted by removing a small number of edges from by the triangle removal lemma. But each edge removed can delete at most one degenerate corner, and the number of degenerate corners is , and so is small as required.

All known proofs of the triangle removal lemma proceed by some version of the following three steps:

• “Regularity lemma step”: Applying tools such as the Szemerédi regularity lemma, one can partition the graph into components between cells of vertices, such that most of the are “pseudorandom”. One way to define what pseudorandom means is to view each graph component as a subset of the Cartesian product , in which case is pseudorandom if it does not have a significant density increment on any smaller Cartesian product of non-trivial size.
• ”Counting lemma step”: By exploiting the pseudorandomness property, one shows that if has a triple of dense pseudorandom graphs between cells of non-trivial size, then this triple must generate a large number of triangles; hence, if has very few triangles, then one cannot find such a triple of dense pseudorandom graphs.
• ”Cleaning step”: If one then removes all components of which are too sparse or insufficiently pseudorandom, one can thus eliminate all triangles.

Pulling this argument back to the corners theorem, we see that cells such as will correspond either to horizontally insensitive sets, vertically insensitive sets, or diagonally insensitive sets. Thus this proof of the corners theorem proceeds by partitioning in three different ways into insensitive sets in such a way that is pseudorandom with respect to many of the cells created by any two of these partitions, counting the corners generated by any triple of large cells in which A is pseudorandom and dense, and cleaning out all the other cells.

It turns out that a variant of this argument can give Theorem 1; this was in fact the original approach studied by the polymath1 project, though it was only after a detour through ergodic theory (as well as the development of the density-increment argument discussed above) that the triangle-removal approach could be properly executed. In particular, an ergodic argument based on the infinitary analogue of the triangle removal lemma (and its hypergraph generalisations) was developed by Austin, which then inspired the combinatorial version sketched here.

The analogue of the vertex cells are given by certain -insensitive sets , -insensitive sets , and -insensitive sets . Roughly speaking, a set would be said to be pseudorandom with respect to a cell if has no further density increment on any smaller cell with a -insensitive subset of , and a -insensitive subset of . (This is an oversimplification, glossing over an important refinement of the concept of pseudorandomness involving the discrepancy between global densities in and local densities in subspaces of .) There is a similar notion of being pseudorandom with respect to a cell or .

We briefly describe the “regularity lemma” step. By modifying the proof of the regularity lemma, one can obtain three partitions

into -insensitive, -insensitive, and -insensitive components respectively, where are not too large, and is pseudorandom with respect to most cells , , and .

In order for the counting step to work, one also needs an additional “stationarity” reduction, which is difficult to state precisely, but roughly speaking asserts that the “local” statistics of sets such as on medium-dimensional subspaces are close to the corresponding “global” statistics of such sets; this can be achieved by an additional pigeonholing argument. We will gloss over this issue, pretending that there is no distinction between local statistics and global statistics. (Thus, for instance, if has large global density in , we shall assume that also has large density on most medium-sized subspaces of .)

Now for the “counting lemma” step. Suppose we can find such that the cells are large, and that intersects , , and in a dense pseudorandom manner. We claim that this will force to have a large number of combinatorial lines , with in , in , and in . Because of the dense pseudorandom nature of in these cells, it turns out that it will suffice to show that there are a lot of lines with , , and .

One way to generate a line is by taking the triple , where is a generic point. (Actually, as we will see below, we would have to to a subspace of before using this recipe to generate lines.) Then we need to find many obeying the constraints

Because of the various insensitivity properties, many of these conditions are redundant, and we can simplify to

Now note that the property “” is 123-insensitive; it is simultaneously 12-insensitive, 23-insensitive, and 13-insensitive. As is assumed to be large, there will be large combinatorial subspaces on which (a suitably localised version of) this property “” will be always true. Localising to this space (taking advantage of the stationarity properties alluded to earlier), we are now looking for solutions to

We’ll pick to be of the form for some . We can then rewrite the constraints on as

The property “” is 123-invariant, and is large, so by arguing as before we can pass to a large subspace where this property is always true. The largeness of then gives us a large number of solutions.

Taking contrapositives, we conclude that if in fact has no combinatorial lines, then there do not exist any triple of large cells with respect to which is dense and pseudorandom. This forces to be confined either to very small cells, or to very sparse subsets of cells, or to the rare cells which fail to be pseudorandom. None of these cases can contribute much to the density of , and so itself is very sparse – contradicting the hypothesis in Theorem 1 that is dense (this is the “cleaning step”). This concludes the sketch of the triangle-removal proof of this theorem.

The ergodic version of this argument, due to Austin, works for all values of , so I expect the combinatorial version to do so as well.

— 4. The finitary Furstenberg-Katznelson argument —

In 1989, Furstenberg and Katznelson gave the first proof of Theorem 1, by translating it into a recurrence statement about a certain type of stationary process indexed by an infinite cube . This argument was inspired by a long string of other successful proofs of density Ramsey theorems via ergodic means, starting with the initial 1977 paper of Furstenberg giving an ergodic theory proof of Szemeredi’s theorem. The latter proof was transcribed into a finitary language by myself, so it was reasonable to expect that the Furstenberg-Katznelson argument could similarly be translated into a combinatorial framework.

Let us first briefly describe the original strategy of Furstenberg to establish Roth’s theorem, but phrased in an informal, and vaguely combinatorial, language. The basic task is to get a non-trivial lower bound on averages of the form

where we will be a bit vague about what are ranging over, and where is some non-negative function of positive mean. It is then natural to study more general averages of the form

Now, it turns out that certain types of functions give a negligible contribution to expressions such as (2). In particular, if is weakly mixing, which roughly means that the pair correlations

are small for most , then the average (2) is small no matter what are (so long as they are bounded). This can be established by some applications of the Cauchy-Schwarz inequality (or its close cousin, the van der Corput lemma). As a consequence of this, all weakly mixing components of can essentially be discarded when considering an average such as (1).

After getting rid of the weakly mixing components, what is left? Being weakly mixing is like saying that almost all the shifts of are close to orthogonal to each other. At the other extreme is that of periodicity – the shifts periodically recur to become equal to again. There is a slightly more general notion of almost periodicity – roughly, this means that the shifts don’t have to recur exactly to again, but they are forced to range in a precompact set, which basically means that for every , that lies within (in some suitable norm) of some finite-dimensional space. A good example of an almost periodic function is an eigenfunction, in which we have for each and some quantity independent of (e.g. one can take for some ). In this case, the finite-dimensional space is simply the scalar multiples of (and one can even take in this special case).

It is easy to see that non-trivial almost periodic functions are not weakly mixing; more generally, any function which correlates non-trivially with an almost periodic function can also be seen to not be weakly mixing. In the converse direction, it is also fairly easy to show that any function which is not weakly mixing must have non-trivial correlation with an almost periodic function. Because of this, it turns out that one can basically decompose any function into almost periodic and weakly mixing components. For the purposes of getting lower bounds on (1), this allows us to essentially reduce matters to the special case when is almost periodic. But then the shifts are almost ranging in a finite-dimensional set, which allows one to essentially assign each shift a colour from a finite range of colours. If one then applies the van der Waerden theorem, one can find many arithmetic progressions which have the same colour, and this can be used to give a non-trivial lower bound on (1). (Thus we see that the role of a compactness property such as almost periodicity is to reduce density Ramsey theorems to colouring Ramsey theorems.)

This type of argument can be extended to more advanced recurrence theorems, but certain things become more complicated. For instance, suppose one wanted to count progressions of length ; this amounts to lower bounding expressions such as

It turns out that being weakly mixing is no longer enough to give a negligible contribution to expressions such as (3). For that, one needs the stronger property of being weakly mixing relative to almost periodic functions; roughly speaking, this means that for most , the expression is not merely of small mean (which is what weak mixing would mean), but that this expression furthermore does not correlate strongly with any almost periodic function (i.e. is small for any almost periodic ). Once one has this stronger weak mixing property, then one can discard all components of which are weakly mixing relative to almost periodic functions.

One then has to figure out what is left after all these components are discarded. Because we strengthened the notion of weak mixing, we have to weaken the notion of almost periodicity to compensate. The correct notion is no longer that of almost periodicity – in which the shifts almost take values in a finite-dimensional vector space – but that of almost periodicity relative to almost periodic functions, in which the shifts almost take values in a finite-dimensional module over the algebra of almost periodic functions. A good example of such a beast is that of a quadratic eigenfunction, in which we have where is itself an ordinary eigenfunction, and thus almost periodic in the ordinary sense; here, the relative module is the one-dimensional module formed by almost periodic multiples of . (A typical example of a quadratic eigenfunction is for some .)

It turns out that one can “relativise” all of the previous arguments to the almost periodic “factor”, and decompose an arbitrary into a component which is weakly mixing relative to almost periodic functions, and another component which is almost periodic relative to almost periodic functions. The former type of components can be discarded. For the latter, we can once again start colouring the shifts with a finite number of colours, but with the caveat that the colour assigned is no longer independent of , but depends in an almost periodic fashion on . Nevertheless, it is still possible to combine the van der Waerden colouring Ramsey theorem with the theory of recurrence for ordinary almost periodic functions to get a lower bound on (3) in this case. One can then iterate this argument to deal with arithmetic progressions of longer length, but one now needs to consider even more intricate notions of almost periodicity, e.g. almost periodicity relative to (almost periodic functions relative to almost periodic functions), etc.

It turns out that these types of ideas can be adapted (with some effort) to the density Hales-Jewett setting. It’s simplest to begin with the situation rather than the situation. Here, we are trying to obtain non-trivial lower bounds for averages of the form

where ranges in some fashion over combinatorial lines in , and is some non-negative function with large mean.

The analogues of weakly mixing and almost periodic in this setting are the -uniform and -low influence functions respectively. Roughly speaking, a function is -low influence if its value usually doesn’t change much if a is flipped to a or vice versa (e.g. the indicator function of a -insensitive set is -low influence); conversely, a -uniform function is a function such that is small for all (bounded) . One can show that any function can be decomposed, more or less orthogonally, into a -uniform function and a -low influence function, with the upshot being that one can basically reduce the task of lower bounding (4) to the case when is -low influence. But then and are approximately equal to each other, and it is straightforward to get a lower-bound in this case.

Now we turn to the setting, where we are looking at lower-bounding expressions such as

with .

It turns out that (say) being -uniform is no longer enough to give a negligible contribution to the average (5). Instead, one needs the more complicated notion of being -uniform relative to -low influence functions; this means that not only are the averages small for all bounded , but furthermore is small for all bounded and all -low influence (there is a minor technical point here that is a function of a line rather than of a point, but this should be ignored). Any component of in (5) which is -uniform relative to -low influence functions are negligible and so can be removed.

One then needs to figure out what is left in when these components are removed. The answer turns out to be functions that are -almost periodic relative to -low influence. The precise definition of this concept is technical, but very roughly speaking it means that if one flips a digit from a to a , then the value of changes in a manner which is controlled by -low influence functions. Anyway, the upshot is that one can reduce in (5) from to the components of which are -almost periodic relative to -low influence. Similarly, one can reduce in (5) from to the components of which are -almost periodic relative to -low influence.

At this point, one has to use a colouring Ramsey theorem – in this case, the Graham-Rothschild theorem – in conjunction with the relative almost periodicity to locate lots of places in which is close to while is simultaneously close to . This turns (5) into an expression of the form , which turns out to be relatively easy to lower bound (because , being projections of , tend to be large wherever is large).

The discussion is evolving rapidly, but here are some of the topics currently being discussed:

• Understanding the density Hales-Jewett numbers , defined as the size of the largest subset of that does not contain a combinatorial line.  The progress on that problem is summarised here.
• Fujimura’s problem in higher dimensions.
• The hyper-optimistic conjecture in higher dimensions.  It looks like the conjecture in fact fails in higher dimensions, though perhaps it could be salvaged by reformulating it.
• Reducing the need for computer assistance in our result on Moser’s problem.
• New lower bounds for the coloring Hales-Jewett numbers.

Note that much of the most recent progress has not yet been ported to the wiki.  In order to help everyone else catch up, it may useful if authors of comments (particularly comments with lengthy computations, or with corrections to previous comments) put their work on the relevant page of the wiki (not necessarily in the most polished format), and perhaps only place a summary of it on the thread itself.

[Incidentally, for the more casual followers of this project, a non-technical introduction to this project can be found at this post of Jason Dyer.]

Comments here should start from 1200.

• : Any subset of with 451 points contains a combinatorial line; and there is exactly one example with 450 points with no combinatorial line.  (We have two proofs, one by a large integer program, and the other being purely human-verifiable; the latter can be found here.)
• : Any subset of with 125 points contains a geometric line; and we have several examples with 124 points with no geometric line.  (The proof is partly computer-assisted; details are here.)
• : A genetic algorithm has constructed 353-point solutions in with no geometric line; in the other direction, linear and integer programming methods have shown that any set with 362 points must have a geometric line.
• : In the triangular grid , any set of 41 points contains an upwards-pointing equilateral triangle with ; and we have 40-point examples without such triangles.  This was conducted by an integer program.

This timeline shows the history of these and other developments in the project.

There are still several numbers that look feasible to compute.  For instance, the bounds on should be able to be narrowed further.  Work is slowly progressing also on the equal-slices Hales-Jewett numbers , which are hyper-optimistically conjectured to equal the Fujimura numbers ; this has been verified by hand up to n=3 and by integer programming up to n=5.  We are also looking at trying to reduce the dependence on computer assistance in establishing the result; the best human result so far is .

Some progress has recently been made on some other related questions.  For instance, we now have a precise description of the lower bound on coming from the Behrend-Elkin construction, namely

for some absolute constant c.

Also, it is probably a good time to transport some of the discussion in earlier threads to the wiki and make it easier for outsiders to catch up.  (Incidentally, we need a logo for that wiki; any suggestions would be welcome!)

Comments on this thread should be numbered starting at 1100.

Kakeya set conjecture for finite fields. Let F be a finite field, and let E be a subset of that contains a line in every direction.  Then E has cardinality at least for some .

The initial argument of Dvir gave .  This was improved to for some explicit by Saraf and Sudan, and recently to by Dvir, Kopparty, Saraf, and Sudan, which is within a factor 2 of the optimal result.

In our work we investigate a somewhat different set of improvements to Dvir’s result.  The first concerns the Kakeya maximal function of a function , defined for all directions in the projective hyperplane at infinity by the formula

where the supremum ranges over all lines in oriented in the direction .  Our first result is the endpoint estimate for this operator, namely

Kakeya maximal function conjecture in finite fields. We have for some constant .

This result implies Dvir’s result, since if f is the indicator function of the set E in Dvir’s result, then for every .  However, it also gives information on more general sets E which do not necessarily contain a line in every direction, but instead contain a certain fraction of a line in a subset of directions.  The exponents here are best possible in the sense that all other mapping properties of the operator can be deduced (with bounds that are optimal up to constants) by interpolating the above estimate with more trivial estimates.  This result is the finite field analogue of a long-standing (and still open) conjecture for the Kakeya maximal function in Euclidean spaces; we rely on the polynomial method of Dvir, which thus far has not extended to the Euclidean setting (but note the very interesting variant of this method by Guth that has established the endpoint multilinear Kakeya maximal function estimate in this setting, see this blog post for further discussion).

It turns out that a direct application of the polynomial method is not sufficient to recover the full strength of the maximal function estimate; but by combining the polynomial method with the Nikishin-Maurey-Pisier-Stein “method of random rotations” (as interpreted nowadays by Stein and later by Bourgain, and originally inspired by the factorisation theorems of Nikishin, Maurey, and Pisier), one can already recover a “restricted weak type” version of the above estimate.  If one then enhances the polynomial method with the “method of multiplicities” (as introduced by Saraf and Sudan) we can then recover the full “strong type” estimate; a few more details below the fold.

It turns out that one can generalise the above results to more general affine or projective algebraic varieties over finite fields.  In particular, we showed

Kakeya maximal function conjecture in algebraic varieties. Suppose that is an (n-1)-dimensional algebraic variety.  Let be an integer. Then we have

for some constant , where the supremum is over all irreducible algebraic curves of degree at most d that pass through x but do not lie in W, and W(F) denotes the F-points of W.

The ordinary Kakeya maximal function conjecture corresponds to the case when N=n, W is the hyperplane at infinity, and the degree d is equal to 1.  One corollary of this estimate is a Dvir-type result: a subset of which contains, for each x in W, an irreducible algebraic curve of degree d passing through x but not lying in W, has cardinality if .  (In particular this implies a lower bound for Nikodym sets worked out by Li.)  The dependence of the implied constant on W is only via the degree of W.

The techniques used in the flat case can easily handle curves of higher degree (provided that we allow the implied constants to depend on d), but the method of random rotations does not seem to work directly on the algebraic variety W as there are usually no symmetries of this variety to exploit.  Fortunately, we can get around this by using a “random projection trick” to “flatten” W into a hyperplane (after first expressing W as the zero locus of some polynomials, and then composing with the graphing map for such polynomials), reducing the non-flat case to the flat case.

Below the fold, I wish to sketch two of the key ingredients in our arguments, the random rotations method and the random projections trick.  (We of course also use some algebraic geometry, but mostly low-tech stuff, on the level of Bezout’s theorem, though we do need one non-trivial result of Kleiman (from SGA6), that asserts that bounded degree varieties can be cut out by a bounded number of polynomials of bounded degree.)

[Update, March 14: See also Jordan's own blog post on our paper.]

– The random rotations method –

The random rotations method allows one to amplify a result about large sets into an estimate about small sets, by observing that large sets can be created by taking many random rotations (or translations, etc.) of a given small set; it is useful in any situation which enjoys symmetries that one can push small sets around this way (e.g. a homogeneous space will do nicely).  Let’s see how it works in this context.  Dvir’s argument lets us give the following conclusion (stated somewhat informally):

Proposition 1. Let E be a subset of which contains lines in J different directions.  If , then .  (Implied constants here are allowed to depend on n.)

We now amplify this by the random rotations method to extend from the large J case to the small J case:

Proposition 2. Let E be a subset of which contains lines in J different directions.  Then .

Proof. If then we are done by Proposition 1 already, so suppose that J is much smaller than .  (There are only directions available in .)  Let m be an integer such that .

We use the probabilistic method (and more precisely, the first moment method).  Now let be m randomly chosen “rotations”, or more precisely invertible linear transformations, and consider the set .  Clearly, .  On the other hand, we expect to contain lines in about different directions.  Indeed, pick any direction .  Then has a probability about of being one of the J directions of the lines that E contains; this is also the probability that contains a line in the direction .  If we choose each of the independently at random, then basic probability then tells us that for each direction , that E’ will contain a line in the direction with probability about .  From linearity of expectation we thus see that the expected number of directions along which E’ contains a line is , so in particular we will have such lines for at least one choice of E’.  Applying Proposition 1 to that choice we obtain , and so as required.

– The random projection trick –

The random projection trick is a different application of the probabilistic method.  It uses the fact that random maps tend to be “mostly injective” whenever the range is larger than the domain (and, dually, are “mostly surjective” when the domain is larger than the range); this often allows one to cut either the domain or range down in size until they are roughly equal.  It is used in a number of places in the literature, for instance in Ruzsa’s proof of Freiman’s theorem.

For instance, we have the following statement (stated a bit informally to simplify the formulation):

Lemma 1. Let E be a subset of , and let be a random linear transformation.  If , then with high probability.

Proof. Consider the number of “collisions”,

.

On the one hand, each pair has a probability of of contributing a collision if x and y are distinct, and a probability 1 of contributing a collision if they are equal.  Thus

.

From Markov’s inequality we conclude that with high probability.

On the other hand, we can write .  Since , we conclude from the Cauchy-Schwarz inequality that

,

and the claim follows.

Remark. While T is “mostly injective” once (in the sense that ), T will not be completely injective with high probability until , thanks to the birthday paradox.

Here is a typical way in which one can hope to use this trick for Kakeya-type problems.  A simple modification of Dvir’s original argument (as done in this paper of Li) gives

Proposition 3. (Nikodym-type estimate)  Let E be a subset of of size , and let be another set in such that for every point x in E, there exists a line passing through x such that G contains .  Then .

The random rotations trick (or more precisely, the random translations trick) from the preceding section lets us extend this result to smaller values of |E| (shrinking |G| proportionally).  But here I wish to discuss a different extension, using the random projections trick instead:

Proposition 3. (Nikodym-type estimate)  Let .  Let E be a subset of of size , and let be another set in such that for every point x in E, there exists a line passing through x such that G contains .  Then .

Proof. By  Lemma 1, we can find a linear map such that .  Then observe that T(G) obeys the same hypotheses as G, but with E replaced by T(E).  By Proposition 2, , and the claim follows.

–

1.  Upper and lower bounds for for small n.

Let be the largest size of a set in without a combinatorial line.   We now have both human and computer-assisted proofs of the first few values of this sequence:

.

The current best-known bounds for are .  Given the gap involved here, and the rate at which the complexity of the problem has increased with n, it seems unlikely that we will be able to compute exactly any time soon, but it is possible that some improvement can still be made here.

2. A hyper-optimistic conjecture

Consider a variant of the above problem in which each element of with a 1s, b 2s, and c 3s is weighted by the factor ; this gives a total weight of . Let be the largest weight of a line-free set of , and let be the largest size of a subset of

which contains no upward-pointing equilateral triangles with r>0. It is known that ; the “hyper-optimistic conjecture” is that one in fact has . This would imply density Hales-Jewett for k=3.

Currently, the conjecture is verified for , where the values of for are 1,2,4,6,9,12 respectively; see this page and this page for details.  It seems feasible to handle .  Currently we know that and .

3.  Asymptotics for Moser’s cube problem

Moser’s cube problem asks to compute the largest size of a subset of the cube without geometric lines.  The first few values of are known:

.

The best asymptotic lower bound known is still of the order of .  Improving this bound seems related to the well-known problem of improving the bounds in Behrend’s construction of an AP-3 free set of integers.

We are quite close now to pinning down ; we know that it is equal to either 124 or 125, and it is looking increasingly unlikely that it is 125.

Comments on this thread should start at 900.

For simplicity I will restrict attention to the Bernoulli random walk.  Given real numbers , one can form the random variable

where are iid random signs (with either sign +1, -1 chosen with probability 1/2).  This is a discrete random variable which typically takes values.  However, if there are various arithmetic relations between the step sizes , then many of the possible sums collide, and certain values may then arise with much higher probability.  To measure this, define the concentration probability by the formula

.

Intuitively, this probability measures the amount of additive structure present between the .  There are two (opposing) problems in the subject:

• (Forward Littlewood-Offord problem) Given some structural assumptions on , what bounds can one place on ?
• (Inverse Littlewood-Offord problem) Given some bounds on , what structural assumptions can one then conclude about ?

Ideally one would like answers to both of these problems which come close to inverting each other, and this is the guiding motivation for our paper.

One of the first forward Littlewood-Offord theorems was by Erdős, who showed

Theorem 1. If at least k of the are non-zero, then .

In fact the sharp bound was computed by Erdős as $\binom{k}{\lfloor k/2\rfloor}/2^k$; the proof relies, incidentally, on Sperner’s theorem, which is of relevance to the ongoing polymath1 project.  (An earlier result of Littlewood and Offord gave a weaker bound of .)  Taking contrapositives, we obtain an inverse Littlewood-Offord theorem:

Corollary 1. If , then at most of the are non-zero.

The bound is sharp in the sense that it is attained in the case when all the are equal.  However, the theorem is far from sharp in other cases; if k of the are non-zero, then can be as small as .

Another forward Littlewood-Offord theorem in a similar spirit is by Särkőzy and Szemerédi, who showed

Theorem 2. If at least k of the are distinct, then .

(The slightly weaker bound of was obtained by Erdős and Moser.)  Again, it has a contrapositive:

Corollary 2. If , then the take on at most distinct values.

Again, Theorem 2 is optimal in the sense that the bound is attained when lie in an arithmetic progression (e.g. ), but is not always sharp otherwise.  There are many further forward and inverse Littlewood-Offord results; see our paper for a discussion of some of these.

In recent years it has become clearer that the concentration probability is connected to the extent to which the lie in a generalised arithmetic progression (GAP)

;

the quantity d is known as the rank of the GAP.  For instance, an easy computation gives

Theorem 3. (Forward Littlewood-Offord)  If lie in a GAP P of rank d, then

.

Intuitively, a random sum of n vectors in P should lie in a dilate of P, which has size about , which leads to the stated concentration result.

One of our main results is a sort of converse to the above claim, in the regime where the concentration probability is of polynomial size:

Theorem 4. (Inverse Littlewood-Offord)  If , and , then there exists a GAP P of rank d at most 2A that contains all but of the , and such that

.

Thus, Theorem 3 is sharp up to epsilon losses, and up to throwing away a small number of vectors.  One can use this theorem to deduce several earlier theorems, such as Theorem 1 and Theorem 2, except for some epsilon losses.  (It is certainly of interest to try to see how one can remove these epsilon losses; I know this problem is currently being looked at.)

In an earlier paper we had proven a weaker version of Theorem 4, in which the rank d was only bounded by instead of the optimal value of 2A, and the nearly-sharp factor was replaced by .

Our methods are purely combinatorial, based on “growing” the GAP P by a greedy algorithm.  Roughly speaking, the idea is as follows:

1. Start with the trivial progression .  Also, pick an integer k a little bit smaller than .
2. Call an element x of Q “bad” if adding the progression to Q significantly increases the size of Q, and “good” otherwise.
3. If only a few of the are bad, STOP.
4. Otherwise, if there are a lot of bad , there is a way to use Hölder’s inequality to find a bad x with the property that S is much more likely to fall into a translate of than it is to Q.    Replace Q with this larger GAP and return to step 2.

This algorithm turns out to terminate in a bounded number of steps if the concentration probability of S was initially of polynomial size, and will trap most of the in the set of good points relating to Q, which turns out to essentially be a GAP P of size about times bigger than that of Q, where d is the rank of Q.

1.  Upper and lower bounds for for small n.

Let be the largest size of a set in without a combinatorial line.  Thanks to efforts from previous threads, we have the first five values of :

.

We also know that , and are working to narrow this further.  The arguments justifying these bounds can be found here.  The latest bounds for the next few values of can be found here.

2.  A hyper-optimistic conjecture

Consider a variant of the above problem in which each element of with a 1s, b 2s, and c 3s is weighted by the factor ; this gives a total weight of .  Let be the largest weight of a line-free set of , and let be the largest size of a subset of

which contains no upward-pointing equilateral triangles with .  It is known that ; the “hyper-optimistic conjecture” is that one in fact has .  This would imply density Hales-Jewett for k=3.

Currently, we know the following values for these sequences:

and

.

There are also some further upper and lower bounds known; see this spreadsheet for the latest bounds, and this page for proofs of the bounds.  The data so far is consistent with the conjecture, but more work would be needed to obtain a more convincing case for it.

3.  Asymptotics for Moser’s cube problem

Moser’s cube problem asks to compute the largest size of a subset of the cube without geometric lines.  The first few values of are known:

.

The best asymptotic lower bound known is .  Given that we have a significantly superior lower bound of for , one might hope to similarly improve the lower bound for Moser’s problem.  But there seems to be a technical snag, based on the observation that between two slices and with the , there are in fact quite a few combinatorial lines connecting them, and so it is not obvious how to create a geometric line-free set that involves more than one slice per value of c-a.

It is also of interest to work out what asymptotic lower bound for one can get for larger values of k than 3.

Comments on this thread should start at 700.  We will also try to record progress made at the polymath1 wiki and at the polymath1 spreadsheet, as appropriate.

]]> http://terrytao.wordpress.com/2009/02/05/upper-and-lower-bounds-for-the-density-hales-jewett-problem/ Fri, 06 Feb 2009 01:05:33 +0000 Terence Tao http://terrytao.wordpress.com/2009/02/05/upper-and-lower-bounds-for-the-density-hales-jewett-problem/ This is a continuation of several threads from Tim Gowers’ massively collaborative mathematics project, “A combinatorial approach to density Hales-Jewett“, which I discussed in my previous post.

For any positive integer n, let be the set of strings of length n consisting of 1s, 2s, and 3s, and define a combinatorial line to be a triple of such strings arising by taking a string in with at least one wildcard x, and substituting x=1, x=2, x=3 in that string (e.g. xx1x3 would give the combinatorial line ).  Call a set of strings line-free if it contains no combinatorial lines, and let be the size of the largest set A which is line-free.  We then have

Density Hales-Jewett theorem (k=3): , i.e. .

This theorem implies several other important results, most notably Roth’s theorem on length three progressions (in an arbitrary abelian group!) and also the corners theorem of Ajtai and Szemeredi (again in an arbitrary abelian group).

Here are some of the questions we are initially exploring in this thread (and which I will comment further on below the fold):

1. What are the best upper and lower bounds for for small n?  Currently we know , with some partial results for higher n.    We also have some relationships between different .
2. What do extremal or near-extremal sets (i.e. line-free sets with cardinality close to ) look like?
3. What are some good constructions of line-free sets that lead to asymptotic lower bounds on ?  The best asymptotic lower bound we have currently is .
4. Can these methods extend to other related problems, such as the Moser cube problem or the capset problem?
5. How feasible is it to extend the existing combinatorial proofs of Roth’s theorem (or the corners theorem), in particular the arguments of Szemeredi, to the Density Hales-Jewett theorem?

But I imagine that further threads might develop in the course of the discussion.

As with the rest of the project, this is supposed to be an open collaboration: please feel free to pose a question or a comment, even if (or especially if) it is just barely non-trivial.  (For more on the “rules of the game”, see this post.)

– I. for small n –

Because the Cartesian product of two line-free sets is line-free, we have a lower bound

(1)

for all , although this seems to be an inferior bound in practice.

There is a trivial bound

(2)

which comes from the observation that any line-free subset of can be split into three slices, each of which is essentially a line-free subset of .  In particular, since , we see that

(3)

for .

If we identify with and consider the set

, (4)

thus .  Observe that a combinatorial line with a,b,c,k 1s, 2s, 3s, and wildcards respectively lies in if and only if k is a multiple of 3, and a is not equal to b mod 3.  In particular, is line-free for , thus matching the upper bound (3) and concluding that

(5)

For n=4, the only combinatorial lines remaining in are and their permutations.  Thus (as observed in Neylon.83) the set is line-free, leading to the lower bound .

This bound was complemented with the lower bound in Jakobsen.90.  Thus .

For , the best lower bound currently is (Neylon.83) and .

No attempt at getting reasonable bounds for , etc. appears in the previous threads.

Question I.A: Can we improve the upper and lower bounds on ?

Question I.B: Is there some reasonably efficient way to automate this process?

Question I.C: What do the extremal line-free sets (sets of size close to ) look like?

At some point we should submit this sequence to the OEIS.

– II. Lower bounds on for large n –

For any integers a,b,c adding up to n, let be the set of all strings with a 1s, b 2s, and c 3s.  Observe that if is any set of triples (a,b,c) that avoids equilateral triangles for , then the set is line-free.  In particular, if we choose B to be the set of all triples such that a+2b lies in a set free of length three arithmetic progressions (e.g. a Behrend set), then is line-free.  This seems to give a lower bound

. (6)

Question II.A: Using the Elkin bound on Behrend sets, what is the precise lower bound one gets? (Solymosi.46.5 notes that we should restrict the Behrend set to scale rather than n).

Question II.B: What does one get for higher k than 3?  Presumably one would use the O’Bryant bound (O’Bryant.126).

Question II.C: Do the examples from Section I extend to give competitive lower bounds for large n?

Question II.D: Are there better ways to avoid triangles or corners in two dimensions than simply lifting up the Behrend example from one dimension?

– III. Miscellanous facts about –

Using the colouring Hales-Jewett theorem, one can show that for every m, one has for sufficiently large m, improving (2) very slightly; see (Tao.85).

Question III.A: Anything else we can say about the ?  For instance, is there a relation to Hales-Jewett numbers HJ(3,r) (defined as the first n such that every r-colouring of contains a monochromatic combinatorial line?)

Note: as pointed out in Gasarch.45.5, Hindman and Tressler have recently established that , , and .

– IV. Related problems –

Let denote the largest subset of which does not contain any geometric line (which is the same as a combinatorial line, but has a second wildcard y which goes from 3 to 1 whilst x goes from 1 to 3, e.g. xx2yy gives the geometric line ).  The Moser cube problem is to understand the behaviour of .  The first few values are (see OEIS A003142):

. (7)

It is clear that .

Elsholtz.43 proposed modifying the construction of (6) to give a similar lower bound on , but a difficulty was found in Solymosi.48.  Currently, the best known asymptotic bounds for are

(8)

Question IV.A.  Can we improve the lower bound in (8)?

Question IV.B. Can we get good bounds on ?

Define to be the largest cardinality of a subset of which contains no algebraic lines, defined as a triple where ; such sets are known as capsets.  Clearly .  As noted my previous blog post, the best asymptotic bounds are

(9)

Question IV.C. Can we improve on these bounds in any way?

Question IV.D. What are the first few values of ?  Brute force calculation reveals .  (Presumably this sequence is also in the OEIS.)

[Update, Feb 7: I will now maintain a running table of the current "world records" for for small n, formed by combining all the bounds already mentioned in this post and comments.]

[Update, Feb 8: A more complete (and up-to-date) spreadsheet can now be found here.]

n
0	1	1	1
1	2	2	2
2	6	6	4
3	18	16	9
4	52	43	20
5	[150,154]	[96,129]	45
6	[450,462]	[258,387]	112
7	[1302,1386]	[688,1161]	[236,292]
8	[3780,4158]	[1849,3483]	[472,773]
9	[11340,12474]	[4128,10449]	[1008,2075]
10	[32864,37422]	[11094,31347]	[2240,5632]
11	[95964,112266]	[29584,94041]	[5040,15425]
12	[287892,336798]	[79507,282123]	[12544,42569]
13	[837850,1010394]	[175504,846369]	[26432,118237]
14	[2458950,3031182]	[477042,2539107]	[52864,330222]
15	[7376850,9093946]	[1272112,7617321]	[112896,926687]

• green – obtained via the inequality (1)
• orange – obtained via the inequality (2)
• lavender – uses the inequality

– V. Szemeredi’s proof of Roth’s theorem –

Szemeredi provided a short proof of Roth’s theorem by purely combinatorial means, and also famously proved the full case of Szemeredi’s theorem by much more intricate combinatorial argument.  Both arguments are based on the density increment method.  In the setting of , the density increment method revolves around the critical density , which exists by (2); the density Hales-Jewett theorem is precisely the statement that vanishes, so suppose for contradiction that is positive.

Then one can find a line-free set of density such that A has density at most for every m-dimensional subspace of (and one can complement this lower bound with an upper bound for “most” subspaces, see Tao.100).  One can also regularise the density on other sets than subspaces, see Tao.121).  So, basically, the moment one gets a significant density increment on a non-trivial subspace, one is done (see Tao.120, O’Donnell.122, Tao.127).

Szemeredi’s proof of Roth’s theorem proceeds by first showing that a dense set A contains a large cube Q, and thus (if it is free of algebraic lines) must completely avoid the set .  This set contains (most of) lots of parallel subspaces and so squeezes A to have higher density on some other subspace, closing the argument.  (See Tao.86, Tao.118, Tao.131)

Question V.A: Is there some way to adapt this argument to density Hales-Jewett, or at least to the corners problem?  The sparsity of lines is a serious difficulty, see Tao.118, Solymosi.135.

One can at least generate lots of cubes inside A without difficulty (Tao.102, Solymosi.135), lots of combinatorial lines with two elements inside A (Tao.130, Bukh.132, O’Donnell.133, Solymosi.135), and many unorganised subspaces in the complement of A (Tao.118, Solymosi.135), but this does not seem to be enough as yet.

Question V.B: What about using the techniques from Szemeredi’s big paper on Szemeredi’s theorem?  (The immediate issue is that there is a key point where one needs to regularise a graph, and the relevant graph in HJ or even for corners is far too sparse; I’ll try to clarify this at some point.)

– A note on comments –

As in the first thread, please number your comments (starting with 200, then 201, etc.) and provide a title, to assist with following the comments.  (Presumably, in future projects of this type, we will use a platform that allows for comment threading; see this post for further discussion.)

I can describe the problem here.  Let n be a large integer, and let be the set of all strings of length n using the alphabet , thus for instance .  A combinatorial line in is a triple of points in that can be formed by taking a string of length n using the alphabet with at least one occurrence of the “wildcard” x, and then substituting the values of 1, 2, 3 for the wildcard.  For instance, the string would lead to the combinatorial line in .  The (k=3) case of the density Hales-Jewett theorem of Furstenberg and Katznelson asserts:

Density Hales-Jewett theorem. Let .  Then if n is sufficiently large depending on , every subset of of density at least contains a combinatorial line.

[Furstenberg and Katznelson handled the case of general k in a subsequent paper.  The k=1 case is trivial, and as pointed out in this post by Gil Kalai, the k=2 case follows from Sperner's theorem.]

Furstenberg and Katznelson’s proof uses ergodic theory, and in particular does not obviously give any bound as to how large n has to be depending on before the theorem takes effect.  No other proofs of this theorem are currently known.  So it would be desirable to have a combinatorial proof of the k=3 density Hales-Jewett theorem.  Since this theorem implies Roth’s theorem, and Roth’s theorem has a combinatorial proof based on the triangle removal lemma (see e.g. my Simons lecture on the subject, or Tim Gowers’ background article for the project), it is thus natural to ask whether the density Hales-Jewett theorem has a proof based on something similar to the triangle removal lemma.  This is basically the question being explored in the above project. (Some further motivation for this problem can be found here.)

Further articles on this project are collected at this page.

Motivation: In the more quantitative areas of mathematics, such as analysis and combinatorics, one has to frequently keep track of a large number of exponents in one’s identities, inequalities, and estimates.  For instance, if one is studying a set of N elements, then many expressions that one is faced with will often involve some power of N; if one is instead studying a function f on a measure space X, then perhaps it is an norm which will appear instead.  The exponent involved will typically evolve slowly over the course of the argument, as various algebraic or analytic manipulations are applied.  In some cases, the exact value of this exponent is immaterial, but at other times it is crucial to have the correct value of at hand.   One can (and should) of course carefully go through one’s arguments line by line to work out the exponents correctly, but it is all too easy to make a sign error or other mis-step at one of the lines, causing all the exponents on subsequent lines to be incorrect.  However, one can guard against this (and avoid some tedious line-by-line exponent checking) by continually calibrating these exponents at key junctures of the arguments by using basic examples of the object of study (sets, functions, graphs, etc.) as test cases.  This is a simple trick, but it lets one avoid many unforced errors with exponents, and also lets one compute more rapidly.

Quick description: When trying to quickly work out what an exponent p in an estimate, identity, or inequality should be without deriving that statement line-by-line, test that statement with a simple example which has non-trivial behaviour with respect to that exponent p, but trivial behaviour with respect to as many other components of that statement as one is able to manage.   The “non-trivial” behaviour should be parametrised by some very large or very small parameter.  By matching the dependence on this parameter on both sides of the estimate, identity, or inequality, one should recover p (or at least a good prediction as to what p should be).

General discussion: The test examples should be as basic as possible; ideally they should have trivial behaviour in all aspects except for one feature that relates to the exponent p that one is trying to calibrate, thus being only “barely” non-trivial.   When the object of study is a function, then (appropriately rescaled, or otherwise modified) bump functions are very typical test objects, as are Dirac masses, constant functions, Gaussians, or other functions that are simple and easy to compute with.  In additive combinatorics, when the object of study is a subset of a group, then subgroups, arithmetic progressions, or random sets are typical test objects.  In graph theory, typical examples of test objects include complete graphs, complete bipartite graphs, and random graphs. And so forth.

This trick is closely related to that of using dimensional analysis to recover exponents; indeed, one can view dimensional analysis as the special case of exponent calibration when using test objects which are non-trivial in one dimensional aspect (e.g. they exist at a single very large or very small length scale) but are otherwise of a trivial or “featureless” nature.   But the calibration trick is more general, as it can involve parameters (such as probabilities, angles, or eccentricities) which are not commonly associated with the physical concept of a dimension.  And personally, I find example-based calibration to be a much more satisfying (and convincing) explanation of an exponent than a calibration arising from formal dimensional analysis.

When one is trying to calibrate an inequality or estimate, one should try to pick a basic example which one expects to saturate that inequality or estimate, i.e. an example for which the inequality is close to being an equality.  Otherwise, one would only expect to obtain some partial information on the desired exponent p (e.g. a lower bound or an upper bound only).  Knowing the examples that saturate an estimate that one is trying to prove is also useful for several other reasons – for instance, it strongly suggests that any technique which is not efficient when applied to the saturating example, is unlikely to be strong enough to prove the estimate in general, thus eliminating fruitless approaches to a problem and (hopefully) refocusing one’s attention on those strategies which actually have a chance of working.

Calibration is best used for the type of quick-and-dirty calculations one uses when trying to rapidly map out an argument that one has roughly worked out already, but without precise details; in particular, I find it particularly useful when writing up a rapid prototype.  When the time comes to write out the paper in full detail, then of course one should instead carefully work things out line by line, but if all goes well, the exponents obtained in that process should match up with the preliminary guesses for those exponents obtained by calibration, which adds confidence that there are no exponent errors have been committed.

Prerequisites: Undergraduate analysis and combinatorics.

Example 1. (Elementary identities)  There is a familiar identity for the sum of the first n squares:

(1)

But imagine that one has forgotten exactly what the RHS of (1) was supposed to be… one remembers that it was some polynomial in n, but can’t remember what the degree or coefficients of the polynomial were.  Now one can of course try to rederive the identity, but there are faster (albeit looser) ways to reconstruct the right-hand side.  Firstly, we can look at the asymptotic test case .  On the LHS, we are summing n terms of size at most , so the LHS is at most ; thus, if we believe the RHS to be a polynomial in n, it should be at most cubic in n.  We can do a bit better by approximating the sum in the LHS by the integral , which strongly suggests that the cubic term on the RHS should be .  So now we have

for some coefficients a,b,c that we still have to work out.

We can plug in some other basic cases.  A simple one is n=0.  The LHS is now zero, and so the constant coefficient c on the RHS should also vanish.  A slightly less obvious base case is n=-1.  The LHS is still zero (note that the LHS for n-1 should be the LHS for n, minus ), and so the RHS still vanishes here; thus by the factor theorem, the RHS should have both n and n+1 as factors.  We are now looking at

for some unspecified constant d.  But now we just need to try one more test case, e.g. n=1, and we learn that , thus recovering the correct formula

. (1′)

Once one has the formula (1′) in hand, of course, it is not difficult to verify by a textbook use of mathematical induction that the formula is in fact valid.  (Alternatively, one can prove a more abstract theorem that the sum of the first n powers is necessarily a polynomial in n for any given k, at which point the above analysis actually becomes a rigorous derivation of (1′).)

Note that the optimal strategy here is to start with the most basic test cases () first before moving on to less trivial cases.  If instead one used, e.g. n=1, n=2, n=3, n=4 as the test cases, one would eventually have obtained the right answer, but it would have been more work.

Exercise 1. (Partial fractions)  If are distinct complex numbers, and P(z) is a polynomial of degree less than k, establish the existence of a partial fraction decomposition

(Hint: use the remainder theorem and induction) and use the test cases for to compute the coefficients .  Use this to deduce the Lagrange interpolation formula.

Example 2. (Counting cycles in a graph)  Suppose one has a graph G on N vertices with an edge density of (thus, the number of edges is , or roughly up to constants).   There is a standard Cauchy-Schwarz argument that gives a lower bound on the number of four-cycles (i.e. a circuit of four vertices connected by four edges) present in G, as a function of and N.  It only takes a few minutes to reconstruct this argument to obtain the precise bound, but suppose one was in a hurry and wanted to guess the bound rapidly.  Given the “polynomial” nature of the Cauchy-Schwarz inequality, the bound is likely to be some polynomial combination of and N, such as (omitting constants and lower order terms).   But what should p and q be?

Well, one can test things with some basic examples.  A really trivial example is the empty graph (where ), but this is too trivial to tell us anything much (other than that p should probably be positive).   At the other extreme, consider the complete graph on N vertices, where ; this renders p irrelevant, but still makes q non-trivial (and thus, hopefully, computable).  In the complete graph, every set of four points yields a four-cycle , so the number of four-cycles here should be about (give or take some constant factors, such as 4! – remember that we are in a hurry here, and are ignoring these sorts of constant factors).  This tells us that q should be at most 4, and if we expect the Cauchy-Schwarz bound to be saturated for the complete graph (which is a good bet – arguments based on the Cauchy-Schwarz inequality tend to work well in very “uniformly distributed” situations) – then we would expect q to be exactly 4.

To calibrate p, we need to test with graphs of density less than 1.  Given the previous intuition that Cauchy-Schwarz arguments work well in uniformly distributed situations, we would want to use a test graph of density that is more or less uniformly distributed.  A good example of such a graph is a random graph G on N vertices, in which each edge has an independent probability of of lying in G.  By the law of large numbers, we expect the edge density of such a random graph to be close to on the average.  On the other hand, each one of the roughly four-cycles connecting the N vertices has a probability about of lying in the graph, since the has four edges, each with an independent probability of of lying in the edge.   The events that each of the four-cycles lies in the graph G aren’t completely independent of each other, but they are still close enough to being so that one can guess using the law of large numbers that the total number of 4-cycles should be about on the average (up to constants).  [Actually, linearity of expectation will give us this claim even without any independence whatsoever.]  So this leads one to predict p=4, thus the number of 4-cycles in any graph on N vertices of density should be for some absolute constant , and this is indeed the case (provided that one also counts degenerate cycles, in which some vertices are repeated).

If one is nervous about using the random graph as the test graph, one could try a graph at the other end of the spectrum – e.g. the complete graph on about vertices, which also has edge density about .  Here one quickly calculates that the number of 4-cycles is about , which is a larger quantity than in the random case (and this fits with the intuition that this graph is packing the same number of edges into a tighter space, and should thus increase the number of cycles).  So the random graph is still the best candidate for a near-extremiser for this bound.  (Actually, if the number of 4-cycles is close to the Cauchy-Schwarz lower bound, then the graph becomes pseudorandom, which roughly speaking means any randomly selected small subgraph of that graph is indistinguishable from a random graph.)

One should caution that sometimes the random object is not the extremiser, and so does not always calibrate an estimate correctly.  For instance, consider Szemerédi’s theorem, that asserts that given any and , that any subset of of density should contain at least one arithmetic progression of length k, if N is large enough.  One can then ask what is the minimum number of k-term arithmetic progressions such a set would contain.  Using the random subset of of density as a test case, we would guess that there should be about (up to constants depending on k).    However, it turns out that the number of progressions can be significantly less than this (basically thanks to the old counterexample of Behrend): given any constant C, one can get significantly fewer than k-term progressions.  But, thanks to an averaging argument of Varnavides, it is known that there are at least k-term progressions (for N large enough), where is a positive quantity.  (Determining the exact order of magnitude of is still an important open problem in this subject.)  So one can at least calibrate the correct dependence on N, even if the dependence on is still unknown.

Example 3. (Sobolev embedding)  Given a reasonable function (e.g. a Schwartz class function will do), the Sobolev embedding theorem gives estimates such as

(2)

for various exponents p, q.  Suppose one has forgotten the exact relationship between p, q, and n and wants to quickly reconstruct it, without rederiving the proof of the theorem or looking it up.  One could use dimensional analysis to work out the relationship (and we will come to that shortly), but an equivalent way to achieve the same result is to test the inequality (2) against a suitably basic example, preferably one that one expects to saturate (2).

To come as close to saturating (2) as possible, one wants to keep the gradient of f small, while making f large; among other things, this suggests that unnecessary oscillations in f should be kept to a minimum.  A natural candidate for an extremiser, then, would be a rescaled bump function , where is some fixed bump function, is an amplitude parameter, and is a parameter, thus f is a rescaled bump function of bounded amplitude O(A) that is supported on a ball of radius O(r) centred at the origin.  [As the estimate (2) is linear, the amplitude A turns out to ultimately be irrelevant here, but the amplitude plays a more crucial role in nonlinear estimates; for instance, it explains why nonlienar estimates typically have the same number of appearances of a given unknown function f in each term.  Also, it is sometimes convenient to carefully choose the amplitude in order to attain a convenient normalisation, e.g. to set one of the norms in (2) equal to 1.]

The ball that f is supported on has volume about (allowing implied constants to depend on n), and so the norm of f should be about (allowing implied constants to depend on q as well).  As for the gradient of f, since f oscillates by O(A) over a length scale of O(L), one expects to have size about on this ball (remember, derivatives measure “rise over run“!), and so the norm of should be about .  Inserting this numerology into (2), and equating powers of L (note A cancels itself into irrelevance, and could in any case be set to equal 1), we are led to the relation

(2)

which is indeed one of the necessary conditions for (2).  (The other necessary conditions are that p and q lie strictly between 1 and infinity, but these require a more complicated test example to establish.)

One can efficiently perform the above argument using the language of dimensional analysis.  Giving f the units of amplitude A, and giving space the units of length L, we see that the n-dimensional integral has units of , and thus norms have units of .  Meanwhile, from the rise-over-run interpretation of the derivative, has units of , thus the LHS and RHS of (2) have units of and respectively.  Equating these dimensions gives (3).  Observe how this argument is basically a shorthand form of the argument based on using the rescaled bump function as a test case; with enough practice one can use this shorthand to calibrate exponents rapidly for a wide variety of estimates.

Exercise 2. Convert the above discussion into a rigorous proof that (3) is a necessary condition for (2).  (Hint: exploit the freedom to send L to zero or to infinity.)  What happens to the necessary conditions if is replaced with a bounded domain (such as the unit cube , assuming Dirichlet boundary conditions) or a discrete domain (such as the lattice , replacing the gradient with a discrete gradient of course)?

Exercise 3. If one replaces (2) by the variant estimate

(2′)

establish the necessary condition

. (3′)

What happens to the dimensional analysis argument in this case?

Remark 1. There are many other estimates in harmonic analysis which are saturated by some modification of a bump function; in addition to the isotropically rescaled bump functions used above, one could also rescale bump functions by some non-isotropic linear transformation (thus creating various “squashed” or “stretched” bumps adapted to disks, tubes, rectangles, or other sets), or modulate bumps by various frequencies, or translate them around in space.  One can also try to superimpose several such transformed bump functions together to amplify the counterexample.  The art of selecting good counterexamples can be somewhat difficult, although with enough trial and error one gets a sense of what kind of arrangement of bump functions are needed to make the right-hand side small and the left-hand side large in the estimate under study.

Example 3 (Scale-invariance in nonlinear PDE)  The model equations and systems studied in nonlinear PDE often enjoy various symmetries, notably scale-invariance symmetry, that can then be used to calibrate various identities and estimates regarding solutions to those equations.  For sake of discussion, let us work with the nonlinear Schrödinger equation (NLS)

(4)

where is the unknown field, is the spatial Laplacian, and is a fixed exponent.  (One can also place in some other constants in (4), such as Planck’s constant , but we have normalised this constant to equal 1 here, although it is sometimes useful to reinstate this constant for calibration purposes.) If u is one solution to (4), then we can form a rescaled family of such solutions by the formula

(5)

for some specific exponents a, b; these play the role of the rescaled bump functions in Example 2.  The exponents a,b can be worked out by testing (4) using (5), and we leave this as an exercise to the reader, but let us instead use the shorthand of dimensional analysis to work these exponents out.  Let’s give u the units of amplitude A, space the units of length L, and time the units of duration T.  Then the three terms in (4) have units , , and respectively; equating these dimensions gives and .  (In particular, time has “twice the dimension” of space; this is a feature of many non-relativistic equations such as Schrödinger, heat, or viscosity equations.  For relativistic equations, of course, time and space have the same dimension with respect to scaling.)  On the other hand, the scaling (5) multiplies A, T, and L by , , and respectively; to maintain consistency with the relations and we must thus have and .

Exercise 4. Solutions to (4) (with suitable smoothness and decay properties) enjoy a conserved Hamiltonian , of the form

for some constants .  Use dimensional analysis (or the rescaled solutions (5) as test cases) to compute q.  (The constant , unfortunately, cannot be recovered from dimensional analysis, and other model test cases, such as solitons or other solutions obtained via separation of variables, also turn out unfortunately to not be sensitive enough to to calibrate this parameter.)

Remark 2. The scaling symmetry (5) is not the only symmetry that can be deployed to calibrate identities and estimates for solutions to NLS.  For instance, we have a simple phase rotation symmetry for such solutions, where is an arbitrary phase.  This symmetry suggests that in any identity involving u and its complex conjugate , the net number of factors of u, minus the factors of , in each term of the identity should be the same.  (Factors without phase, such as |u|, should be ignored for this analysis.)  Other important symmetries of NLS, which can also be used for calibration, include space translation symmetry, time translation symmetry, and Galilean invariance.  (While these symmetries can of course be joined together, to create a large-dimensional family of transformed solutions arising from a single base solution u, for the purposes of calibration it is usually best to just use each of the generating symmetries separately.)  For gauge field equations, gauge invariance is of course a crucial symmetry, though one can make the calibration procedure with respect to this symmetry automatic by working exclusively with gauge-invariant notation (see also my earlier post on gauge theory).  Another important test case for Schrödinger equations is the high-frequency limit , closely related to the semi-classical limit , that allows one to use classical mechanics to calibrate various identities and estimates in quantum mechanics. 

Exercise 5. Solutions to (4) (again assuming suitable smoothness and decay) also enjoy a virial identity of the form

where the right-hand side only involves u and its spatial derivatives , and does not explicitly involve the spatial variable x.  Using the various symmetries, predict the type of terms that should go on the right-hand side.  (Again, the coefficients of these terms are unable to be calibrated using these methods, but the exponents should be accessible.)

Remark 3. Einstein used this sort of calibration technique (using the symmetry of spacetime diffeomorphisms, better known as the general principle of relativity, as well as the non-relativistic limit of Newtonian gravity as another test case) to derive the Einstein equations of gravity, although the one constant that he was unable to calibrate in this fashion was the cosmological constant.

Example 4 (Fourier-analytic identities in additive combinatorics).   Fourier analysis is a useful tool in additive combinatorics for counting various configurations in sets, such as arithmetic progressions of length three.  (It turns out that classical Fourier analysis is not able to handle progressions of any longer length, but that is a story for another time – see e.g. this paper of Gowers for some discussion.)  A typical situation arises when working in a finite group such as , and one has to compute an expression such as

(6)

for some functions (for instance, these functions could all be the indicator function of a single set ).  The quantity (6) can be expressed neatly in terms of the Fourier transforms , which we normalise as .  It is not too difficult to compute this expression by means of the Fourier inversion formula and some routine calculation, but suppose one was in a hurry and only had a vague recollection of what the Fourier-analytic expression of (6) was – something like

(7)

for some coefficients p, a, b, c, but the precise values of which have been forgotten.  (In view of some other Fourier-analytic formulae, one might think that some of the Fourier transforms might need to be complex conjugated for (7), but this should not happen here, because (6) is linear in f,g,h rather than anti-linear; cf. the discussion in Example 3 about factors of u and .)  How can one quickly calibrate the values of p,a,b,c without doing the full calculation?

To isolate the exponent p, we can consider the basic case , in which case the Fourier transforms are just the Kronecker delta function (e.g. equals 1 for and vanishes otherwise).  The expression (6) is just , while the expression (7) is (because only one of the summands is non-trivial); thus p must equal 2.  (Exercise: reinterpret the above analysis as a dimensional analysis.)

Next, to calibrate a,b,c, we modify the above basic test case slightly, modulating the f,g,h so that a different element of the sum in (7) is non-zero.  Let us take , , for some fixed frequency ; then (4) is again equal to , while (6) is equal to

In order for this to equal for any , we need the linear form to vanish identically, which forces a=c and b=-2a.  We can normalise a=1 (by using the change of variables ), thus leading us to the correct expression for (7), namely

.

Once one actually has this formula, of course, it is a routine matter to check that it actually is the right answer.

Remark 4. One can also calibrate a,b,c in (7) by observing the identity (which reflects the fact that the second derivative of a linear function is necessarily zero), which gives a modulation symmetry , , to (6).  Inserting this symmetry into (7) reveals that and as before.

Remark 5. By choosing appropriately normalised conventions, one can avoid some calibration duties altogether.  For instance, when using Fourier analysis on a finite group such as , if one expects to be analysing functions that are close to constant (or subsets of the group of positive density), then it is natural to endow physical space with normalised counting measure (and thus, by Pontryagin duality, frequency space should be given non-normalised counting measure).  [Conversely, if one is analysing functions concentrated on only a bounded number of points, then it may be more convenient to give physical space counting measure and frequency space normalised counting measure.]  In practical terms, this means that any physical space sum, such as , should instead be replaced with a physical space average , while keeping sums over frequency space variables unchanged; when one does so, all powers of N “miraculously” disappear, and there is no longer any need to calibrate using the constant function 1 as was done above. Of course, this does not eliminate the need to perform other calibrations, such as that of the coefficients a,b,c above.

Kakeya set conjecture: Let , and let contain a unit line segment in every direction (such sets are known as Kakeya sets or Besicovitch sets).  Then E has Hausdorff dimension and Minkowski dimension equal to n.

One reason why I find these conjectures fascinating is the sheer variety of mathematical fields that arise both in the partial results towards this conjecture, and in the applications of those results to other problems.  See for instance this survey of Wolff, my Notices article and this article of Łaba on the connections between this problem and other problems in Fourier analysis, PDE, and additive combinatorics; there have even been some connections to number theory and to cryptography.  At the other end of the pipeline, the mathematical tools that have gone into the proofs of various partial results have included:

• Maximal functions, covering lemmas, methods (Cordoba, Strömberg, Cordoba-Fefferman);
• Fourier analysis (Nagel-Stein-Wainger);
• Multilinear integration (Drury, Christ)
• Paraproducts (Katz);
• Combinatorial incidence geometry (Bourgain, Wolff);
• Multi-scale analysis (Barrionuevo, Katz-Łaba-Tao, Łaba-Tao, Alfonseca-Soria-Vargas);
• Probabilistic constructions (Bateman-Katz, Bateman);
• Additive combinatorics and graph theory (Bourgain, Katz-Łaba-Tao, Katz-Tao, Katz-Tao);
• Sum-product theorems (Bourgain-Katz-Tao);
• Bilinear estimates (Tao-Vargas-Vega);
• Perron trees (Perron, Schoenberg, Keich);
• Group theory (Katz);
• Low-degree algebraic geometry (Schlag, Tao, Mockenhaupt-Tao);
• High-degree algebraic geometry (Dvir, Saraf-Sudan);
• Heat flow monotonicity formulae (Bennett-Carbery-Tao)

[This list is not exhaustive.]

Very recently, I was pleasantly surprised to see yet another mathematical tool used to obtain new progress on the Kakeya conjecture, namely (a generalisation of) the famous Ham Sandwich theorem from algebraic topology.  This was recently used by Guth to establish a certain endpoint multilinear Kakeya estimate left open by the work of Bennett, Carbery, and myself.  With regards to the Kakeya set conjecture, Guth’s arguments assert, roughly speaking, that the only Kakeya sets that can fail to have full dimension are those which obey a certain “planiness” property, which informally means that the line segments that pass through a typical point in the set must be essentially coplanar. (This property first surfaced in my paper with Katz and Łaba.)  Guth’s arguments can be viewed as a partial analogue of Dvir’s arguments in the finite field setting (which I discussed in this blog post) to the Euclidean setting; in particular, both arguments rely crucially on the ability to create a polynomial of controlled degree that vanishes at or near a large number of points.  Unfortunately, while these arguments fully settle the Kakeya conjecture in the finite field setting, it appears that some new ideas are still needed to finish off the problem in the Euclidean setting.  Nevertheless this is an interesting new development in the long history of this conjecture, in particular demonstrating that the polynomial method can be successfully applied to continuous Euclidean problems (i.e. it is not confined to the finite field setting).

In this post I would like to sketch some of the key ideas in Guth’s paper, in particular the role of the Ham Sandwich theorem (or more precisely, a polynomial generalisation of this theorem first observed by Gromov).

– The polynomial Ham Sandwich theorem –

Let us first recall the classical Ham Sandwich theorem:

Ham Sandwich theorem. Let be n bounded open sets in .  Then there exists a hyperplane in that divides each of the open sets into two sets of equal volume.

(The name of the theorem derives from the special case when and are two slices of bread and a slice of ham.  One can view this theorem as a “thickened” version of the Euclidean geometry axiom that every n points in determine at least one hyperplane.)

There are many proofs of this theorem, but I will focus on the proof that is based on the Borsuk-Ulam theorem:

Borsuk-Ulam theorem: Let be a continuous map from the n-dimensional sphere to the Euclidean space which is antipodal (which means that for all .  Then for at least one .

Proof. (Sketch)  The set of zeroes of an antipodal map automatically come in antipodal pairs x,-x.   To prove the theorem, we shall establish the stronger fact that for an odd number of disjoint antipodal pairs, counting multiplicity (avoiding the degenerate antipodal maps which vanish at an infinite set of points).  To see this, first observe that this is true for at least one antipodal map (e.g. one can use the horizontal projection map ).  Also, the space of all antipodal maps is a vector space, and thus connected (though it takes some effort to show that the space of non-degenerate antipodal maps is still connected).  So one just needs to show that the parity of the number of pairs of antipodal points where f vanishes (counting multiplicity) is unchanged with respect to continuous deformations of f.  But some elementary degree theory (or Morse theory) shows that any (non-degenerate) perturbation of f can annihilate two such antipodal pairs by collision, or (by the reverse procedure) spontaneously create two such antipodal pairs from nothing, but cannot otherwise affect the number of pairs; thus the parity of the number of such pairs remains invariant.  (It takes some non-trivial effort to make this informal argument rigorous; see for instance this chapter of Matousek’s book on the Borsuk-Ulam theorem, which also contains a number of other proofs of this result.  [Thanks to Benny Sudakov for this great reference.] One can also formalise this argument using the language of singular cohomology.)

Remark 1. The Borsuk-Ulam theorem is tied to the more general theory of Lyusternik-Schnirelmann category, which is the viewpoint taken in Guth’s paper, but we will not explicitly use this theory here.

Proof of the Ham-Sandwich theorem using the Borsuk-Ulam theorem. We can identify with the space of affine-linear forms on .  Each non-trivial affine-linear form determines a hyperplane that divides into two half-spaces and .  We can then define to be the function whose coordinate at P is the volume of minus the volume of ; thus f measures the extent to which the hyperplane fails to bisect all of the .  It is easy to see that f is continuous, homogeneous of degree zero, and odd, and so its restriction to is an antipodal map.  By the Borsuk-Ulam theorem, there exists P such that , and the claim follows.

Gromov observed the following polynomial generalisation of the Ham Sandwich theorem:

Polynomial Ham Sandwich theorem. Let , and let be bounded open sets in .  Then there exists a non-trivial polynomial of degree at most d such that the sets , partition each of the into two sets of equal measure.

Note that the ordinary Ham-Sandwich theorem corresponds to the d=1 case of this theorem.  This theorem can be deduced from the Borsuk-Ulam theorem in exactly the same way that the ordinary one is (note that the space of polynomials of degree at most d has dimension ; the continuity of the appropriate antipodal function follows from the dominated convergence theorem and the basic observation that a non-trivial polynomial is non-zero almost everywhere).

Remark 2. One can also deduce the polynomial Ham Sandwich theorem directly from the ordinary Ham Sandwich theorem (in dimensions) by embedding into via the Veronese embedding, and then thickening the images of slightly in an appropriate fashion; we leave the details as an exercise to the reader.  

The polynomial Ham Sandwich theorem should be compared with the following finitary counterpart, which morally corresponds to the case when all the are points (but works over arbitrary fields F), and which was a key tool in Dvir’s proof of the Kakeya conjecture in finite fields:

Lemma. Let and let be points in for some field F.  Then there exists a non-trivial polynomial of degree at most d which vanishes on all of .

Proof. The evaluation map is a linear map from a -dimensional space to a -dimensional space, and must therefore have a non-trivial kernel.

– Connection with the Kakeya problem –

Now we connect the polynomial Ham Sandwich theorem to the Kakeya problem.  We begin by replacing the continuous Kakeya set conjecture with a more quantitative “-discretised” problem:

Kakeya maximal conjecture. Let , and let be a collection of cylindrical tubes pointing in a -separated set of directions (thus the directions of any two of the tubes make an angle of at least ).  For each , let be the set of points x which are contained in at least of the tubes .  Then the volume of obeys the bound for any .

Here we are using the asymptotic notation that if for some positive constant c (if the is subscripted by parameters, this indicates that c is allowed to depend on those parameters); we always allow constants to depend on the dimension n.  This conjecture (which is limiting the extent to which tubes in different directions can overlap) implies the Kakeya set conjecture (for both Minkowski and Hausdorff dimension) by fairly standard arguments from geometric measure theory, see e.g. this paper of Bourgain or these lecture notes of myself. The factor of is natural (and best possible), as can be seen by considering the example in which and all the tubes pass through a common point.

[The name "maximal conjecture" has to do with the formulation of the above conjecture involving the Kakeya maximal function, which I will not discuss here.]

The maximal conjecture (and the set conjecture) is verified in the two-dimensional case (with the one-dimensional case being trivial), but only partial results are known in higher dimensions.  However, one can do better if one only considers certain types of overlap.  Let us say (somewhat informally) that a point x has non-planar multiplicity with respect to a given collection of tubes if there exist separate families of tubes each passing through x, such that given any tubes from each of these three families, the solid angle between the directions is comparable to 1.  (Informally, this is a stronger assertion than saying that x has tubes passing through it, because we prohibit these tubes from being essentially contained in a hyperplane).  Then, as a special case of Guth’s results, one has

Multilinear Kakeya conjecture (special case): Let be as in the Kakeya maximal conjecture, and let be the set of points with non-planar multiplicity .  Then .

Informally, this implies that the only counterexamples to the Kakeya maximal conjecture can come from configurations of tubes such that the tubes that pass through a typical point largely lie in a hyperplane.  A previous paper of Bennett, Carbery, and myself established this estimate with an additional loss of by a totally different method (based on heat flow monotonicity formulae).  For a precise statement of the full multilinear Kakeya conjecture (which is now proven without any epsilon loss), I refer you to that paper (or the paper of Guth).

Let’s now sketch why the above result is true (details can be found in the paper of Guth).  I’ll drop the dependence of implied constants on n.  Let be a maximal -net of (i.e. a set of -separated points in that is maximal with respect to set inclusion), then it will suffice to show that

(1).

Let be the cube of sidelength centred at with sides parallel to the axes.  Applying the polynomial Ham Sandwich theorem, we can find a non-trivial polynomial P of degree whose zero locus bisects each of the cubes .

For each j, we claim that the hypersurfaces have surface area .  Indeed, if instead one of the had surface area , this would imply that the projection of to any (n-1)-dimensional coordinate subspace of has area , in contrast with the projection of itself which has area .  Thus for each the complement of V in contains a subset of of relative density that consists entirely of line segments of length in the basis direction .  From this it is not hard to see that contains a path-connected component of relative density , which contradicts the claim that bisects .

On the other hand, we know that meets tubes , which are arranged in a non-planar fashion.  Because of this, one can show that for a “typical” tube hitting , the projection of to the orthogonal complement of the direction of has area .  (Basically, the point is that at any given point of , the normal vector cannot be perpendicular (or close to perpendicular) to all the directions of all the simultaneously, due to non-planarity.)  To simplify the exposition, let us assume that in fact all tubes touching are typical.

Each touches tubes (they may touch more than this, but for sake of exposition let us suppose that they touch exactly this number of tubes).  By double counting, this means that each tube touches about

(2)

cubes on the average, where the inequality in (2) comes from the -separated directions of the tubes.  In particular, we can find a (typical) tube which touches at least such balls. Let be the direction vector of .

Now look at .  (Technically, one has to replace by a slight thickening of itself here, but let us ignore this technicality.)  This set contains disjoint sets of the form .  Each of these sets, when projected to the orthogonal complement of , has measure .  On the other hand, itself, when projected to this complement, has a measure of .  By the pigeonhole principle, we may thus find a positive measure family of lines in the direction passing through which intersect at  of the .  In particular, all lines in this family intersect V in different points.

On the other hand, the restriction of P to is a polynomial of degree .  If this degree is much less than , this forces P to vanish on each line [cf. Dvir's argument]; since the set of such lines has positive measure, this forces P to be identically zero, a contradiction.  Hence we must have

which when combined with (2), gives (1).

In previous lectures, we considered the problem of detecting tuples of primes in various linear or convex sets; in particular, we considered the size of sets of the form , where is the set of primes, and V is some affine subspace of .  (For instance, the twin prime conjecture would correspond to the case when k=2 and , while the Green-Tao theorem would correspond to the case .  We refer to elements of as prime points.  The prime tuples conjecture implies the following qualitative criterion for when such a set of prime points should be “large”:

Qualitative prime tuples conjecture. Let V be an affine subspace of .  Suppose that

1. (No obstructions at infinity)  For any N, affinely spans all of V, where .  (In particular, is non-empty.)
2. (No obstructions at q)  For any , affinely spans all of V, where .  (In particular, is non-empty.)

Then affinely spans all of V.  (In particular, V contains at least one prime point.)

Both of the hypotheses in this conjecture are easily verified for any given V, the first by (integer) linear programming and the second by modular arithmetic.  This conjecture would imply several other results and conjectures in number theory, including the twin prime conjecture and the Green-Tao theorem.  Needless to say, it remains open in general (though the results mentioned in the previous lecture give partial results in the case when V is at least two-dimensional and non-degenerate).

Now we attempt to generalise the above conjecture to the setting in which V is an algebraic variety rather than an affine subspace.  (This would cover some famous open problems in number theory, for instance the Landau problem that asks whether there are infinitely many primes of the form .) The notion of a set affinely spanning V is then naturally replaced by the notion of a set being Zariski dense in V, which means that the set is not contained in any strictly smaller subvariety of V.  One could then formulate a naive generalisation of the above conjecture by replacing “affine space” and “affinely spans all of” with “algebraic variety” and “is Zariski dense in” respectively.  However, the hypotheses are now no longer easy to verify; indeed, just the problem of determining whether V contains an integer point is essentially Hilbert’s tenth problem, which by Matiyasevich’s theorem is known to be undecidable for general V.  Indeed, since one can encode any computable set in terms of the integer points of a variety V, it is not too difficult to see that this conjecture fails in general.  (An amusing historical connection here: one of the first demonstrations of the undecidability of Hilbert’s tenth problem by Robinson, Davis, and Putnam was conditional on the existence of arbitrarily long progressions of primes (i.e. the Green-Tao theorem), although subsequent proofs did not need this fact.)

Since arbitrary algebraic varieties are far too general to have any hope of a reasonable theory, one should look for prime points in much more special sets.  An important class here is that of an orbit in , where b is some vector in and is some finitely generated subgroup of .  (One can also consider the slightly more general set of images under a polynomial map, but for simplicity let us stick to just orbits.) Of course one should take b to be primitive (not a multiple of any smaller vector), since one clearly will have a difficult time finding prime points in otherwise.

The orbit will be Zariski dense in some algebraic variety V, and is clearly a collection of integer points (though it may not cover all of ).  Assuming no local obstructions at infinity or at q (which means that and are Zariski dense in V), one could then conjecture that is also Zariski dense in V (which, if V is infinite, would in particular imply that the orbit contains infinitely many prime points).

For simplicity let us restrict attention to the two-dimensional case k=2, which is already highly non-trivial; Bourgain, Gamburd and Sarnak have recently begun to get some preliminary results in k=3 but I will not discuss them here.  Thus is now a finitely generated subgroup of .  If this subgroup is elementary – e.g. if it is cyclic – then the orbit can be exponentially sparse (a ball of radius R may only contain points), and it becomes extremely difficult to do any sieving or primality detection.  (In this case, the problem becomes comparable to such notoriously difficult questions as whether there are infinitely many Mersenne primes.)  It thus makes sense to restrict attention to non-elementary subgroups of – groups which contain a copy of the free non-abelian group on two generators, or equivalently any group whose Zariski closure is all of (or equivalently yet again, a group whose limit set consists of more than one point).  In this situation, Bourgain, Gamburd, and Sarnak conjectured:

Conjecture 2. Let be a non-elementary subgroup of , and let b be a primitive element of .  Suppose that there are no local obstructions at infinity or at finite places q.  Then is Zariski dense in the plane (in particular, is infinite).

This conjecture remains open.  However, as in the linear situation, one can make progress if one replaces primes with almost primes – products of at most r primes for some bounded r.  (There are certainly prior results for nonlinear patterns in the almost primes; for instance, it is a famous result of Iwaniec that there are infinitely many numbers of the form that are the product of at most two primes.)  In particular, Bourgain, Gamburd, and Sarnak were able to show

Theorem. Let be as in Conjecture 2.  Then there exists an r such that is Zariski dense in the plane, where is the set of numbers that are the product of at most r primes.

Several further generalisations and extensions of this result, with a similar flavour, are known, but will not be discussed here.  There are a number of amusing special cases of these results, for instance one can show that there exist infinitely many Appollonian circle packings of the unit circle by four other mutually tangent circles, all of whose radii is the reciprocal of an almost prime, or infinitely many Pythagorean triples whose area is an almost prime (for a sufficiently large r in the definition of “almost prime”).

Let me now discuss some of the key ideas in the proof of this theorem.  One begins by rephrasing the question in a more quantitative (or finitary) manner.  In the linear case, this would be done by counting the number of points in that lie inside some large Euclidean ball, thus using the Euclidean (or Archimedean) notion of distance to localise the problem.  This can also be done here, but it turns out to be more convenient to instead use the word metric induced by the finite generating set S of (which we can take to be symmetric for convenience, thus ).  One thus looks at sets of the form , where consists of all words formed by products of at most R elements of S.  A major new difficulty here compared to the linear theory is the exponential growth of (a consequence of the non-elementary nature of ).  (Though it is not immediately apparent, the same problem also arises if one uses Euclidean balls instead of word metric balls, due to the multiplicative rather than additive nature of the group .)

The next step is to use sieve theory.  Recall the sieve of Eratosthenes, which expresses the set of all (large) primes as the integers, minus the multiples of two, minus the multiples of three, and so forth.  Using the inclusion-exclusion principle, we can thus view the indicator function of the primes, when restricted to an interval such as , as equal to 1, minus the indicator function of the even numbers, minus the indicator function of the multiples of three, plus the indicator function of the multiples of six, and so forth.  This leads to the Legendre sieve

, (1)

valid in an interval as long as one restricts d to those integers which are products of primes less than N.  Here is the Möbius function.

The basic idea of sieve theory is to replace the indicator function of the primes (or almost primes) by a more general divisor sum

,

where the sieve weights are chosen in order to optimise the final bounds in the sieve (they typically resemble “smoothed out” versions of the Möbius functionin order that these sieves be large on the almost primes and small elsewhere).  In order for the sieve to be practical, one wants to restrict d in this sum to be relatively small, for instance for some absolute constant (values such as are fairly typical).  The selection of the sieve weights is now a well-developed science (see also my earlier post, and Kowalski’s guest post, on this topic), and Bourgain, Gamburd and Sarnak basically use off-the-shelf sieves (in particular, combinatorial sieves and the Selberg sieve) in their work.  Inserting these standard sieves into the problem at hand, the task of counting almost primes in the finite set then quickly reduces to the question of getting good estimates on sets such as for various q.  This amounts to much the same thing as asking for good equidistribution bounds for modulo q, thus we project the generating set S, and the ball it produces, from to .  For sieving purposes it turns out to be necessary to consider all squarefree moduli q, but for simplicity we shall only discuss the (massively easier) case when q is prime.

The reduction to an equidistribution problem converts the original sieving problem to a more combinatorial one, involving the Cayley graph G on induced by the set S, thus two vertices are connected by an edge in G if lie in S (modulo q).  The image of the ball in is then the set of points one can reach in the graph G from the origin by walking on a path of length at most R.  The desired equidistribution result one needs can then be viewed as a mixing result for the random walk along the graph G.

Standard graph theory then tells us that the task reduces to showing that the graphs G form a family of expander graphs as (recall we are restricting q to be prime for simplicity).  There are many equivalent definitions of what an expander graph is, but let us give a spectral definition that is specialised to Cayley graphs.  The symmetric generating set S induces a natural measure

that is the uniform distribution on S, which controls the random walk along G; note that if is a function, then the convolution is another function, whose value at any vertex x is the average value of f at all the neighbours of x.  The operation is then a self-adjoint contraction on which leaves the function 1 invariant, so its largest eigenvalue is equal to 1.  The expander graph condition is then equivalent to the existence of a spectral gap for the second largest eigenvalue, where is a constant independent of q.

Of course, to have a spectral gap, one necessary condition is that be strictly less than 1.  This can be easily seen to be equivalent to the statement that G is connected, which in turn is equivalent to the statement that the projection of S to generates all of .  This statement can be verified to be true, either by direct consideration of all possible subgroups of , or by the strong approximation property.  However, mere connectedness is not enough to ensure that a Cayley graph is an expander family (which can be viewed as a sort of “robust” version of connectedness, which can survive the deletion of large numbers of edges).  For instance, the Cayley graph of the generating set in is connected, but does not form an expander family as ; the second largest eigenvalue is about only.  (One can also see that the random walk on this Cayley graph takes a long time (about O(N^2) steps) before it mixes to be close to the uniform distribution; with expander graphs on a set of N vertices, mixing instead occurs in time , thanks to the spectral gap.)

Obtaining the spectral gap property requires more work.  When the original subgroup of is as large as a finite index subgroup (in particular, if it is a congruence subgroup), this gap follows from a celebrated theorem of Selberg providing a similar spectral gap for arithmetic quotients of the upper half-plane.  Smaller examples (in which the index is now infinite) were first constructed by Shalom and by Gamburd, with the latter following the method of Sarnak and Xue.  Then (in the case of prime q), Bourgain and Gamburd extended the method using additional tools from additive combinatorics to handle all non-elementary subgroups.

Let us now describe the method of proof.  As mentioned briefly earlier, the existence of a spectral gap implies a strong mixing property: the iterated convolutions of the probability measure (which can be interpreted as the probability distribution of a random walk on n steps) converges exponentially fast to the constant distribution on .  Since the latter distribution has an l^2 norm of , we see in particular that for any fixed , we will have

(2)

once n is a sufficiently large multiple of .  This can also be seen explicitly from the trace formula

. (3)

In general, this implication between spectral gap and rapid mixing (2) cannot be reversed; the problem is that only directly influences one term in the summation on the right-hand side of (3), and so upper bounds on the left-hand side do not translate efficiently to upper bounds on . However, there is an algebraic miracle that happens in the case of groups such as that allows one to reverse the implication:

Lemma (Frobenius)  Let q be prime.  Then every non-trivial finite-dimensional unitary representation of has dimension at least (q-1)/2.

Proof. Observe that can be generated by parabolic elements, so given a non-trivial representation , there exists a parabolic element a whose representation is non-trivial.  By a change of basis we may take

On the one hand, we have and hence ; thus all eigenvalues of are roots of unity.  On another hand, is non-trivial, so at least one of the eigenvalues of differs from 1.  Thirdly, conjugating a by diagonal matrices in , we see that a is conjugate to whenever m is a quadratic residue mod q, and so the eigenvalues of must be stable under the operation of taking powers.  On the other hand, there are (q-1)/2 quadratic residues.  Putting all this together we see that must take at least (q-1)/2 distinct eigenvalues, and the claim follows.

Remark. For our purposes, the exact value of is irrelevant; any multiplicity which grows like a power of q would suffice.

Applying this lemma to the eigenspace of , we obtain

Corollary. The second eigenvalue of the operation appears with multiplicity at least .

Combining this corollary with (3), one can now reverse the previous implication and obtain a spectral gap as soon as one gets a mixing estimate (2) for some and some sufficiently small .

The task is now to obtain the mixing estimate (2).  The quantity starts at 1 when and decreases with n.  If we assume (as we may) that S generates a free group, then it is not hard to see that expands rapidly for (because all the words generated by S will be distinct until one encounters the “wrap-around” effect of taking residues modulo q).  Using this one can get a preliminary mixing bound

for some absolute constant and some .  Also, since S modulo q generates all of , we know that the probability measure is not trapped inside any proper subgroup H of ; indeed, using the classification of subgroups of (or some general “escape from subvarieties” machinery of Eskin, Moses, and Oh) one can show that

for any such subgroup, and some .  The result now follows from iterating the following lemma, which is the heart of the argument:

flattening lemma. Let be a symmetric probability measure on which is a little bit dispersed in the sense that

for some , and is not concentrated in a subvariety in the sense that for any proper subgroup H of .  Suppose also that is not entirely flat in the sense that

(note that the minimal norm for a probability measure is comparable to , attained for the uniform distribution).  Then is “flatter” than in the sense that

for some depending on .

In the special case when is the uniform distribution on some set A, the flattening lemma is very close to the following theorem of Helfgott:

Product theorem. Let q be a prime.  Let A be a subset of which is not too big in the sense that for some , and which is not contained in any proper subgroup H of .  Then for some depending on .

Indeed, by using some standard additive combinatorics, in particular a (non-commutative version of) the Balog-Szemerédi-Gowers lemma (which can be found for instance in my book with Van Vu), which connects “statistical” multiplication, such as that provided by convolution , with “combinatorial” multiplication, coming from the product set operation , one can show that these two statements are in fact equivalent to each other.

The product theorem is a manifestation of certain “nonlinear” or “noncommutative” behaviour in the group ; see my Milliman lecture for a bit more discussion on this.  For now, let me just say that Helfgott’s proof on this uses a variety of algebraic and combinatorial computations exploiting the special structure of (especially how commutativity or non-commutativity of various elements in this group interact with the trace of various combinations of these elements), as well as the following sum-product estimate of Bourgain-Katz-Tao and Bourgain-Glibichuk-Konyagin:

Sum-product theorem. Let q be prime. Let A be a subset of which is not too big in the sense that for some .  Then for some depending only on .

There are now some quite elementary proofs of this theorem, but I will not discuss them here (see e.g. my earlier blog post on this topic).  I should note, though, that the bulk of the Bourgain-Gamburd-Sarnak work is preoccupied with establishing a suitable extension of this sum-product theorem to the case when q is not prime, in a manner which is uniform in the number of prime factors; this turns out to be a surprisingly difficult task.

However, for most patterns, there is no analogue of Szemerédi’s theorem, and the strategy used in the previous lecture or AMS lecture cannot be directly applied.  For instance, it is certainly not true that any subset of integers with positive upper density contains any twins ; the multiples of three, for instance, form a counterexample, among many others.  (In fact, there are so many counterexamples here, that it looks unlikely that the twin prime conjecture can be attacked by this method without a significant new idea.)

Furthermore, even in the cases when these methods do work, for instance in demonstrating for each k that there are infinitely many progressions of length k inside the primes, they do not settle the more quantitative problem of how many progressions of length k there are asymptotically in any given finite range of primes, e.g. the primes less than a number N in the asymptotic limit .  This is because Szemerédi’s theorem provides a lower bound for the number of progressions in a large finite set, but not a matching upper bound.  (For instance, given a subset A of of density 1/2, the number of progressions of length 3 in A can be as large as (if consists of the even integers from 1 to N, for instance) and as small as (if is a randomly chosen set of density 1/2), and can even be a little bit smaller by perturbing this example slightly.)

On the other hand, as discussed in the previous lecture or AMS lecture, one can use standard random models for the primes to predict what the correct asymptotic for these questions should be.  For instance, the number of arithmetic progressions of a fixed length k consisting of primes less than N should be asymptotically

(1)

where the product is over all primes p, and the quantity is defined as

for , and

for .

The various terms in this complicated-looking formula can be explained as follows.  The “Archimedean” factors and come from the fact that the number of arithmetic progressions of natural numbers less than N is .  The “density” factor comes from the prime number theorem, which roughly speaking asserts that each of the k elements in a typical arithmetic progression has a “probability” of being prime.  The “local” factors measures how much bias arithmetic progressions with respect to being coprime to a fixed prime p, which is relevant for progressions of primes, since primes of course tend to be coprime to p.  More precisely, can be defined as the probability that a random arithmetic progression of length k has all entries coprime to p, divided by the probability that a random collection of k independent numbers are all coprime to p.  It is not difficult to show that the product converges to some finite non-zero number for each k.

Similar heuristic asymptotic formulae exist for the number of many other patterns of primes; for instance, the number of representations of a large integer N as the sum of two primes should be equal to , where is the probability that two randomly chosen numbers conditioned to be coprime to p sum to N modulo p, divided by the probability that two randomly chosen numbers sum to N modulo p.  (In particular, for odd N, reflecting the fact that it is very difficult for an odd number to be representable as the sum of two primes.  More generally, one can compute that when p divides N, and otherwise.)  A more general prediction for counting linear patterns inside primes exists, and is essentially the Hardy-Littlewood prime tuples conjecture.  This conjecture, which is widely believed to be true, would imply many other conjectures in the subject, such as the twin primes conjecture and the Goldbach conjecture (for sufficiently large even numbers).  Unfortunately, the cases of the prime tuples conjecture which would have these consequences remain out of reach of current technology.

Using some elementary linear algebra, one can recast the prime tuples conjecture not as a question of finding linear patterns inside primes, but rather that of solving linear equations in which all the unknowns are required to be prime, subject to some additional linear inequalities.  For instance, finding progressions of length k consisting entirely of primes less than N is essentially the same as asking for solutions to the system of equations

and inequalities

where the unknowns are required to be primes.  More generally, one could imagine the question of asking the number of k-tuples consisting entirely of primes which is contained in some convex set B in of some intermediate dimension , which is contained in a ball of radius O(N) around the origin.  For instance, in the above example B is the 2-dimensional set

.

(We make the technical assumption that the linear coefficients of the equations defining the d-dimensional subspace that B lives in are independent of N; k and d are of course also assumed to be independent of N.  The constant coefficients, however, are allowed to vary with N; this is the situation that comes up for instance in the Goldbach conjectures.)  One can think of the problem of finding points in B as that of solving k-d equations in k unknowns.  One can also generalise this problem slightly by enforcing some residue constraints on the unknowns, but we will ignore this minor extension to simplify the discussion.

The prime tuples conjecture for this problem can roughly speaking be phrased as follows.  Suppose that the number if k-tuples in B consisting of natural numbers is known to be

for some constant independent of N (one can think of this constant as the normalised volume of B).  Suppose also that for any fixed prime p, the number of k-tuples in B consisting of natural numbers coprime to p is known to be

for some constant $latex\ beta_p$ independent of n.  (The factor is natural, as it represents the proportion of tuples of k natural numbers in which all the entries are coprime to p.)  Then the prime tuples conjecture asserts that the number of k-tuples in B consisting of primes should be

(2)

In particular, this can be shown to imply that if (thus there are no obstructions to solving the system of equations at infinity) and if for all p (thus there are no obstructions to solvability mod p for any p) then there will exist many solutions to the system of equations in primes when N is large enough.

As mentioned earlier, this conjecture remains open in several important cases, most particularly in the one-dimensional case d=1.  For instance, the twin prime conjecture would follow from the case , but this case remains open.  However, there has now been significant progress in the higher dimensional cases , especially when the codimension (representing the number of equations in the system) is low.  Firstly, the prime number theorem settles the zero codimension case (and is pretty much the only situation in which we can handle a $d=1$ case).  The Hardy-Littlewood circle method, based on Fourier analysis, settles all “non-degenerate” cases when , where “non-degenerate” roughly speaking means that the problem does not secretly contain a problem inside it as a lower-dimensional projection (e.g. would be degenerate; more generally, B is non-degenerate if it is not contained in any hyperplane that can be defined using at most two of the unknowns).  It can also handle some cases in which the codimension k-d exceeds 1 (e.g. one could take the Cartesian product of some codimension 1 examples); the precise description of what problems are within reach of this method is a little technical to state and will not be given here.

The main result of this paper of Ben Green and myself (combined with results from these other papers of ours) is the following:

Theorem. The prime tuples conjecture is true in all non-degenerate situations in which .  If the inverse conjecture for the Gowers norms over the integers is true, then the prime tuples conjecture is true in all non-degenerate situations in which .

I will say a little bit more about what the inverse conjecture for the Gowers norms is later.  This theorem unfortunately does not touch the most interesting case (when the patterns one is seeking only have one degree of freedom), but it does largely settle all the other cases.  For instance, this theorem implies the asymptotic (1) for prime progressions of length k for (the cases were established earlier by van der Corput using the circle method), and the case of higher k would also follow from the theorem once the inverse conjecture is proven.  As with the circle method, we can also unconditionally handle some cases in which d is less than , but the precise statement here is technical and will be omitted.  (Details and further examples can of course be found in our paper.)

Now I would like to turn to the proof of this theorem.  At first glance, the result looks like it is going to be quite complicated, due to the presence of all the different factors in the asymptotic (2) that one is trying to prove.  However, most of the factors can be dealt with by various standard tricks.  The Archimedean factor can be eliminated from the problem by working locally (with respect to the infinite place), covering B by cubes of sidelength o(N).  For similar reasons, the local factors can be eliminated by working locally mod p (i.e. restricting to a single residue class mod p for various small p).  The factors , which come from the density of primes in the region of interest (i.e. from the prime number theorem), can largely be compensated for (with some effort) from the transference principle technology developed in our earlier paper on long progressions in the primes, which was discussed in the previous lecture (or at the AMS lecture).  After all this, the problem basically boils down to the following.  We have a certain subset A of the integers with some density (one should think of A as being a “model” for the primes, after all the distorting structure coming from local obstructions has been stripped out; in actuality, one has to replace the set A by a weight function of mean value , but let us ignore this technicality to simplify the discussion).  We pick a random instance of some linear pattern inside (for sake of concreteness, let us pick a random arithmetic progression ) and ask what is the probability that all the elements of this pattern lie in A.  Since A has density , we expect each element of our random progression to have a probability to lie in A.  (This is not quite the case if A is biased to lie on one side of than on the other, but it turns out that one can ignore this possibility.)  Since our pattern consists of k elements, we thus expect

  (3)

Roughly speaking, the key issue in proving the theorem is to work out some “easily checkable” conditions on A that would guarantee that the heuristic (3) is in fact valid.  One then verifies that these “easily checkable” conditions do indeed hold for the set A of interest (which is a proxy for the set of primes).

As stated before, we expect for each .  Thus (3) is asserting in some sense that the events are “approximately independent”.  This would be a reasonable assertion if A was pseudorandom (i.e. it behaved like a random subset of of the given density ), and is consistent with the general heuristic from number theory that we expect the prime numbers to behave randomly once all the “obvious” irregularities in distribution (in particular, irregularity modulo p for small p) has been dealt with.  But if A exhibits certain types of structure (or at least some bias towards structure), then (3) can fail.  For instance, suppose that A consists entirely of odd numbers.  Then, if the first two elements n, n+r of an arithmetic progression lie in A, they are necessarily odd, which then forces the rest of the elements of this progression to be odd.  As A is concentrated entirely in these odd numbers, these elements of the progression are thus expected to have an elevated probability of lying in A, and so the left-hand side of (3) would be expected to significantly exceed the former once .  (The asymptotic (3) becomes trivially true for .) A similar distorting effect occurs if A is not entirely contained in the odd numbers, but is merely biased towards them, in that odd numbers are more likely to lie in A than even numbers.  In this example, the bias in A caused the number of progressions to go up from the expected number predicted by (3); it is also possible (but more tricky) to concoct examples in which bias in A forces the number of progressions to go down somewhat, though Szemerédi’s theorem does prevent one from extinguishing these progressions completely when N is large.

Bias towards odd or even numbers is equivalent to a correlation between A and the linear character ; the algebraic constraints between the , and in particular the relationship

(4)

can be viewed as the underlying source of the distorting effect that can prevent (3) from holding for .  The same algebraic constraint holds for any other linear character, e.g. the Fourier character (where ) for some fixed frequency , for much the same reason that two points on a line determine the rest of the line (it is also closely related to the fact that the second derivative of a linear function vanishes).  Because of this, we expect (3) to be distorted when A correlates with such a character (which means that the Fourier coefficient is unexpectedly large in magnitude).

It turns out that in the case k=3 of progressions of length three, correlation with a linear character is the only source of distortion in the count (3).  A sign of this can be seen from the identity

which can be viewed as a “Fourier transform” of the algebraic identity (4).  One can formalise this using (a slight variant of) the above identity and some other Fourier-analytic tools (in particular, the Plancherel identity) to conclude

Inverse theorem for length three progressions. (Informal statement)  Let k=3. Suppose that A is a subset of of density for which (3) fails.  Then A correlates with a non-trivial linear character .  (“Non-trivial” basically means that oscillates at least once on the interval .)

Applying this theorem in the contrapositive, we conclude that we can justify the asymptotic (3) in the k=3 case as long as we can show that A does not correlate with a linear character.  In the case when A is a proxy for the primes, this task essentially boils down to that of establishing non-trivial estimates for exponential sums over primes, such as

;

for technical reasons it is more convenient to deal with slight variants of this sum such as

(5)

where is the von Mangoldt function, or

(6)

where is the Möbius function.  (There are various elementary identities, such as summation by parts, that allow one to express one of these sums in terms of the others.  One has a lot of flexibility in here as long as one retains a factor in the sum which is somehow sensitive to the prime factorisation of n.)  The reason for using these functions instead is that they enjoy a number of very useful identities, such as

(7)

and

(8)

for any Dirichlet character (where is the Dirichlet L-function), and also multiplicative identities such as

(9)

and

(10)

To cut a long story very short, identities such as (7), (8) are useful for estimating (5), (6) respectively in the major arc case when is rational or close to rational (with small denominator), while (variants of) identities such as (9) or (10) (in particular, certain truncated versions of (9) and (10) such as Vaughan’s identity) are useful for estimating (5), (6) respectively in the minor arc case when is far from a rational with small denominator (or close to a rational with large denominator).  This theory was pioneered by Vinogradov (and also Hardy and Littlewood), and refined and simplified over the years with many contributions by Vaughan, Davenport, Heath-Brown, and others, with the upshot being that we now have a fairly good understanding of sums such as (5) and (6), and in particular that the sum (6) exhibits a strong cancellation (by a factor of for any fixed A) uniformly in (i.e. we can handle both major and minor arcs with a uniform estimate).  Combining this with the inverse theorem and the previous reductions, one can eventually establish the asymptotic (1) in the k=3 case.  (This is not exactly how the original proof of (1) by van der Corput in this case proceeded, but both proofs use the same general ingredients and method, i.e. the Hardy-Littlewood circle method and the Vinogradov method for estimating exponential sums.)

Now we turn to progressions of longer length, such as the case k=4.  Here, linear characters continue to cause bias that distorts the expected asymptotic (3), and so it is still necessary to control sums such as (5) or (6) to prevent such bias from occuring.  However, a major new difficulty arises that new sources of bias also arise.  For instance, if one takes a quadratic character for some , then one easily verifies the identity

(11)

which reflects the fact that the third derivative of a quadratic function (such as ) is zero (it also reflects the fact that three points on the graph of a quadratic (i.e. a parabola) determine the entire parabola).  One consequence of this is that if are all close to 1 (say), then will be also.  This constraint between the four values of along an arithmetic progression suggests that if A exhibits significant correlation with , then the event that n+3r lies in A will be influenced in some non-trivial manner by whether n, n+r, and n+2r already lie in A, which will lead to some distortion in (3).  Thus one will need to update the inverse theorem by taking quadratic characters into account.  (Easy examples show that it is possible for a set to correlate with a quadratic character without exhibiting any correlation with linear characters, by choosing a quadratic character with irrational coefficients.)  The most optimistic conjecture in this regard would be

Proposed inverse theorem for length four progressions. (Informal statement)  Let k=4. Suppose that A is a subset of of density for which (3) fails.  Then A correlates with a non-trivial quadratic character .

Unfortunately, this conjecture fails.  The easiest way to see this is to consider a bracket quadratic character, such as , where is the greatest integer function.  This is not quite a quadratic character, because is not quite a linear function.  However, this function is linear “a positive fraction of the time”; if one picks x and y to be some generic real numbers, one expects to equal about half of the time.  Because of this, we see that while the identity (11) certainly doesn’t hold all the time for , it does hold a positive fraction of the time, and this is enough to still cause significant bias to disrupt (3) if A correlates with this object.  It is furthermore possible to concoct examples of sets A that correlate with bracket quadratic characters such as but not any linear or quadratic characters.  (The same phenomenon is not visible at the linear level; a bracket linear phase such as can be rewritten as , which by Fourier series can be expressed as a linear combination of linear characters for integer k.  Note that the same trick does not work for .)  Once one throws in these bracket quadratics, it turns out that these do in fact constitute all the possible obstructions to (3) holding in the k=4 case, as shown by Ben Green and myself:

Inverse theorem for length four progressions. (Informal statement)  Let k=4. Suppose that A is a subset of of density for which (3) fails.  Then A correlates with a non-trivial bracket quadratic character for some real numbers and bounded J.

The proof of this result involves both Fourier analysis and additive combinatorics, relying heavily on ideas from a paper of Gowers on Szemerédi’s theorem for progressions of length 4.  It will not be discussed here.

In view of the inverse theorem, the problem of establishing the asymptotic (1) for length 4 progressions then reduces (by suitable generalisations of the various methods discussed previously) to that of estimating exponential sums of which

is a typical example.  One can begin to apply the methods of Vinogradov and Vaughan to control this type of expression.  But one is soon faced with the problem of understanding the distribution of quadratic phases such as , and in particular to estimate exponential sums such as

.

This turns out to be somewhat unpleasant; the standard technology of Weyl differencing and the van der Corput lemma eventually works (see this paper of Ben Green and myself), but does not scale well to bracket polynomials of higher degree such as , which would be necessary if one were to extend (1) to k beyond 4.

To resolve this, and inspired by the work in ergodic theory by Furstenberg-Weiss, Host-Kra, Bergelson-Leibman, and others, we re-interpreted bracket polynomials from a more dynamical systems perspective.  To motivate this, observe that the linear character is closely tied to the circle rotation on the unit circle , in that the character can be described as a function of an orbit on this system, where is the origin and is the exponential function.  In a similar spirit, a quadratic character such as can be expressed in terms of the skew shift system on the torus , being of the form where and .  More generally, one can also express bracket quadratic polynomials such as in the form , where T is now the action of a group element on a 2-step nilmanifold , and F is some reasonable (e.g. piecewise smooth) function on this nilmanifold.  (See my lecture notes, this paper of Bergelson and Leibman, or this paper of Ben and myself for details.)  The relevance of 2-step nilpotent groups and nilmanifolds to length 4 progressions can be glimpsed in the identity

which is valid for all g, x in a 2-step nilpotent group G (compare this with (11)); it is an instructive exercise to prove this identity and to see how the 2-step nilpotency is used.  (To make the connection more precise, one needs a variant of this constraint in which x lies in rather than G, which is harder to state; see Ziegler’s thesis, or this paper of Ben and myself, for details.)  Indeed, one can reformulate the inverse theorem for length 4 progressions in an equivalent form:

Inverse theorem for length four progressions, again. (Informal statement)  Let k=4. Suppose that A is a subset of of density for which (3) fails.  Then A correlates with a non-trivial 2-step nilsequence for some 2-step nilmanifold (of bounded “complexity”), some group rotation , some starting point , and some function (also of “bounded complexity”; e.g. bounded Lipschitz norm will do).

The precise formulation of the theorem is a little technical; see this paper of Ben and myself for details.  Using this theorem and all the standard machinery, the task of establishing asymptotics such as (1) in the k=4 case now reduces to that of understanding sums such as

.

At this point, one can start using the existing theory of equidistribution of orbits on homogeneous spaces (of which nilmanifolds are an important example).  It turns out that the existing theory is not quite quantitative enough for our purposes, and we had to develop a quantitative analogue of this theory; see these blog posts for more discussion.  Anyway, it all works, and gives asymptotics for progressions of length 4 in the primes, as well as other linear patterns of similar “complexity” (e.g. any non-degenerate system of two equations in four prime unknowns is OK).  To handle higher patterns, what we need is

Inverse conjecture for arithmetic progressions. (Informal statement)  Let .  Suppose that A is a subset of of density for which (3) fails.  Then A correlates with a non-trivial -step nilsequence for some (k-2)-step nilmanifold (of bounded “complexity”), some group rotation , some starting point , and some function (also of “bounded complexity”).

This is a consequence of (and very closely related to) the inverse conjecture for the Gowers norm.  It is already known for k=3 and k=4, and hopefully the higher k cases will be resolved in the near future, presumably using a mix of techniques from Fourier analysis, additive combinatorics, and ergodic theory.  At this time of writing, this is the only remaining obstacle before we can understand the asymptotics of linear patterns in primes which genuinely involve two or more free parameters (as mentioned earlier, one-parameter problems such as the twin prime conjecture seem well out of reach of these methods for a number of reasons, one of which is that there is definitely no analogue of the inverse conjecture for such one-parameter patterns).

Polynomial Freiman-Ruzsa conjecture. Let be a function which is an approximate homomorphism in the sense that for all and some set .  Then there exists a genuine homomorphism such that takes at most values.

Remark 1. The key point here is that the bound on the range of is at most polynomial in |S|.  An exponential bound of can be trivially established by splitting into the subspace spanned by S (which has size at most ) and some complementary subspace, and then letting g be the projection of f to that complementary subspace.

Recently, Ben Green and I have shown that this conjecture is equivalent to a certain polynomially quantitative strengthening of the inverse conjecture for the Gowers norm ; I hope to talk about this in a future post.  For this (somewhat technical) post, I want to comment on a possible further strengthening of this conjecture, namely

Strong Polynomial Freiman-Ruzsa conjecture. Let be a function which is an approximate homomorphism in the sense that for all and some set .  Then there exists a genuine homomorphism such that takes values in the sumset for some fixed .

This conjecture is known to be true for certain types of set S (e.g. for Hamming balls, this is a result of Farah).  Unfortunately, it is false in general; the purpose of this post is to describe one counterexample (related to the failure of the inverse conjecture for the Gowers norm for for classical polynomials; in particular, the arguments here have several features in common with those in the papers of Lovett-Meshulam-Samorodnitsky and Green-Tao).  [A somewhat different counterexample also appears in the paper of Farah.]  The verification of the counterexample is surprisingly involved, ultimately relying on the multidimensional Szemerédi theorem of Furstenberg and Katznelson.

(The results here are derived from forthcoming joint work with Ben Green.)

– Description of counterexample –

We let n be a large number, and replace by the -dimensional vector space V of quadratic forms (with a basis given by the monomials with ).  We let be defined by the formula

.

A brief computation shows that for any , the quadratic form is of rank at most three, by which we mean that it is a function of at most three linear forms.  More specifically, we have

(1)

where

Thus, if we let S be the space of quadratic forms of rank at most 3, the hypotheses of the polynomial Freiman-Ruzsa conjecture hold.

– Verification of counterexample –

To establish the counterexample, we assume for contradiction that there exists a linear function such that has bounded rank for all h, and deduce a contradiction (for n sufficiently large).

By hypothesis, we have linear forms for all and some and coefficients for all such that

and in particular (by (1) and linearity of g)

. (2)

The key point is that the linear forms are usually “independent” of the linear forms .  The key lemma in this regard is

Lemma 1. If h, k are selected uniformly and independently at random, then with probability 1-o(1), is not a linear combination of the .  Similarly for .

Proof. By cyclically permuting h,k,h+k it suffices to show this for .  Since there are at most O(1) possible linear combinations amongst the , it suffices to show that for any given assignments of linear forms, that the probability of the event

(3)

is o(1).  Suppose for contradiction that the event (3) holds for a set E of pairs (h,k) in of positive density.  Applying the Furstenberg-Katznelson multidimensional Szemerédi theorem) we can find (for n large enough) a square in E with r non-zero.  Applying (3) for all four pairs and summing, we obtain

(recall we are in characteristic 2).  But the left-hand side is equal to the linear form , which is non-zero, a contradiction.

Now we can obtain the desired contradiction.  For a generic choice of h,k, we now know that none of the are linear combinations of the .  Thus, on a given level set of the (which form a subspace of ), the linear functions are non-constant, and so the range of the triplet must be an affine subspace of which is not contained in any plane in that is parallel to the two of the coordinate axes (i.e. of the form , , or .  This forces the subspace to be parallel to (1,1,1).  But this implies that is non-constant on this space, contradicting (2).

Remark 2. The function f appearing in the above example is closely related to the symmetric polynomial

.

Indeed, one can show that the derivative of is equal to , plus some additional terms which involve only a finite number of linear forms, and the quadratic polynomial .    If it was the case that could be approximated by a linear map modulo low rank errors, then it one could use this to eventually show that correlated with a cubic polynomial; but it is known (from the papers of Lovett-Meshulam-Samorodnitsky and Green-Tao) that this is not the case.  Thus there is an alternate way to verify that the above example is indeed a counterexample to the strong polynomial Freiman-Ruzsa conjecture.

In the rest of this post, I would like to describe the inverse conjecture (in both combinatorial and ergodic forms), and sketch how one deduces one from the other via the correspondence principle (together with two additional ingredients, namely a statistical sampling lemma and a local testability result for polynomials).

– Polynomials and the Gowers norm –

Let F be a finite field, and let be a vector space over that field.  Given any function taking values in the unit circle , we define the additive derivative of P in the direction h by the formula

.

Now let .  A function is said to be a polynomial of degree if one has for all .  Thus for instance, the only polynomial of degree <0 is the zero function, the only polynomials of degree <1 are the constants, the only polynomial of degree <2 are the affine characters, and so forth.  If is a field of prime order, and we make the additional assumption that P takes values in the roots of unity (which we can identify with F in the usual manner), then we can express the polynomial P in the customary manner as

(1)

for some coefficients ; let us refer to these as classical polynomials of degree <k.  However, if one does not require P to take values in roots of unity, then the above definition also encompasses some non-classical polynomials.  For instance, if , the function defined by P(0)=0 and P(1)=1/4 is of degree <3 (i.e. is a “quadratic” polynomial), but is not classical.  [However, it is a nice exercise to show that in the high characteristic case , every polynomial of degree can be expressed as the sum of a classical polynomial and a constant; thus the genuinely non-classical polynomials are a purely low characteristic phenomenon.  I may write more about the relationship between classical and non-classical polynomials in a future post.]

We can define multiplicative analogues of the above concept.  Given a function , we define the multiplicative derivative to be the function

and say that f is a phase polynomial of degree <k if for all .  It is not hard to show that f is a phase polynomial of degree <k if and only if for some polynomial of degree <k, where is the standard character .

The Gowers uniformity norms are a means to measure the extent to which an arbitrary function behaves like a phase polynomial; if , they are defined by the formula

where denotes the average of f on A.  Thus, for instance, if f is bounded in magnitude by 1, then is a number between 0 and 1, and equals 1 if and only if f is a phase polynomial of degree <k.

We have just seen that exact extremisers of the Gowers norms are given by phase polynomials.  This statement is in fact robust, in that near-exact extremisers of the Gowers norms are close to phase polynomials:

Theorem 1 (local testabilty of phase polynomials)  Suppose that is bounded in magnitude by 1 and obeys the bound .  Then there exists a phase polynomial g of degree <k such that , where denotes a quantity that goes to zero as for fixed k, F, uniformly in the choice of f or n.

This fact is essentially due to Alon, Kaufman, Krivelevich, Litsyn, and Ron; their paper is focused on the case when and P is a classical polynomial, but it is not too difficult to extend the result to the generality of Theorem 1 (this is done in an appendix to our paper, as we need this local testability result in our arguments).  One consequence of this theorem is that one can test (with high confidence) whether a given function is close to a phase polynomial of degree <k, by randomly computing some derivatives and seeing whether they are consistently close to 1.  Note that this test is local, in the sense that one only has to evaluate f at a bounded number of positions in (even as ) in order to perform the test.

Theorem 1 describes what happens when a function “behaves like a phase polynomial 99% of the time”, in the sense that the Gowers norm is very close to 1.  The inverse conjecture for the Gowers norm addresses the more general case in which a function “behaves like a phase polynomial 51% of the time”, in the sense that the Gowers norm is bounded below by some small constant .  (Note that if and was a random boolean function, then would equal +1 about 50% of the time and -1 for the other 50%, leading to a very small Gowers norm.)  More precisely, we have

Inverse Conjecture for the Gowers norm over finite fields. Let be a function bounded in magnitude by 1 such that .  Then there exists a phase polynomial g of degree <k such that for some independent of n.

This conjecture has a number of applications to the structural theory of additive patterns (e.g. arithmetic progressions) in subsets of ; see for instance this paper of Gowers-Wolf.  It is also connected to various problems in computer science, such as generation of pseudorandom bits (see e.g. this paper of Bogdanov-Viola), and to polynomiality testing (see e.g. this paper of Samorodnitsky-Trevisan).

Previously, this conjecture had been verified for by Samorodnitsky (in even characteristic) and by Ben Green and myself (in odd characteristic).  In high characteristic , a partial result (assuming that f was already a classical polynomial of bounded degree) was obtained by Ben and myself; but in low characteristic, it was discovered in that paper and simultaneously by Lovett-Meshulam-Samorodnitsky that the conjecture can fail if one restricts g to be a classical phase polynomial.  Nevertheless we believe the conjecture should still in the low characteristic case hold if we allow g to be non-classical.

Our main results are as follows:

Theorem 2. The inverse conjecture is true in the high characteristic case .

Theorem 3. In general characteristic, a weaker form of the inverse conjecture is true, in which we allow g to be a phase polynomial of degree rather than , for some C(k) depending only on k.

As mentioned above, we expect that we should be able to take C(k)=k in any characteristic, but this seems to require a delicate algebraic analysis and we were not able to establish it here.

– Ergodic theory –

We were able to deduce the combinatorial results in Theorems 2, 3 from ergodic analogues of these results.  Roughly speaking, these ergodic results correspond to “local” versions of the “global” theorems in Theorem 2, 3, because they involve averaging over only a small portion of the underlying space, as opposed to the global averages used to define the Gowers norms.  [In this regard, the ergodic theory results are weaker than the combinatorial ones.  However, there is another direction in which the ergodic results are stronger, in that they encode some additional "measurability" information that is not present in the combinatorial setting; roughly speaking, this information asserts that the function g appearing in the above conjecture can be "locally reconstructed" from f in some sense, as opposed to merely existing in some non-constructive sense.] Fortunately, thanks to some “local-to-global” principles (including the local testability result in Theorem 1), we are able to pass back and forth between the two types of theorems in a fairly “soft” manner.

Whereas the combinatorial results take place in the finitary setting of a finite-dimensional vector space , the ergodic theory results take place in the infinitary setting of a probability space with a measure-preserving action of the countably infinite vector space .   Given a measurable function , one can define multiplicative derivatives

for , and we say that f is a phase polynomial of degree <k if a.e. for all .  We can also define the Gowers-Host-Kra uniformity seminorm for by the formula

it can be shown that the limits exist and that this is indeed a seminorm.  Thus for instance, as in the combinatorial setting, phase polynomials of degree <k have a Gowers-Host-Kra seminorm of 1.

There is an analogue of the inverse conjecture:

Ergodic Inverse Conjecture for the Gowers norm over finite fields. Suppose that X is ergodic.  Let be such that .  Then there exists a phase polynomial g of degree <k such that .

In a forthcoming paper of Bergelson, Ziegler, and myself, we show

Theorem 2′. The ergodic inverse conjecture is true in the high characteristic case .

Theorem 3′. In general characteristic, a weaker form of the ergodic inverse conjecture is true, in which we allow g to be a phase polynomial of degree rather than , for some C(k) depending only on k.

The aim of the current paper is to deduce Theorems 2, 3 from Theorems 2′, 3′ respectively.

– Outline of proof –

The main idea here is to use the compactness and contradiction argument as embodied in the Furstenberg correspondence principle (which was discussed in my previous blog post).  Suppose for instance that Theorem 2 failed; then one would have a sequence of bounded functions (where is going off to infinity) with Gowers norms of order k uniformly bounded from below, but such that becomes increasingly orthogonal to all phase polynomials of degree <k.  The idea is then to exploit weak compactness to take a “limit”, which would be a bounded measurable function on some measure space X with an action.

The problem in doing this is that the convergence given by weak compactness is only in the “weak” or “local” sense; for instance, one can establish convergence results for local statistics, e.g.

for any fixed , but one cannot immediately establish convergence results for global statistics, e.g.

(2)

for the limit objects X, f obtained by the traditional Furstenberg correspondence principle.  To get around this, we first “shuffle” each of the original spaces by a randomly chosen general linear transformation before applying the correspondence principle argument (actually, to be more precise, we pull back to using a random linear transformation from the latter to the former, but never mind this technical detail).   The point of doing this shuffling is that it makes local averages approximate global averages with high confidence, for exactly the same reason that random polls with moderately large sample sizes can be accurate even when the overall population is enormous (see my previous post for more discussion on this).  To cut a long story short, this shuffling (or “statistical sampling”) enables one to obtain convergence results such as (2).  This serves several useful purposes in our paper.  For instance, it allows us to ensure that the limit system X is ergodic.  But more importantly, it allows us to carry the information that each has large Gowers norm into the limit, concluding that f has large Gowers-Host-Kra seminorm.  Thus we are now in a position to apply the hypothesis Theorem 2′ (or Theorem 3′, if the objective is to prove Theorem 3) and conclude that f correlates with a phase polynomial g.

Now we want to pass back from the limit system to the original finitary systems and conclude the desired contradiction.  A new problem emerges: the nature of the convergence of the latter to the former allows one to relate (local) statistics of to (local) statistics of (which in turn relate to global statistics of ), but says nothing about g.  Fortunately, the ergodic inverse theorems tell us that g is measurable, which turns out to imply that g is “nearly local” in the sense that it can be approximated to arbitrary accuracy as some function g’ of finitely many shifts of f, which is almost a phase polynomial and will continue to correlate with f if the approximation is good enough.  This latter object can be passed back to the original finitary systems, creating functions correlating with which are also almost polynomials.  To finish off the proof, we use the local testability (Theorem 1) to approximate by a genuine phase polynomial; this means that the do, after all, have significant correlation with a phase polynomial, contradicting our hypothesis and establishing the result.

It is tempting to try to use this type of argument to handle the inverse conjecture for the Gowers norm on the integers , especially since ergodic inverse conjectures in this setting are already known thanks to the work of Host-Kra (and also Ziegler).  However, there are two non-trivial obstacles here.  Firstly, one cannot “shuffle” the integers (or related objects, such as cyclic groups ) with the same degree of freedom as one can for a vector space, causing a difficulty in relating local and global statistics.  Secondly, whereas polynomials are known to be local testable, the local testability of the analogous object for the integer case, namely that of nilsequences, is unknown.  These difficulties do not appear to be totally insurmountable, but they would require some additional effort to resolve.

]]> http://terrytao.wordpress.com/2008/10/18/from-the-littlewood-offord-problem-to-the-circular-law-universality-of-the-spectral-distribution-of-random-matrices/ Sat, 18 Oct 2008 20:13:51 +0000 Terence Tao http://terrytao.wordpress.com/2008/10/18/from-the-littlewood-offord-problem-to-the-circular-law-universality-of-the-spectral-distribution-of-random-matrices/ Van Vu and I have just uploaded to the arXiv our survey paper “From the Littlewood-Offord problem to the Circular Law: universality of the spectral distribution of random matrices“, submitted to Bull. Amer. Math. Soc..  This survey recaps (avoiding most of the technical details) the recent work of ourselves and others that exploits the inverse theory for the Littlewood-Offord problem (which, roughly speaking, amounts to figuring out what types of random walks exhibit concentration at any given point), and how this leads to bounds on condition numbers, least singular values, and resolvents of random matrices; and then how the latter then leads to universality of the empirical spectral distributions (ESDs) of random matrices, and in particular to the circular law for the ESDs for iid random matrices with zero mean and unit variance (see my previous blog post on this topic, or my Lewis lectures).  We conclude by mentioning a few open problems in the subject.

While this subject does unfortunately contain a large amount of technical theory and detail, every so often we find a very elementary observation that simplifies the work required significantly.  One such observation is an identity which we call the negative second moment identity, which I would like to discuss here.    Let A be an matrix; for simplicity we assume that the entries are real-valued.  Denote the n rows of A by , which we view as vectors in .  Let be the singular values of A. In our applications, the vectors are easily described (e.g. they might be randomly distributed on the discrete cube ), but the distribution of the singular values is much more mysterious, and understanding this distribution is a key objective in this entire theory.

In general, the relationship between the singular values (which encode spectral information about A) and the rows (which encode geometric information about A) are rather complicated.  However, there are some simple identities (or “trace formulae”, if you will) that link the two.  For instance, by computing the second moment in two different ways, one obtains the second moment identity

(1)

where denotes the length of .  This simple identity is already enough to get some crude upper bounds on the “average” value of , although it does not preclude the possibility that a lot of singular values are very close to zero, or that a few singular values are extremely large.

The latter scenario (a few very large singular values) can be controlled by higher moment identities, for instance based on the fourth moment .  But these moments are not good at controlling the former scenario – when one or more singular values comes close to zero, so that A becomes close to singular (or more precisely, ill-conditioned).  For this, we need a different set of identities.  One such identity comes from computing the unsigned determinant in two different ways, one using the singular values and one using the elementary base-times-height formula for volume of a parallelopiped.  This gives the useful identity

or equivalently (assuming A is non-singular)

. (2)

This identity has some ability to control concentration of singular values near the origin (as the logarithm is large in that region), once one understands the distance between a random vector and a subspace spanned by other random vectors.  This is the philosophy used for instance in this paper of mine with Van Vu; related ideas also appear in these papers by Rudelson and Vershynin.

The identity (2) is particularly useful for controlling the least singular value , but is not so useful for controlling other low singular values (e.g. for some moderately small k).  For this, we found an alternate identity, based on the negative second moment .   Observe that the column of has an inner product of 1 with and is orthogonal to all the other rows of A.  A little bit of high school geometry then tells us that the length of this column is equal to .  Since is equal to both and , we conclude that

(3)

(compare with (1) and (2)).  We found the identity (3) to be useful for preventing too many singular values of A from clustering near the origin, much as (1) prevents too many singular values of A from becoming extremely large, thus saving us from having to deploy more sophisticated and lengthier methods to control the singular values .  It may well be that further identities or inequalities of this form may simplify these sorts of arguments further.

I was asked recently to explain what a gauge theory was, and so I will try to do so in this post.  For simplicity, I will focus exclusively on classical gauge theories; quantum gauge theories are the quantization of classical gauge theories and have their own set of conceptual difficulties (coming from quantum field theory) that I will not discuss here. While gauge theories originated from physics, I will not discuss the physical significance of these theories much here, instead focusing just on their mathematical aspects.  My discussion will be informal, as I want to try to convey the geometric intuition rather than the rigorous formalism (which can, of course, be found in any graduate text on differential geometry).

– Coordinate systems –

Before I discuss gauges, I first review the more familiar concept of a coordinate system, which is basically the special case of a gauge when the base space (or parameter space) is trivial.

Classical mathematics, such as practised by the ancient Greeks, could be loosely divided into two disciplines, geometry and number theory, where I use the latter term very broadly, to encompass all sorts of mathematics dealing with any sort of number.  The two disciplines are unified by the concept of a coordinate system, which allows one to convert geometric objects to numeric ones or vice versa.  The most well known example of a coordinate system is the Cartesian coordinate system for the plane (or more generally for a Euclidean space), but this is just one example of many such systems.  For instance:

1. One can convert a length (of, say, an interval) into an (unsigned) real number, or vice versa, once one fixes a unit of length (e.g. the metre or the foot).  In this case, the coordinate system is specified by the choice of length unit.
2. One can convert a displacement along a line into a (signed) real number, or vice versa, once one fixes a unit of length and an orientation along that line.  In this case, the coordinate system is specified by the length unit together with the choice of orientation.  Alternatively, one can replace the unit of length and the orientation by a unit displacement vector along the line.
3. One can convert a position (i.e. a point) on a line into a real number, or vice versa, once one fixes a unit of length, an orientation along the line, and an origin on that line.  Equivalently, one can pick an origin and a unit displacement vector .  This coordinate system essentially identifies the original line with the standard real line .
4. One can generalise these systems to higher dimensions.  For instance, one can convert a displacement along a plane into a vector in , or vice versa, once one fixes two linearly independent displacement vectors (i.e. a basis) to span that plane; the Cartesian coordinate system is just one special case of this general scheme.  Similarly, one can convert a position on a plane to a vector in once one picks a basis for that plane as well as an origin , thus identifying that plane with the standard Euclidean plane .  (To put it another way, units of measurement are nothing more than one-dimensional (i.e. scalar) coordinate systems.)
5. To convert an angle in a plane to a signed number (modulo multiples of ), or vice versa, one needs to pick an orientation on the plane (e.g. to decide that anti-clockwise angles are positive).
6. To convert a direction in a plane to a signed number (again modulo multiples of ), or vice versa, one needs to pick an orientation on the plane, as well as a reference direction (e.g. true or magnetic north is often used in the case of ocean navigation).
7. Similarly, to convert a position on a circle to a number (modulo multiples of ), or vice versa, one needs to pick an orientation on that circle, together with an origin on that circle.  Such a coordinate system then equates the original circle to the standard unit circle (with the standard origin and the standard anticlockwise orientation ).
8. To convert a position on a two-dimensional sphere (e.g. the surface of the Earth, as a first approximation) to a point on the standard unit sphere , one can pick an orientation on that sphere, an “origin” (or “north pole”) for that sphere, and a “prime meridian” connecting the north pole to its antipode.  Alternatively, one can view this coordinate system as determining a pair of Euler angles (or a latitude and longitude) to be assigned to every point on one’s original sphere.
9. The above examples were all geometric in nature, but one can also consider “combinatorial” coordinate systems, which allow one to identify combinatorial objects with numerical ones.  An extremely familiar example of this is enumeration: one can identify a set A of (say) five elements with the numbers 1,2,3,4,5 simply by choosing an enumeration of the set A.  One can similarly enumerate other combinatorial objects (e.g. graphs, relations, trees, partial orders, etc.), and indeed this is done all the time in combinatorics.  Similarly for algebraic objects, such as cosets of a subgroup H (or more generally, torsors of a group G); one can identify such a coset with H itself by designating an element of that coset to be the “identity” or “origin”.

More generally, a coordinate system can be viewed as an isomorphism between a given geometric (or combinatorial) object A in some class (e.g. a circle), and a standard object G in that class (e.g. the standard unit circle).  (To be pedantic, this is what a global coordinate system is; a local coordinate system, such as the coordinate charts on a manifold, is an isomorphism between a local piece of a geometric or combinatorial object in a class, and a local piece of a standard object in that class.  I will restrict attention to global coordinate systems for this discussion.)

Coordinate systems identify geometric or combinatorial objects with numerical (or standard) ones, but in many cases, there is no natural (or canonical) choice of this identification; instead, one may be faced with a variety of coordinate systems, all equally valid.  One can of course just fix one such system once and for all, in which case there is no real harm in thinking of the geometric and numeric objects as being equivalent.  If however one plans to change from one system to the next (or to avoid using such systems altogether), then it becomes important to carefully distinguish these two types of objects, to avoid confusion.  For instance, if an interval AB is measured to have a length of 3 yards, then it is OK to write (identifying the geometric concept of length with the numeric concept of a positive real number) so long as you plan to stick to having the yard as the unit of length for the rest of one’s analysis.  But if one was also planning to use, say, feet, as a unit of length also, then to avoid confusing statements such as “ and “,  one should specify the coordinate systems explicitly, e.g. “ and “.  Similarly, identifying a point P in a plane with its coordinates (e.g. ) is safe as long as one intends to only use a single coordinate system throughout; but if one intends to change coordinates at some point (or to switch to a coordinate-free perspective) then one should be more careful, e.g. writing , or even , if the origin O and basis vectors of one’s coordinate systems might be subject to future change.

As mentioned above, it is possible to in many cases to dispense with coordinates altogether.  For instance, one can view the length of a line segment AB not as a number (which requires one to select a unit of length), but more abstractly as the equivalence class of all line segments CD that are congruent to AB.  With this perspective, no longer lies in the standard semigroup , but in a more abstract semigroup (the space of line segments quotiented by congruence), with addition now defined geometrically (by concatenation of intervals) rather than numerically.  A unit of length can now be viewed as just one of many different isomorphisms between and , but one can abandon the use of such units and just work with directly.  Many statements in Euclidean geometry involving length can be phrased in this manner.  For instance, if B lies in AC, then the statement can be stated in , and does not require any units to convert to ; with a bit more work, one can also make sense of such statements as for a right-angled triangle ABC (i.e. Pythagoras’ theorem) while avoiding units, by defining a symmetric bilinear product operation from the abstract semigroup of lengths to the abstract semigroup of areas.  (Indeed, this is basically how the ancient Greeks, who did not quite possess the modern real number system , viewed geometry, though of course without the assistance of such modern terminology as “semigroup” or “bilinear”.)

The above abstract coordinate-free perspective is equivalent to a more concrete coordinate-invariant perspective, in which we do allow the use of coordinates to convert all geometric quantities to numeric ones, but insist that every statement that we write down is invariant under changes of coordinates.  For instance, if we shrink our chosen unit of length by a factor , then the numerical length of every interval increases by a factor of , e.g. .  The coordinate-invariant approach to length measurement then treats lengths such as as numbers, but requires all statements involving such lengths to be invariant under the above scaling symmetry.  For instance, a statement such as is legitimate under this perspective, but a statement such as or is not.  [In other words, co-ordinate invariance here is the same thing as being dimensionally consistent.  Indeed, dimensional analysis is nothing more than the analysis of the scaling symmetries in one's coordinate systems.]  One can retain this coordinate-invariance symmetry throughout one’s arguments; or one can, at some point, choose to spend (or break) this coordinate invariance by selecting (or fixing) the coordinate system (which, in this case, means selecting a unit length).  The advantage in spending such a symmetry is that one can often normalise one or more quantities to equal a particularly nice value; for instance, if a length is appearing everywhere in one’s arguments, and one has carefully retained coordinate-invariance up until some key point, then it can be convenient to spend this invariance to normalise to equal 1.  (In this case, one only has a one-dimensional family of symmetries, and so can only normalise one quantity at a time; but when one’s symmetry group is larger, one can often normalise many more quantities at once; as a rule of thumb, one can normalise one quantity for each degree of freedom in the symmetry group.)  Conversely, if one has already spent the coordinate invariance, one can often buy it back by converting all the facts, hypotheses, and desired conclusions one currently possesses in the situation back to a coordinate-invariant formulation.  Thus one could imagine performing one normalisation to do one set of calculations, then undoing that normalisation to return to a coordinate-free perspective, doing some coordinate-free manipulations, and then performing a different normalisation to work on another part of the problem, and so forth.  (For instance, in Euclidean geometry problems, it is often convenient to temporarily assign one key point to be the origin (thus spending translation invariance symmetry), then another, then switch back to a translation-invariant perspective, and so forth.  As long as one is correctly accounting for what symmetries are being spent and bought at any given time, this can be a very powerful way of simplifying one’s calculations.)

Given a coordinate system that identifies some geometric object A with a standard object G, and some isomorphism of that standard object, we can obtain a new coordinate system of A by composing the two isomorphisms.  [I will be vague on what "isomorphism" means; one can formalise the concept using the language of category theory.] Conversely, every other coordinate system of arises in this manner.  Thus, the space of coordinate systems on A is (non-canonically) identifiable with the isomorphism group of G.  This isomorphism group is called the structure group (or gauge group) of the class of geometric objects.  For example, the structure group for lengths is ; the structure group for angles is ; the structure group for lines is the affine group ; the structure group for -dimensional Euclidean geometry is the Euclidean group ; the structure group for (oriented) 2-spheres is the (special) orthogonal group ; and so forth.  (Indeed, one can basically describe each of the classical geometries (Euclidean, affine, projective, spherical, hyperbolic, Minkowski, etc.) as a homogeneous space for its structure group, as per the Erlangen program.)

– Gauges –

In our discussion of coordinate systems, we focused on a single geometric (or combinatorial) object : a single line, a single circle, a single set, etc.  We then used a single coordinate system to identify that object with a standard representative of such an object.

Now let us consider the more general situation in which one has a family (or fibre bundle) of geometric (or combinatorial) objects (or fibres) : a family of lines (i.e. a line bundle), a family of circles (i.e. a circle bundle), a family of sets, etc.  This family is parameterised by some parameter set or base point x, which ranges in some parameter space or base space X.  In many cases one also requires some topological or differentiable compatibility between the various fibres; for instance, continuous (or smooth) variations of the base point should lead to continuous (or smooth) variations in the fibre.  For sake of discussion, however, let us gloss over these compatibility conditions.

In many cases, each individual fibre in a bundle , being a geometric object of a certain class, can be identified with a standard object in that class, by means of a separate coordinate system for each base point x.  The entire collection is then referred to as a (global) gauge or trivialisation for this bundle (provided that it is compatible with whatever topological or differentiable structures one has placed on the bundle, but never mind that for now).  Equivalently, a gauge is a bundle isomorphism from the original bundle to the trivial bundle , in which every fibre is the standard geometric object G.  (There are also local gauges, which only trivialise a portion of the bundle, but let’s ignore this distinction for now.)

Let’s give three concrete examples of bundles and gauges; one from differential geometry, one from dynamical systems, and one from combinatorics.

Example 1: the circle bundle of the sphere. Recall from the previous section that the space of directions in a plane (which can be viewed as the circle of unit vectors) can be identified with the standard circle after picking an orientation and a reference direction.  Now let us work not on the plane, but on a sphere, and specifically, on the surface X of the earth.  At each point x on this surface, there is a circle of directions that one can travel along the sphere from x; the collection of all such circles is then a circle bundle with base space X (known as the circle bundle; it could also be viewed as the sphere bundle, cosphere bundle, or orthonormal frame bundle of X). The structure group of this bundle is the circle group if one preserves orientation, or the semi-direct product otherwise.

Now suppose, at every point x on the earth X, the wind is blowing in some direction .  (This is not actually possible globally, thanks to the hairy ball theorem, but let’s ignore this technicality for now.)  Thus wind direction can be thought of as a collection of representatives from the fibres of the fibre bundle ; such a collection is known as a section of the fibre bundle (it is to bundles as the concept of a graph of a function is to the trivial bundle ).

At present, this section has not been represented in terms of numbers; instead, the wind direction is a collection of points on various different circles in the circle bundle SX.  But one can convert this section w into a collection of numbers (and more specifically, a function from X to ) by choosing a gauge for this circle bundle – in other words, by selecting an orientation and a reference direction for each point x on the surface of the Earth X.  For instance, one can pick the anticlockwise orientation and true north for every point x (ignore for now the problem that this is not defined at the north and south poles, and so is merely a local gauge rather than a global one), and then each wind direction can now be identified with a unit complex number (e.g. if the wind is blowing in the northwest direction at x).  Now that one has a numerical function u to play with, rather than a geometric object w, one can now use analytical tools (e.g. differentiation, integration, Fourier transforms, etc.) to analyse the wind direction if one desires.  But one should be aware that this function reflects the choice of gauge as well as the original object of study.  If one changes the gauge (e.g. by using magnetic north instead of true north), then the function u changes, even though the wind direction w is still the same.  If one does not want to spend the U(1) gauge symmetry, one would have to take care that all operations one performs on these functions are gauge-invariant; unfortunately, this restrictive requirement eliminates wide swathes of analytic tools (in particular, integration and the Fourier transform) and so one is often forced to break the gauge symmetry in order to use analysis.  The challenge is then to select the gauge that maximises the effectiveness of analytic methods. 

Example 2: circle extensions of a dynamical system. Recall (see e.g. my lecture notes) that a dynamical system is a pair X = (X,T), where X is a space and is an invertible map.  (One can also place additional topological or measure-theoretic structures on this system, as is done in those notes, but we will ignore these structures for this discussion.)  Given such a system, and given a cocycle (which, in this context, is simply a function from X to the unit circle), we can define the skew product of X and the unit circle , twisted by the cocycle , to be the Cartesian product with the shift ; this is easily seen to be another dynamical system.  (If one wishes to have a topological or measure-theoretic dynamical system, then will have to be continuous or measurable here, but let us ignore such issues for this discussion.)  Observe that there is a free action of the circle group on the skew product that commutes with the shift ; the quotient space of this action is isomorphic to X, thus leading to a factor map , which is of course just the projection map .  (An example is provided by the skew shift system, described in my lecture notes.)

Conversely, suppose that one had a dynamical system which had a free action commuting with the shift .  If we set to be the quotient space, we thus have a factor map , whose level sets are all isomorphic to the circle ; we call a circle extension of the dynamical system X.  We can thus view as a circle bundle with base space X, thus the level sets are now the fibres of the bundle, and the structure group is .  If one picks a gauge for this bundle, by choosing a reference point in the fibre for each base point x (thus in this context a gauge is the same thing as a section ; this is basically because this bundle is a principal bundle), then one can identify with a skew product by identifying the point with the point for all , and letting be the cocycle defined by the formula

One can check that this is indeed an isomorphism of dynamical systems; if all the various objects here are continuous (resp. measurable), then one also has an isomorphism of topological dynamical systems (resp. measure-preserving systems).  Thus we see that gauges allow us to write circle extensions as skew products.  However, more than one gauge is available for any given circle extension; two gauges , will give rise to two skew products , which are isomorphic but not identical.  Indeed, if we let be a rotation map that sends to , thus , then we see that the two cocycles and are related by the formula

.  (1)

Two cocycles that obey the above relation are called cohomologous; their skew products are isomorphic to each other.  An important general question in dynamical systems is to understand when two given cocycles are in fact cohomologous, for instance by introducing non-trivial cohomological invariants for such cocycles.

As an example of a circle extension, consider the sphere from Example 1, with a rotation shift T given by, say, rotating anti-clockwise by some given angle around the axis connecting the north and south poles.  This rotation also induces a rotation on the circle bundle , thus giving a circle extension of the original system .  One can then use a gauge to write this system as a skew product.  For instance, if one selects the gauge that chooses to be the true north direction at each point x (ignoring for now the fact that this is not defined at the two poles), then this system becomes the ordinary product of the original system X with the circle , with the cocycle being the trivial cocycle 0.  If we were however to use a different gauge, e.g. magnetic north instead of true north, one would obtain a different skew-product , where is some cocycle which is cohomologous to the trivial cocycle (except at the poles).  (A cocycle which is globally cohomologous to the trivial cocycle is known as a coboundary.  Not every cocycle is a coboundary, especially once one imposes topological or measure-theoretic structure, thanks to the presence of various topological or measure-theoretic invariants, such as degree.)

There was nothing terribly special about circles in this example; one can also define group extensions, or more generally homogeneous space extensions, of dynamical systems, and have a similar theory, although one has to take a little care with the order of operations when the structure group is non-abelian; see e.g. my lecture notes on isometric extensions.

Example 3: Orienting an undirected graph. The language of gauge theory is not often used in combinatorics, but nevertheless combinatorics does provide some simple discrete examples of bundles and gauges which can be useful in getting an intuitive grasp of the concept.  Consider for instance an undirected graph G = (V,E) of vertices and edges.  I will let X=E denote the space of edges (not the space of vertices)!.  Every edge can be oriented (or directed) in two different ways; let be the pair of directed edges of e arising in this manner.  Then is a fibre bundle with base space X and with each fibre isomorphic (in the category of sets) to the standard two-element set , with structure group .

A priori, there is no reason to prefer one orientation of an edge e over another, and so there is no canonical way to identify each fibre with the standard set .  Nevertheless, we can go ahead and arbitrary select a gauge for X by orienting the graph G.  This orientation assigns an oriented edge to each edge , thus creating a gauge (or section) of the bundle .  Once one selects such a gauge, we can now identify the fibre bundle with the trivial bundle by identifying the preferred oriented edge of each unoriented edge with , and the other oriented edge with .  In particular, any other orientation of the graph G can be expressed relative to this reference orientation as a function , which measures when the two orientations agree or disagree with each other.

Recall that every isomorphism of a standard geometric object G allowed one to transform a coordinate system on a geometric object A to another coordinate system .  We can generalise this observation to gauges: every family of isomorphisms on G allows one to transform a gauge to another gauge (again assuming that respects whatever topological or differentiable structure is present).  Such a collection is known as a gauge transformation.  For instance, in Example 1, one could rotate the reference direction at each point anti-clockwise by some angle ; this would cause the function to rotate to .   In Example 2, a gauge transformation is just a map (which may need to be continuous or measurable, depending on the structures one places on X); it rotates a point to , and it also transforms the cocycle by the formula (1).  In Example 3, a gauge transformation would be a map ; it rotates a point to .

Gauge transformations transform functions on the base X in many ways, but some things remain gauge-invariant.  For instance, in Example 1, the winding number of a function along a closed loop would not change under a gauge transformation (as long as no singularities in the gauge are created, moved, or destroyed, and the orientation is not reversed).  But such topological gauge-invariants are not the only gauge invariants of interest; there are important differential gauge-invariants which make gauge theory a crucial component of modern differential geometry and geometric PDE.  But to describe these, one needs an additional gauge-theoretic concept, namely that of a connection on a fibre bundle.

– Connections –

There are many essentially equivalent ways to introduce the concept of a connection; I will use the formulation based primarily on parallel transport, and on differentiation of sections.  To avoid some technical details I will work (somewhat non-rigorously) with infinitesimals such as dx.  (There are ways to make the use of infinitesimals rigorous, such as non-standard analysis, but this is not the focus of my post today.)

In single variable calculus, we learn that if we want to differentiate a function at some point x, then we need to compare the value f(x) of f at x with its value f(x+dx) at some infinitesimally close point x+dx, take the difference , and then divide by dx, taking limits as , if one does not like to use infinitesimals:

In several variable calculus, we learn several generalisations of this concept in which the domain and range of f to be multi-dimensional.  For instance, if is now a vector-valued function on some multi-dimensional domain (e.g. a manifold) X, and v is a tangent vector to X at some point x, we can define the directional derivative of f at x by comparing with for some infinitesimal dt, take the difference , divide by dt, and then take limits as :

.

[Strictly speaking, if X is not flat, then x+vdt is only defined up to an ambiguity of o(dt), but let us ignore this minor issue here, as it is not important in the limit.]  If f is sufficiently smooth (being continuously differentiable will do), the directional derivative is linear in v, thus for instance . One can also generalise the range of f to other multi-dimensional domains than ; the directional derivative then lives in a tangent space of that domain.

In all of the above examples, though, we were differentiating functions , thus each element in the base (or domain) gets mapped to an element in the same range Y.  However, in many geometrical situations we would like to differentiate sections instead of functions, thus f now maps each point in the base to an element of some fibre in a fibre bundle .  For instance, one might want to know how the wind direction changes as one moves x in some direction v; thus computing a directional derivative of w at x in direction v.  One can try to mimic the previous definitions in order to define this directional derivative.  For instance, one can move x along v by some infinitesimal amount dt, creating a nearby point , and then evaluate w at this point to obtain .  But here we hit a snag: we cannot directly compare with , because the former lives in the fibre while the latter lives in the fibre .

With a gauge, of course, we can identify all the fibres (and in particular, and ) with a common object G, in which case there is no difficulty comparing with .  But this would lead to a notion of derivative which is not gauge-invariant, known as the non-covariant or ordinary derivative in physics.

But there is another way to take a derivative, which does not require the full strength of a gauge (which identifies all fibres simultaneously together).  Indeed, in order to compute a derivative , one only needs to identify (or connect) two infinitesimally close fibres together: and .  In practice, these two fibres are already “within O(dt) of each other” in some sense, but suppose in fact that we have some means of identifying these two fibres together.  Then, we can pull back from to through to define the covariant derivative:

.

In order to retain the basic property that is linear in v, and to allow one to extend the infinitesimal identifications to non-infinitesimal identifications, we impose the property that the to be approximately transitive in that

(1)

for all x, dx, dx’, where the symbol indicates that the error between the two sides is o(|dx| + |dx’|).  [The precise nature of this error is actually rather important, being essentially the curvature of the connection at x in the directions , but let us ignore this for now.]  To oversimplify a little bit, any collection of infinitesimal maps obeying this property (and some technical regularity properties) is a connection.

[There are many other important ways to view connections, for instance the Christoffel symbol perspective that we will discuss a bit later.  Another approach is to focus on the differentiation operation rather than the identifications or , and in particular on the algebraic properties of this operation, such as linearity in v or derivation-type properties (in particular, obeying various variants of the Leibnitz rule).  This approach is particularly important in algebraic geometry, in which the notion of an infinitesimal or of a path may not always be obviously available, but we will not discuss it here.]

The way we have defined it, a connection is a means of identifying two infinitesimally close fibres of a fibre bundle .  But, thanks to (1), we can also identify two distant fibres , provided that we have a path from to , by concatenating the infinitesimal identifications by a non-commutative variant of a Riemann sum:

(2)

where ranges over partitions.  This gives us a parallel transport map identifying with , which in view of its Riemann sum definition, can be viewed as the “integral” of the connection along the curve .  This map does not depend on how one parametrises the path , but it can depend on the choice of path used to travel from x to y.

We illustrate these concepts using several examples, including the three examples introduced earlier.

Example 1 continued. (Circle bundle of the sphere) The geometry of the sphere X in Example 1 provides a natural connection on the circle bundle SX, the Levi-Civita connection , that lets one transport directions around the sphere in as “parallel” a manner as possible; the precise definition is a little technical (see e.g. my lecture notes for a brief description).  Suppose for instance one starts at some location x on the equator of the earth, and moves to the antipodal point y by a great semi-circle going through the north pole.  The parallel transport along this path will map the north direction at x to the south direction at y.  On the other hand, if we went from x to y by a great semi-circle going along the equator, then the north direction at x would be transported to the north direction at y.  Given a section u of this circle bundle, the quantity can be interpreted as the rate at which u rotates as one travels from x with velocity v.

Example 2 continued. (Circle extensions) In Example 2, we change the notion of “infinitesimally close” by declaring x and Tx to be infinitesimally close for any x in the base space X (and more generally, x and are non-infinitesimally close for any positive integer n, being connected by the path , and similarly for negative n).  A cocycle can then be viewed as defining a connection on the skew product , by setting (and also and to ensure compatibility with (1); to avoid notational ambiguities let us assume for sake of discussion that are always distinct from each other).  The non-infinitesimal connections are then given by the formula for positive n (with a similar formula for negative n).  Note that these iterated cocycles also describe the iterations of the shift , indeed .

Example 3 continued. (Oriented graphs) In Example 3, we declare two edges e, e’ in X to be “infinitesimally close” if they are adjacent.  Then there is a natural notion of parallel transport on the bundle ; given two adjacent edges , , we let be the isomorphism from to that maps to and to .  Any path of edges then gives rise to a connection identifying with .  For instance, the triangular path induces the identity map on , whereas the U-turn path induces the anti-identity map on .

Given an orientation of the graph G, one can “differentiate” at an edge in the direction to obtain a number , defined as +1 if the parallel transport from and preserves the orientations given by , and -1 otherwise.  This number of course depends on the choice of orientation.  But certain combinations of these numbers are independent of such a choice; for instance, given any closed path of edges in X, the “integral” is independent of the choice of orientation (indeed, it is equal to +1 if is the identity, and -1 if is the anti-identity. 

Example 4. (Monodromy)  One can interpret the monodromy maps of a covering space in the language of connections.  Suppose for instance that we have a covering space of a topological space X whose fibres are discrete; thus is a discrete fibre bundle over X.  The discreteness induces a natural connection on this space, which is given by the lifting map; in particular, if one integrates this connection on a closed loop based at some point x, one obtains the monodromy map of that loop at x.

Example 5. (Definite integrals) In view of the definition (2), it should not be surprising that the definite integral of a scalar function can be interpreted as an integral of a connection.  Indeed, set , and let be the trivial line bundle over X.  The function f induces a connection on this bundle by setting

The integral of this connection along is then just the operation of translation by in the real line.

Example 6. (Line integrals) One can generalise Example 5 to encompass line integrals in several variable calculus.  Indeed, if is an n-dimensional domain, then a vector field induces a connection on the trivial line bundle by setting

The integral of this connection along a curve is then just the operation of translation by the line integral in the real line.

Note that a gauge transformation in this context is just a vertical translation of the bundle by some potential function , which we will assume to be smooth for sake of discussion.  This transformation conjugates the connection to the connection .  Note that this is a conservative transformation: the integral of a connection along a closed loop is unchanged by gauge transformation.

Example 7. (ODE) A different way to generalise Example 5 can be obtained by using the fundamental theorem of calculus to interpret as the final value of the solution to the initial value problem

for the ordinary differential equation .  More generally, the solution u(b) to the initial value problem

for some taking values in some manifold Y, where is a function (let us take it to be Lipschitz, to avoid technical issues), can also be interpreted as the integral of a connection on the trivial vector space bundle , defined by the formula

Then will map to , this is nothing more than the Euler method for solving ODE.   Note that the method of integrating factors in solving ODE can be interpreted as an attempt to simplify the connection via a gauge transformation.  Indeed, it can be profitable to view the entire theory of connections as a multidimensional “variable-coefficient” generalisation of the theory of ODE. 

Once one selects a gauge, one can express a connection in terms of that gauge.  In the case of vector bundles (in which every fibre is a d-dimensional vector space for some fixed d), the covariant derivative of a section w of that bundle along some vector v emanating from x can be expressed in any given gauge by the formula

where we use the gauge to express w(x) as a vector , the indices are summed over the fibre dimensions (and summed over the base dimensions) as per the usual conventions, and the are the Christoffel symbols of this connection relative to this gauge.

One example of this, which models electromagnetism, is a connection on a complex line bundle in spacetime .  Such a bundle assigns a complex line (i.e. a one-dimensional complex vector space, and thus isomorphic to ) to every point in spacetime.  The structure group here is U(1) (strictly speaking, this means that we view the fibres as normed one-dimensional complex vector spaces, otherwise the structure group would be ). A gauge identifies V with the trivial complex line bundle , thus converting sections of this bundle into complex-valued functions .  A connection on V, when described in this gauge, can be given in terms of fields for ; the covariant derivative of a section in this gauge is then given by the formula

.

In the theory of electromagnetism, and are known (up to some normalising constants) as the electric potential and magnetic potential respectively.  Sections of V do not show up directly in Maxwell’s equations of electromagnetism, but appear in more complicated variants of these equations, such as the Maxwell-Klein-Gordon equation.

A gauge transformation of V is given by a map ; it transforms sections by the formula , and connections by the formula , or equivalently

.   (2)

In particular, the electromagnetic potential is not gauge invariant (which broadly corresponds to the concept of being nonphysical or nonmeasurable in physics), as gauge symmetry allows one to add an arbitrary gradient function to this potential.  However, the curvature tensor

of the connection is gauge-invariant, and physically measurable in electromagnetism; the components for of this field have a physical interpretation as the electric field, and the components for have a physical interpretation as the magnetic field.  (The curvature tensor can be interpreted as describing the parallel transport of infinitesimal rectangles; it measures how far off the connection is from being flat, which means that it can be (locally) “straightened” via some choice of gauge to be the trivial connection.  In nonabelian gauge theories, in which the structure group is more complicated than just the abelian group U(1), the curvature tensor is non-scalar, but remains gauge-invariant in a tensor sense (gauge transformations will transform the curvature as they would transform a tensor of the same rank).

Gauge theories can often be expressed succinctly in terms of a connection and its curvatures.  For instance, Maxwell’s equations in free space, which describes how electromagnetic radiation propagates in the presence of charges and currents (but no media other than vacuum), can be written (after normalising away some physical constants) as

where is the 4-current.  (Actually, this is only half of Maxwell’s equations, but the other half are a consequence of the interpretation (*) of the electromagnetic field as a curvature of a U(1) connection.  Thus this purely geometric interpretation of electromagnetism has some non-trivial physical implications, for instance ruling out the possibility of (classical) magnetic monopoles.)  If one generalises from complex line bundles to higher-dimensional vector bundles (with a larger structure group), one can then write down the (classical) Yang-Mills equation

which is the classical model for three of the four fundamental forces in physics: the electromagnetic, weak, and strong nuclear forces (with structure groups U(1), SU(2), and SU(3) respectively).  (The classical model for the fourth force, gravitation, is given by a somewhat different geometric equation, namely the Einstein equations , though this equation is also “gauge-invariant” in some sense.)

The gauge invariance (or gauge freedom) inherent in these equations complicates their analysis.  For instance, due to the gauge freedom (2), Maxwell’s equations, when viewed in terms of the electromagnetic potential , are ill-posed: specifying the initial value of this potential at time zero does not uniquely specify the future value of this potential (even if one also specifies any number of additional time derivatives of this potential at time zero), since one can use (2) with a gauge function U that is trivial at time zero but non-trivial at some future time to demonstrate the non-uniqueness.  Thus, in order to use standard PDE methods to solve these equations, it is necessary to first fix the gauge to a sufficient extent that it eliminates this sort of ambiguity.  If one were in a one-dimensional situation (as opposed to the four-dimensional situation of spacetime), with a trivial topology (i.e. the domain is a line rather than a circle), then it is possible to gauge transform the connection to be completely trivial, for reasons generalising both the fundamental theorem of calculus and the fundamental theorem of ODEs.  (Indeed, to trivialise a connection on a line , one can pick an arbitrary origin and gauge transform each point by .)  However, in higher dimensions, one cannot hope to completely trivialise a connection by gauge transforms (mainly because of the possibility of a non-zero curvature form); in general, one cannot hope to do much better than setting a single component of the connection to equal zero.  For instance, for Maxwell’s equations (or the Yang-Mills equations), one can trivialise the connection in the time direction, leading to the temporal gauge condition

.

This gauge is indeed useful for providing an easy proof of local existence for these equations, at least for smooth initial data.  But there are many other useful gauges also that one can fix; for instance one has the Lorenz gauge

which has the nice property of being Lorentz-invariant, and transforms the Maxwell or Yang-Mills equations into linear or nonlinear wave equations respectively.  Another important gauge is the Coulomb gauge

where i only ranges over spatial indices 1,2,3 rather than over spacetime indices 0,1,2,3.  This gauge has an elliptic variational formulation (Coulomb gauges are critical points of the functional ) and thus are expected to be “smaller” and “smoother” than many other gauges; this intuition can be borne out by standard elliptic theory (or Hodge theory, in the case of Maxwell’s equations).  In some cases, the correct selection of a gauge is crucial in order to establish basic properties of the underlying equation, such as local existence.  For instance, the simplest proof of local existence of the Einstein equations uses a harmonic gauge, which is analogous to the Lorenz gauge mentioned earlier; the simplest proof of local existence of Ricci flow uses a gauge of de Turck that is also related to harmonic maps (see e.g. my lecture notes); and in my own work on wave maps, a certain “caloric gauge” based on harmonic map heat flow is crucial (see e.g. this post of mine).  But in many situations, it is not yet fully understood whether the use of the correct choice of gauge is a mere technical convenience, or is more innate to the equation.  It is definitely conceivable, for instance, that a given gauge field equation is well-posed with one choice of gauge but ill-posed with another.  It would also be desirable to have a more gauge-invariant theory of PDEs that did not rely so heavily on gauge theory at all, but this seems to be rather difficult; many of our most powerful tools in PDE (for instance, the Fourier transform) are highly non-gauge-invariant, which makes it very inconvenient to try to analyse these equations in a purely gauge-invariant setting.

My lectures are devoted to the correspondence principle between finite dynamical systems and infinite dynamical systems, that allows one to convert certain statements about the former to logically equivalent statements about the latter.  (I will be vague here about what “dynamical system” means; very broadly, just about anything with a group action could qualify here.)  A little more specifically, the correspondence principle equates four types of results, which we informally describe as follows:

1. Local quantitative results for concrete finite systems.
2. Local qualitative results for concrete infinite systems.
3. Continuous quantitative results for abstract finite systems.
4. Continuous qualitative results for abstract infinite systems.

(The meaning of these terms should become clearer once we give some specific examples.)

There are many contexts in which this principle shows up (e.g. in Ramsey theory, recurrence theory, graph theory, group theory, etc.) but the basic ingredients are always the same.  Namely, the correspondence between Type 1 and Type 2 (or Type 3 and Type 4) arises from a weak sequential compactness property, which, roughly speaking asserts that given any sequence of (increasingly large) finite systems, there exists a subsequence of such systems which converges (in a suitably “weak” sense) to an infinite system.  [We will define these terms more precisely in concrete situations later.]  More informally, any “sufficiently large” finite system can be “approximated” in some weak sense by an infinite system.   (One can make this informal statement more rigorous using nonstandard analysis and/or ultrafilters, but we will not take such an approach here.)  Because of this, we obtain a correspondence principle: any qualitative statement about infinite systems (e.g. that a certain quantity is always strictly positive) is equivalent to a quantitative statement about sufficiently large finite systems (e.g. a certain quantity is always uniformly bounded from below).  This principle forms a crucial bridge between finitary (or quantitative) mathematics and infinitary (or qualitative) mathematics; in particular, by taking advantage of this principle, tools from one type of mathematics can be used to prove results in the other.  (See my previous post on soft analysis and hard analysis for further discussion.)

In addition to the use of compactness, the other key pillar of the correspondence principle is that of abstraction – the ability to generalise from a concrete system (on a very explicit space, e.g. the infinite cube ) to an more general abstract setting (e.g. an abstract dynamical system, measure-preserving system, group, etc.)  One of the reasons for doing this is that there are various maneuvres one can do in the abstract setting (e.g. passing from a system to a subsystem, a factor, or an extension, or by reasoning by analogy from other special systems that are different from the original concrete system) which can be quite difficult to execute or motivate if one stays within the confines of a single concrete setting.

We now turn to several specific examples of this principle in various contexts.  We begin with the more “combinatorial” or “non-ergodic theoretical” instances of this principle, in which there is no underlying probability measure involved; these situations are simpler than the ergodic-theoretic ones, but already illustrate many of the key features of this principle in action.

1.  The correspondence principle in Ramsey theory

We begin with the classical correspondence principle that connects Ramsey results about finite colourings, to Ramsey results about infinite colourings, or (equivalently) about the topological dynamics of covers of open sets.  We illustrate this by demonstrating the equivalence of three statements. The first two are as follows:

Theorem 1. (van der Waerden theorem, Type 2 formulation)  Suppose the integers are coloured by finitely many colours.  Then there exist arbitrarily long monochromatic arithmetic progressions.

Theorem 2. (van der Waerden theorem, Type 1 formulation) For every c and k there exists N such that whenever is coloured by c colours, there exists a monochromatic arithmetic progression of length k.

It is easy to see that Theorem 2 implies Theorem 1.  Conversely, to deduce Theorem 2 from Theorem 1 we use the correspondence principle as follows.  Assume for contradiction that Theorem 1 was true, but Theorem 2 was false. Untangling the quantifiers, this means that there exist positive integers k, c, a sequence going to infinity, and colourings of into c colours, none of which contain any monochromatic arithmetic progressions of length k.

By shifting the sets and redefining a little, we can replace by .  This sequence of colourings on the increasingly large sets .  One can now extract a subsequence of such colourings on finite sets of integers that converge pointwise or weakly to a colouring on the whole set of integers by the usual “Arzelà-Ascoli diagonalisation trick”.   Indeed, by passing to an initial subsequence (and using the infinite pigeonhole principle), one can ensure that all of these colourings eventually become a constant colour at 0; refining to another subsequence, we can ensure it is a constant colour at 1; then at -1, 2, -2, and so forth.  Taking a diagonal subsequence of these sequences, we obtain a final subsequence of finite colourings that converges pointwise to an infinite limit colouring.  By Theorem 1, this limit colouring contains an monochromatic arithmetic progression of length k.  Now note that the property of being monochromatic at this progression is a local one: one only needs to inspect the colour of finitely many of the integers in order to verify this property.  Because of this, this property of the infinite limit colouring will also be shared by the finite colourings that are sufficiently far along the converging sequence.  But we assumed at the very beginning that none of these finite colourings have a monochromatic arithmetic progression of length k, a contradiction, and the claim follows.

The above argument has all the basic ingredients of the correspondence principle in action: a proof by contradiction, use of weak compactness to extract an infinite limiting object, application of the infinitary result to that object, and checking that the conclusion of that result is sufficiently “finitary”, “local”, or “continuous” that it extends back to some of the finitary sequence, leading to the desired contradiction.  [It is essential that one manages to reduce to purely local properties before passing from the converging sequence to the limit, or vice versa, since non-local properties are usually not preserved by the limit. For instance, consider the colouring of which colours every integer between -N/2 and N/2 blue, and all the rest red.  Then this converges weakly to the all-blue colouring, and clearly the (non-local) property of containing at least one red element is not preserved by the limit.]

A key disadvantage of the use of the correspondence principle, though, is that it is quite difficult to extract specific quantitative bounds from any argument that uses this principle; for instance, while one can eventually “proof mine” the above argument (combined with some standard proof of Theorem 1) to eventually get a bound on N in terms of k and d, such a bound is extremely poor (of Ackermann function type).

Theorem 1 can be reformulated in a more abstract form:

Theorem 3. (van der Waerden theorem, Type 4 version)  Let X be a compact space, let be a homeomorphism, and let be an open cover of X.  Then for any k there exists a positive integer n and an open set in the cover such that is non-empty.

The deduction of Theorem 3 from Theorem 1 is easy, after using the compactness to refine the open cover to a finite subcover, picking a point in X, and then colouring each integer n by the index of the first open set that contains . The converse deduction of Theorem 1 from Theorem 3 is the one which shows the “dynamical” aspect of this theorem: we can encode a colouring of the integers as a point in the infinite product space , where is the unique class such that (indeed, one can think of this product space as the space of all c-colourings of the integers).  The infinite product space is compact with the product (or weak) topology used earlier, thus a sequence of colourings converge to a limit iff they converge locally (or pointwise).  This space also comes with the standard shift (corresponding to right shift on the space of colourings).  If we let X be the closure of the orbit , and let be the open cover , it is straightforward to show that Theorem 3 implies Theorem 1.

2.  The correspondence principle for finitely generated groups

The combinatorial correspondence used above for colourings can also be applied to other situations, such as that of finitely generated groups.  Recall that if G is a group generated by a finite set S, we say that G has polynomial growth if there exists constants K, d such that the ball of radius r (i.e. the set of words in S of length at most r) has cardinality at most .  Such groups were classified by a well-known theorem of Gromov:

Theorem 4. (Gromov’s theorem on polynomial growth, Type 4 version) Let G be a finitely generated group of polynomial growth.  Then G is virtually nilpotent (i.e. it has a finite index subgroup that is nilpotent).

As observed in Gromov’s original paper, this result is equivalent to a finitary version:

Theorem 5. (Gromov’s theorem on polynomial growth, Type 3)  For every integers s, K, d, there exists R such that any finitely generated group with s generators, such that has cardinality at most for all , is virtually nilpotent.

It is clear that Theorem 5 implies Theorem 4; for the converse implication, we use the correspondence principle.  We sketch the details as follows.  First, we make things more concrete (i.e. move from Type 3 and Type 4 to Type 1 and Type 2 respectively) by observing that every group G on s generators can be viewed as a quotient of the (nonabelian) free group on s generators by some normal subgroup .

Suppose for contradiction that Theorem 5 failed in this concrete setting; then there exists s, K, d, a sequence going to infinity, and a sequence of groups such that each obeys the volume condition for all .

The next step, as before, is to exploit weak sequential compactness and extract a subsequence of groups that “converge” to some limit , in the “weak” or “pointwise” sense that converges pointwise (or locally) to (much as with the convergence of colourings in the previous setting).  The Arzelà-Ascoli argument as before shows that we can find a subsequence of which do converge pointwise to some limit object ; one can check that the property of being a normal subgroup is sufficiently “local” that it is preserved under limits, thus is a normal subgroup of and so G is well-defined as a group.  (One way to view this convergence is that algebraic identity obeyed by the generators of G, is eventually obeyed by the groups sufficiently far along the convergent subsequence, and conversely.)

As volume growth is a local condition (involving only words of bounded length for any fixed r), we then easily conclude that G is of polynomial growth, and thus by Theorem 4 is virtually nilpotent.  Some nilpotent algebra reveals that every virtually nilpotent group is finitely presented, so in particular there are a finite list of relations among the generators which guarantee this virtual nilpotency property.  Such properties are local enough that they must then persist to groups sufficiently far along the subsequence, contradicting Theorem 5.

A slight modification of the above argument also reveals that the step and index of the nilpotent subgroup of G can be bounded by some constant depending only on K, d, s; this gives Theorem 5 meaningful content even when G is finite (in contrast to Theorem 4, which is trivial for finite groups).  On the other hand, no explicit bound for this constant (or for R) in terms of s, K, d is currently known, though presumably some such bound can eventually be extracted from the above argument and the existing proofs of Gromov’s theorem by proof mining techniques.

3.  The correspondence principle for dense sets of integers

Now we turn to the more “ergodic” variants of the correspondence principle, starting with the fundamental Furstenberg correspondence principle connecting combinatorial number theory with ergodic theory.  We will illustrate this with the classic example of Szemerédi’s theorem.

There are many finitary versions of Szemerédi’s theorem.  Here is one:

Theorem 6. (Szemerédi’s theorem, Type 1 version) Let and .  Then there exists a positive integer such that every subset A of with contains at least one k-term arithmetic progression.

The standard “Type 2″ formulation of this theorem is the assertion that any subset of the integers of positive upper density has arbitrarily long arithmetic progressions.  While this statement is indeed easily shown to be equivalent to Theorem 6, the Furstenberg correspondence principle instead connects this formulation to a rather different one, in which the deterministic infinite set is replaced by a random one. Recall that a random subset of integers is a random variable A taking values in the power set of the integers (or more formally, with a distribution that is a Borel probability measure on with the product topology), and so in particular the probabilities of any cylinder events such as

that involve only finitely many of the elements of A, are well-defined as numbers between 0 and 1.  The Carathéodory extension theorem (combined with some topological properties of ) shows, conversely, that any assignment of numbers between 0 and 1 to each cylinder set, which obeys various compatibility conditions such as

can be shown to give rise to a well-defined random set A.

We say that a random set A of integers is stationary if for every integer h, the shifted set A+h has the same probability distribution as A.  In terms of cylinder events, this is equivalent to a collection of assertions such as

and so forth.  One can then equate Theorem 6 with

Theorem 7. (Szemerédi’s theorem, Type 2 version) Let A be a stationary random infinite set of integers such that (which, by stationarity, implies that for all n), and let .  Then, with positive probability, A contains a k-term arithmetic progression for each k.

It is not difficult to show that Theorem 6 implies Theorem 7.  We briefly sketch the converse implication, which goes through the usual compactness-and-contradiction method.  Suppose for contradiction that Theorem 7 is true, but Theorem 6 fails.  Then we can find k and , a sequence of going to infinity, and sets with with no k-term arithmetic progressions.

We now need to extract a stationary random infinite set A of integers as a limit of the deterministic finite sets .  The way one does that is by randomly translating each of the .  More precisely, let denote the random finite set , where is chosen from uniformly at random.  The probability distribution of is a discrete probability measure on which is “almost stationary” in the sense that (say) has a distribution very close to ; for instance probabilities such as and can easily be seen to differ only by .  Also, the fact that equates to the assertion that