Not long after Newton published his universal law of gravitation -- the law that describes how two masses attract each other -- scientists started to wonder what happens when you have three bodies interacting. However, as mathematicians soon discovered, the problem turned out to be much harder than expected. Even two of the eighteenth century's mathematical giants -- Euler and Lagrange -- only managed to produce solutions for very special cases (when three bodies lie on a rotating straight line, or form the vertices of a rotating equilateral triangle, respectively). A full century passed before the astronomer Charles-Eugène Delaunay produced an extremely long, very slowly converging, approximate solution to the motions of the Sun-Earth-Moon system.
The next significant step forward was taken by the great mathematical physicist Henri Poincaré (Figure 1). In 1890, he showed (after correcting an initial error) that even what became known as the "restricted three-body problem" was far more intricate than previously envisaged. In the restricted three-body problem, one of the bodies is assumed to have such a tiny mass that it doesn't influence the motions of the two other bodies. It would be as if we solve for the motion of a grain of sand, under the combined influence of the Earth's and Moon's gravity. What Poincaré discovered is what is known today as chaotic behavior -- solutions in which even the slightest change in the initial conditions produces an entirely different end result. This discovery had dramatic implications, since it demonstrated that even systems in which the laws precisely predict the future behavior ("deterministic") can exhibit random characteristics. Today's "chaos theory" was born from these concepts.
Figure 1. The French mathematical physicist Henri Poincaré. From: Wikipedia
In 1993, American computer scientist Cris Moore showed numerically that three equal-mass bodies can follow each other without colliding along a figure-eight-shaped curve. His results were given a firm mathematical basis by Alain Chenciner and Richard Montgomery in 2000, and the orbit was also shown to be stable (except perhaps for an extremely slow drift) by Catalan mathematician Carles Simó.
Intriguingly, Simó's simulations (and subsequent ones) showed that even when the three masses are not precisely equal, the figure-eight orbit is still stable. This raised the question: Could such a triple system of stars really exist in the universe? In an attempt to provide an answer, in 2000 astrophysicist Douglas Heggie performed simulations of binary-binary scattering -- studies of the dynamical behavior of two binaries that approach from a large distance and interact for a while. He found that the probability of formation of such an orbit is a very small fraction of one percent. So even if such systems exist, our chances of detecting them are extremely low.
Many questions related to the three-body problem, and even more questions related to the general n-body problem, remain open. In particular, we don't even know with certainty whether or not our own solar system is stable in the very long run.