three-body | WebVR story experience based on the book Three Body Problem | Augmented Reality library

I see you're trying to do the calculations using the real physical values. The problem with that is that the astronomical masses, distances and forces are astronomical, not just literally but also figuratively. The numbers are so big that the results of the intermediate calculations cannot be represented using the limited range of the float data type.

The solution is to change the units. For example, instead of using meters, kilograms, and seconds, measure the distance in "astronomical units" (distance between Earth and Sun), mass in multiples of Earth masses, and time in years. Or use the standard "astronomical system of units". Most importantly, this changes the value of the gravitational constant G.

If you're not too attached to making a simulation that lets you predict the real positions of the planets, it's much easier to just "wing it" and set G=1, set the masses to small numbers, and play around with the velocities. You'll see it's very easy to come up with a realistic simulation.