Like many words, "caustic" has a variety of meanings. All seem to go back to both Latin and Greek where the word refers to burning. In the context of this post it refers to the concentration of light that is caused by reflection or refraction for any curved surface. A glass sphere in the sun could burn the table it was resting on which gives the motivation for the use of the word "caustic". Most have us have seen this is the bottom of a cup or glass.
A similar thing happens when light bounces around in a raindrop. The diagram below shows a ray of light entering the water drop some distance away from the center, It hits at what we will is refracted call the incident angle i and refracted to a path at the refracted angle r. This refraction depends on the color of the light so the angles are a bit different for different colors. It then hits the back of the drop and reflects at the same angle r and travels to the front of the drop and exits at an angle i. The light isn't reflected straight back but it is deflected by an angle φ. It isn't too hard to see that φ=4*r-2*i.
So we get light coming out of the raindrop at an angle that depends on where it went in to the drop and the color of the light. But what about light that didn't enter at just the right spot? Here's an image that shows the paths of many different rays.
We can see that there is a maximum angle of deflection but it is hard to tell what is really going on. To get a better idea let's take a look from further away from the rain drop.
Now we can see that not only is there a maximum deflection angle but there is a concentration of rays around the maximum deflection. The next image shows the relative amount of light sent back from a raindrop taking the refraction and the geometry of the raindrop into consideration.
The distribution is very sharply peaked at the maximum possible deviation. Like the caustic on the bottom of the cup shown above most of the light is concentrated at the edge of the pattern. As mentioned above the amount of deviation depends on the color of the light so what we get it nearly all of the light at a particular color being deflected into that very sharp peak. Each color of light forms a ring as seen by the observer of all the raindrops that are in the right position and the result is a rainbow.