Generally speaking, the difference between a horse and a pony comes down to height: a horse is usually at least 14.2 hands (4 ft 10 in, 147 cm) tall at the area between the shoulders, while ponies are shorter than that. Besides the height distinction, ponies tend to have a set of physical characteristics that horses don't, and are generally very strong for their size. There are a number of exceptions though, both because of height variations within breeds and because of traditions about the names of certain breeds. Despite their differences, both animals can be used for riding, pulling carts, and farming, among other things.
Distinguishing Characteristics
Ponies generally have stocky frames, with short legs, big chests, and thick necks. They generally are much stronger, relative to size, than horses, and tend to be hardier and more resistant to cold weather. They also tend to be quite intelligent, and as a result can often be very stubborn.
Horses tend to have longer, rangier bodies, as well as narrower faces and necks. Their bone structure is generally lighter than that of ponies, and their hair is often thinner. Like ponies, they are generally intelligent, though they don't always have the same tendency to stubbornness. Both animals can be used for similar tasks, though sometimes specific breeds of each are better suited for particular tasks. For instance, Thoroughbred horses are good for racing, while Hackney ponies are good for harness sports, like pulling a cart.
Variations and Exceptions
There can be a lot of variation in both height and features in both horse and pony breeds, which can blur the distinction between the two. Horses can range from just under 14.2 hands to in excess of 20 hands (6 ft 8 in, about 2 m) in extreme cases. Similarly, there are some breeds of pony that sometimes exceed the 14.2 hand limit, like Connemara Ponies and Dale Ponies.
There are also some breeds of horses that are almost always less than 14.2 hands when they're fully grown. For example, Caspians horses rarely grow larger than 12 hands, and Icelandic horses rarely get bigger than 14 hands. Despite this, they're still considered to be horses because of their physical features and because of the tradition of describing them by this name. Likewise, the American Miniature Horse is almost never larger than 8 hands, but is still considered to be a true horse.
Adult vs Foal
It is important to recognize that the distinction between pony and horse is judged based on an animal's height when it is fully grown, and based on the characteristics of the breed as a whole. Young horses and ponies, called foals, are usually less than 14.2 hands, but if they are horses then they'll nearly always get bigger than that as they grow. People often confuse foals for ponies, but ponies are strong and functional, while foals are still growing, and can't be used for riding or other work.