For one, contactors are held closed electrically. Latching relays are held closed either mechanically or with a permanent magnet and must be unlatched by a separate signal. There are latching contactors however, which brings us to:
Contactors are rated to switch high-induction loads - meaning the contacts are designed to break a significant amount of current and survive the resulting arc when the energy stored in the magnetic field of the inductor collapses. Relays are generally not rated for high induction loads, and relays are generally limited to lower currents. Relays are used in control circuits. Contactors are used in power switching.