The existence, spatial distribution, and style of volcanism on terrestrial planets is an expression of their internal dynamics and evolution. On Earth a physical link has been proposed between hot spots, regions with particularly persistent, localized, and high rates of volcanism, and underlying deep mantle plumes. Such mantle plumes are thought to be constructed of large spherical heads and narrow trailing conduits. This plume model has provided a way to interpret observable phenomena including the volcanological, petrological, and geochemical evolution of ocean island volcanoes, the relative motion of plates, continental breakup, global heat flow, and the Earth's magnetic field within the broader framework of the thermal history of our planet. Despite the plume model's utility the underlying dynamics giving rise to hot spots as long-lived stable features have remained elusive. Accordingly, in this review we combine results from new and published observational, analog, theoretical, and numerical studies to address two key questions: (1) Why might mantle plumes in the Earth have a head-tail structure? (2) How can mantle plumes and hot spots persist for large geological times? We show first that the characteristic head-tail structure of mantle plumes, which is a consequence of hot upwellings having a low viscosity, is likely a result of strong cooling of the mantle by large-scale stirring driven by plate tectonics. Second, we show that the head-tail structure of such plumes is a necessary but insufficient condition for their longevity. Third, we synthesize seismological, geodynamic, geomagnetic, and geochemical constraints on the structure and composition of the lowermost mantle to argue that the source regions for most deep mantle plumes contain dense, low-viscosity material within D″ composed of partial melt, outer core material, or a mixture of both (i.e., a dense layer). Fourth, using results from laboratory experiments on thermochemical convection and new theoretical scaling analyses, we argue that the longevity of mantle plumes in the Earth is a consequence of the interactions between plate tectonics, core cooling, and dense, low-viscosity material within D″. Conditions leading to Earth-like mantle plumes are highly specific and may thus be unique to our own planet. Furthermore, long-lived hot spots should not a priori be anticipated on other terrestrial planets and moons. Our analysis leads to self-consistent predictions for the longevity of mantle plumes, topography on the dense layer, and composition of ocean island basalts that are consistent with observations.