An understanding of the tectonic stress field is geologically important because it is the agent that preserves in the crust a memory of dynamical processes. In an effort to elucidate the origin of the present state of stress of the lithosphere we use a finite element model of the Earth's lithosphere to calculate stresses induced by mantle flow, crustal heterogeneity, and topography and compare these to observations of intraplate stresses as given by the World Stress Map. We explore two models of lithospheric heterogeneity, one based directly on seismic and other observational constraints (Crust 2.0), and another that assumes isostatic compensation. Mantle tractions are computed from two models of mantle density heterogeneity: a model based on the history of subduction of the last 180 Myr, which has proved successful at accurately reproducing the present-day geoid and Cenozoic plate velocities, and a model inferred from seismic tomography. We explore the effects of varying assumptions for the viscosity structure of the mantle, and the effects of lateral variations in viscosity in the form of weak plate boundaries. We find that a combined model that includes both mantle and lithospheric sources of stress yields the best match to the observed stress field (~60% variance reduction), although there are many regions where agreement between observed and predicted stresses is poor. The stress field produced by mantle tractions alone shows a greater degree of long-wavelength structure than is apparent in the stress observations but agrees very well with observations in some areas where radial mantle tractions are particularly strong such as in southeast Asia and the western Pacific. The stress field produced by lithospheric heterogeneity alone depends strongly on the assumed crustal model: Whereas the isostatically compensated model yields very poor agreement with observations, the model based on Crust 2.0 matches the observations about as well as mantle tractions alone and matches very well in certain areas where the influence of high topography is very important (e.g., Andes, East Africa). A possible interpretation of our results is that the stress field is significantly influenced by lateral variations in the viscosity of the mantle, which leads to variable amounts of decoupling between lithosphere and mantle, allowing the mantle signature to dominate in some areas and the crustal signature to dominate in others. The poor fit between the isostatically compensated crustal model and observations and the large differences between the two crustal models point toward the importance of dynamic topography and remaining uncertainties in crustal structure and rheology. We also consider the possibility that observations of stress from the shallow crust may not reflect the state of stress of the entire plate; stresses in the upper plate may be at least partially decoupled from broader-scale plate driving forces by lateral and vertical variations in lithospheric rheology.