The Earth's gravity spectrum can be used as an observational constraint on geophysical modelling. Here we show how the spectrum up to degree l = 31 can be explained to a large part by viscous mantle flow in combination with a very simple model of random mantle density anomalies. Efforts to make the calculation more realistic by considering effects of thermal boundary layers, or using density anomalies based on tomography, or geodynamic modelling, tend to worsen the fit. Results are rather sensitive to assumptions on density anomalies in the upper thermal boundary layer. We suggest that, in combination with other observations, the shape of the Earth's gravity spectrum can serve to better constrain radial viscosity structure, density anomalies and flow in the Earth's mantle. Appropriate treatment of the lithosphere and of lateral viscosity variations will be the main challenges in modelling this spectrum.