Measurements from various instruments and analysis techniques are used to directly compare changes in Earth-atmosphere shortwave (SW) top-of-atmosphere (TOA) radiation between 2000 and 2005. Included in the comparison are estimates of TOA reflectance variability from published ground-based Earthshine observations and from new satellite-based CERES, MODIS and ISCCP results. The ground-based Earthshine data show an order-of-magnitude more variability in annual mean SW TOA flux than either CERES or ISCCP, while ISCCP and CERES SW TOA flux variability is consistent to 40%. Most of the variability in CERES TOA flux is shown to be dominated by variations global cloud fraction, as observed using coincident CERES and MODIS data. Idealized Earthshine simulations of TOA SW radiation variability for a lunar-based observer show far less variability than the ground-based Earthshine observations, but are still a factor of 4--5 times more variable than global CERES SW TOA flux results. Furthermore, while CERES global albedos exhibit a well-defined seasonal cycle each year, the seasonal cycle in the lunar Earthshine reflectance simulations is highly variable and out-of-phase from one year to the next. Radiative transfer model (RTM) approaches that use imager cloud and aerosol retrievals reproduce most of the change in SW TOA radiation observed in broadband CERES data. However, assumptions used to represent the spectral properties of the atmosphere, clouds, aerosols and surface in the RTM calculations can introduce significant uncertainties in annual mean changes in regional and global SW TOA flux.