Model simulations are compared to the typically observed evolution of MeV electron fluxes during geomagnetic storms to investigate whether radial diffusion alone can account for the observed variability and to estimate the effect of electron lifetimes. We demonstrate that knowledge of lifetimes is crucial for understanding the radial structure of the storm-time radiation belts and their temporal evolution. Our model results suggest that outer zone lifetimes at 1 MeV are on the order of few days during quite-times and less than a day during storm-time conditions. Losses outside plasmasphere should be included in the modeling of electron fluxes since effective lifetimes are much shorter than that of plasmaspheric losses. Simulations with variable outer boundary conditions show that the depletion of the main phase relativistic electron fluxes at L ≤ 4 can not be explained only by variations in fluxes near geosynchronous orbit and require local lifetimes as short as 0.5 day. Radial diffusion alone is unable to account for either the gradual build up of relativistic electron fluxes or the maxima in phase space density near L = 4 - 5 observed during the recovery phase of many storms, which suggests that an additional local acceleration source is also required.