The behavior of high-energy electrons trapped in the Earth's Van Allen radiation belts has been extensively studied, through both experimental and theoretical techniques. While the evidence for whistler induced electron precipitation (WEP) from the radiation belts is overwhelming, and the mechanisms behind WEP are well understood, the overall significance of WEP on radiation belt loss rates has not been clear. In this paper we investigate the L-shell variation and significance of WEP-driven loss of Van Allen belt electrons by combining in situ measurements of electron precipitation, local WEP rates determined from Trimpi perturbations, and global lightning distributions. Our modeling suggests that long-term WEP driven losses are more significant than all other inner radiation belt loss processes for electron kinetic energies in the range ~50--150 keV in the L-shell range L = 2--2.4. These calculated lifetimes are comparable to the observed decay rates of artificially injected high-energy electrons. The upper energy limit of the WEP significance range increases with decreasing L to ~225 keV at L = 2. For electron energies above this range manmade VLF transmitters and plasmaspheric hiss should dominate over all other loss processes. However, as our lifetimes are based on rather conservative parameter estimates, these conclusions should represent the lower bounds for the energy ranges over which WEP losses are significant. For lower L-shells the coupling of lightning activity to the production of WEP events rapidly decreases, such that by L ~ 1.7 WEP will be unimportant in the overall loss processes.