Recent observations suggest a close relationship between the evolution of the solar magnetic field and high-energy cosmic ray modulations at 1 AU on timescales of ≳1 year. We investigate this relationship using the simple concept that changes in the solar magnetic field propagate from the Sun and cause a change in the radial diffusion coefficient, assumed to scale as some inverse power of the interplanetary magnetic field (IMF) magnitude (K ∝ B-n). Increases in the IMF cause a reduction in the cosmic ray density in the inner heliosphere. A continuous recovery process is also assumed to operate, represented by a timescale τ which physically is related to particle entry into the depleted regions of the heliosphere by drift and diffusion processes. We integrate numerically the spherically symmetric equation representing this process, and incorporate the observed variations of the parameters included in the equation. The concept is able to account for the variations in cosmic ray intensity at 1 AU during solar cycles 20--23 remarkably successfully using physically plausible values of n ~ 1--2. An important requirement is that recovery times are shorter (τ ~ 30 days for >2 GV cosmic rays observed by neutron monitors) during epochs when the polarity of the solar global magnetic field A > 0, than when A 0 epochs than along the heliospheric current sheet when A < 0. We also identify a period around solar maximum when recovery times are long, consistent with the disappearance of latitudinal intensity gradients observed by Ulysses approaching the maximum of solar cycle 23. This period commences when the axisymmetric component of the solar open flux reaches a minimum, essentially corresponding to the disappearance of the polar coronal holes, prior to solar maximum. Part of the energy dependence of the size of cosmic ray intensity variations can be accounted for by recovery times that decrease with increasing particle energy. Despite the simplicity of the concept, we suggest that it provides important insight into the relationships between variations of the solar magnetic field, interplanetary parameters, and cosmic ray modulation.