The three-dimensional (3-D) structure of the heliosphere is investigated using in situ and remote sensing measurements. The 3-D structure of both neutral interplanetary gas and solar wind ions are affected by the solar latitude variation of the solar radiation fields (solar wind and solar EUV). Neutral hydrogen atom lifetimes against charge-exchange with solar wind protons in the 3-D heliosphere are calculated from measurements of the solar wind proton velocity, density, and mass flux by Ulysses SWOOPS (Solar Wind Observations Over the Poles of the Sun) from 1990--2001. These are compared to in-ecliptic H atom lifetimes derived from solar wind measurements by the spacecraft IMP (Interplanetary Monitoring Platform) 6, 7, and 8, WIND, and ACE (Advanced Composition Explorer) SWEPAM (Solar Wind Electron Proton Alpha Monitor). Recent observations during the Ulysses rapid solar pole-to-solar pole passage (fast-latitude scan) at solar maximum find a more isotropic rotationally averaged solar wind mass flux (and H atom lifetime) than was found during the previous fast latitude scan at solar minimum. During the solar minimum fast-latitude scan the 27-day averaged SWOOPS lifetime passed through two distinct regimes: it doubled from 2 ¿ 106 s at low latitudes to 4 ¿ 106 s at high latitudes in both hemispheres. During the solar maximum pass the 27-day averaged SWOOPS H atom lifetime (corrected to 1 AU) at all southern latitudes and at all northern latitudes below 60¿ was (2--2.5) ¿ 106 s, while at the very end of the pass, above 60¿ N latitude, it rose to 4 ¿ 106 s as polar coronal holes reformed. Remote sensing studies of interplanetary H Lyman-α emission, white light solar coronal observations, and radio scintillation experiments have also indicated that the time-averaged solar wind mass flux (and H atom lifetime) is more isotropic at solar maximum than at solar minimum. This enlarges the polar H cavity at solar maximum. Results from in situ measurements are compared to remote sensing data. Ulysses GAS Lyman- α maps are modeled, with updated results. Comparisons of the H atom charge exchange loss rate with a weaker H atom loss process, photoionization, are made using the SOLAR2000 model <Tobiska et al., 2000> for in-ecliptic solar Extreme UltraViolet (EUV) fluxes. The relative latitude invariance of the H atom lifetime at solar maximum is related to the absence of high-speed solar wind at solar maximum and to the large inclination of the heliospheric current sheet.