Between February 1992 and April 1998 the Ulysses spacecraft carried out the first survey of how the properties of the solar wind and heliospheric magnetic field vary with latitude during the declining and minimum phases of the solar cycle. In this paper we report on how the underlying direction of the heliospheric magnetic field varied through the various phases of the Ulysses first solar orbit. To a first approximation both the azimuth angle of the magnetic field with respect to the radial direction and the meridional (north-south) angle agree with the predictions of the simple Parker spiral model. However, there are a number of notable deviations. For example, at high southerly latitudes the most probable azimuth angle was found to be ~24¿ more tightly wound than expected, although the mean angle was less tightly wound than expected. In contrast, at high northerly latitudes, the most probable azimuth angle agreed with the Parker prediction, but the meridional angle showed a notably double peaked distribution. We discuss possible interpretations of these and other results in the context of recent ideas on the large-scale behavior of the heliospheric magnetic field. No evidence of a heliolongitude dependence of the underlying field, predicted by one model, is found. It is suggested that the presence of large-scale Alfv¿n waves in the high-latitude heliosphere may lead to double peak distributions of the magnetic field angles, consistent with evidence that longer time averaging removes some of the unexpected features noted in the results. Our analysis does not rule out that systematic deviations due to field line foot point motions could be present but suggests that their amplitude may be too low to be reliably detected in in situ heliospheric magnetic field data.