We consider the hypothesis that Mars never experienced an early warm, wet period and the implications of a continuously cold Martian climate for the hydrology and geochemistry of the planet. Geomorphic evidence previously interpreted to indicate a more temperate early climate can be explained by cold climate processes and the transient flow of liquid water at the surface rather than a terrestrial-like hydrological cycle. In this scenario, freezing and confinement of crustal aquifers lead to the eruption of water or brines to the surface: Hesperian events formed massive ice sheets, and triggered floods that carved the outflow channels, while smaller, present-day aqueous eruptions are responsible for the seepage and gully-like landforms identified in high-resolution imaging. This process may have fluxed a significant volume of groundwater to the surface and the polar caps. A primordial source of groundwater reconciles the present D/H isotopic ratio of the atmosphere with fractionating atmospheric escape and a primordial Martian D/H inferred from hydrous phases of SNC meteorites. A cold early climate and a thin CO2 atmosphere are consistent with the ubiquity of primary igneous minerals and the apparent absence of secondary minerals and copious carbonates on the surface. Aqueous eruptions introduced soluble chloride and sulfate salts into Martian soils, leaving clays and carbonates within the crust. If the Martian crust hosts a deep biosphere, aqueous eruptions can bring organisms or their chemical signature to the surface. The biotic component of any recent eruptions may be preserved in transient ice in cold traps on the surface.