We studied the mechanics of compactant failure in four sandstones associated with a broad range of failure modes in the brittle-ductile transition. While Berea and Bentheim sandstones can fail by compaction localization, homogeneous cataclastic flow dominates failure modes in Adamswiller and Darley Dale sandstones at high effective pressures. We acquired new experimental data to complement previous studies, focusing on the strain hardening behavior in samples under drained conditions. The initial yield stresses were identified as the critical stresses at the onset of shear-enhanced compaction, subsequent yield stresses were considered to depend on hardening given by plastic volumetric strain. The yield stresses were described by elliptical yield caps in the stress space, and we compared the cap evolution with two constitutive models: the critical state model and the cap model. Bentheim sandstone showed the best agreement with both models to relatively large strains. Darley Dale sandstone showed the best agreement with the associated flow rule as prescribed by the normality condition, which is implicitly assumed in both constitutive models. Shear-enhanced compaction in Bentheim and Berea sandstones was appreciably more than that predicted for an associative flow rule, with the implication that a nonassociative model is necessary for capturing the inelastic and failure behavior of these sandstones over a broad range of effective pressures. With reference to the nonassociative model formulated by Rudnicki and Rice, bifurcation analysis would predict the transition of failure mode from shear band to compaction band and ultimately to cataclastic flow, in qualitative agreement with the experimental observations.