Repeatable, hysteretic loops in quasi-static loading measurements on rocks are well known; the fundamental processes responsible for them are not. The grain contact region is usually treated as the site of these processes, but there is little supporting experimental evidence. We have performed simultaneous neutron diffraction and quasi-static loading experiments on a selection of rocks to experimentally isolate the response of these contact regions. Neutron diffraction measures strain in the lattice planes of the bulk of the grain material, so differences between this strain and the macroscopic response yield information about grain contact behavior. We find the lattice responds linearly to stress in all cases, oblivious to the macroscopic unrecoverable strains, curvature, and hysteresis, localizing these effects to the contacts. Neutron diffraction shows that the more granular rocks appear to distribute stresses so that the same strain appears in all the grains, independent of crystallographic orientation.