Infrared spectral observations have yielded many important insights into the nature of Mars. However, spectral effects tied to surface roughness and particle size can make mineral deposits substantially more difficult to detect and identify than is generally known, including for current investigations. The effects hold for Mars and other planets, but here we focus on the impact on the exploration of Mars. Laboratory spectra of pure, smooth-surfaced minerals have previously been used to conclude that the 1996 Global Surveyor Thermal Emission Spectrometer (TES) and the 2001 Mars Odyssey Thermal Emission Imaging System (THEMIS) will detect a mineral deposit that covers ~10% of a pixel. Those laboratory spectra have formed the foundation for modeling and interpretations of TES/THEMIS data and for development of the 2003 Mars rover MiniTES. Here, for the first time, we compare TES/THEMIS foundation data to laboratory, field, and unique airborne spectra of rough surfaces. We demonstrate that substantial, well-crystalline, regional mineral deposits, including rock outcrops, can be 100% exposed and remain undetected at the sensitivity of TES/THEMIS. We then detail the physics, which apply to all geologic materials. Previous studies have demonstrated the complicating physical effects for solids versus particulates for a range of minerals, including oxides, sulfates, carbonates, and each of the seven silicate classes. We identify three cases of spectral behavior that can impact interpretations: In Case 1, physical effects cause a target to be undetectable by a given infrared instrument (infrared stealthy). In Case 2, variations in spectral contrast mimic abundance variations, which confounds abundance mapping. In Case 3, alteration of the spectral band shape impacts compositional interpretations. Optically smooth materials will be well represented in TES/THEMIS/MiniTES identifications. Conversely, if a material forms an optically rough surface or sufficiently small particles on Mars, then it will be comparatively underrepresented or missed. Surface roughness occurs on three broad scales, and roughness at the grain scale can be a primary variable in whether a mineral is detectable. Nondetection of certain key minerals (e.g., carbonates) can significantly impact geologic and climatic interpretations and landing site selection. The TES/THEMIS detection threshold may be 10% coverage on the basis of specific laboratory samples, but for field measurements, that prediction represents an idealized lower bound. Interpretations built on nondetection should be revisited after defining the uncertainties in detection capabilities for specific instruments as applied to expected mineral surface textures on Mars, including at the grain scale. The impact on future measurement strategies (including MiniTES) should also be assessed.