We have derived quantitative color estimates of the Martian sky from data acquired by the Panoramic Cameras (Pancams) on the Mars Exploration Rovers Spirit and Opportunity. We calculate the perceptual color of the sky directly from the absolute radiometric calibration of the cameras, following similar approaches to those used in previous studies with Viking Lander and Mars Pathfinder data. We further use these measurements to study changes in sky color throughout the MER missions and to compare these to changes in atmospheric opacity determined from direct solar imaging by the Pancams. We have derived a functional relationship between sky color and optical depth and discuss its possible uses and limitations. Finally, we simulate changes in sky color as suspended dust is removed and present visual representations of these based on modeling results, past studies, and observed MER sky brightnesses. The color of the Martian sky as opacity decreases from 1.0 to 0.0 is estimated to change from "dark yellowish brown" at high opacity to "bluish-black" or "black" if dust-free.