During the past two decades there has been considerable discussion about the relative contribution of different factors to the temperature changes observed now over the past 142 years. Among these factors are the external factors of human (anthropogenic) activity, volcanoes and putative variations in the irradiance of the sun, and the internal factor of natural variability. Here, by using a simple climate/ocean model to simulate the observed temperature changes for different state-of-the-art radiative-forcing models, we present strong evidence that while the anthropogenic effect has steadily increased in size during the entire 20th century such that it presently is the dominant external forcing of the climate system, there is a residual factor at work within the climate system, whether a natural oscillation or something else as yet unknown. This has an important implication for our expectation of future temperature changes.