Convection in the Earth's outer core excites torsional oscillations in the fluid, which couple to rigid-body motion of the inner core and mantle. The torsional oscillations are detected as time variations of the magnetic field, whereas the motion of the mantle is observed as changes in the length of day. We develop a model for the motion of the core-mantle system using a Green's function for the impulse response to a localized source of excitation. The response to a source which is distributed in both space and time is obtained by convolving the Green's function with the appropriate source function. The Green's function is constructed by summing the normal modes of the system. We derive an orthogonality condition for the normal modes and use it to determine the coefficients of the normal mode expansion. Examples of the Green's function are presented for a variety of source locations. The predictions may be compared with observations to provide insights into the convective processes that excite the oscillations. It may also be possible to recover the physical properties that determine the period of the normal modes, including the structure of the magnetic field inside the outer core and the nature of coupling at the fluid boundaries. We outline a strategy for inverting the observations and identify potential difficulties due to the nonuniqueness of the inverse problem.