Torsional oscillations of the fluid core produce decadal variations in the Earth's magnetic field and in the length of day. The magnetic field variations can be used to infer the fluid velocity of the oscillations. An additional constraint is imposed by the length-of-day variations because coupling mechanisms are required to transfer angular momentum between the core and mantle. Proposed mechanisms included electromagnetic, topographic, and gravitational couplings. We make use of a finite volume model of the core-mantle system to evaluate each of these mechanisms with respect to observed spectral properties of torsional oscillations. We find that the different coupling mechanisms have different spectral characters which may be useful in distinguishing their importance within the Earth's core-mantle system. For reasonable values of core and mantle properties, gravitational coupling between the mantle and inner core is the candidate mechanism most likely to be able to reproduce all spectral features of the observed transfer of angular momentum between the core and mantle. Possible spatial distributions of the torsional oscillation excitation source are also investigated.