Radar imaging of ocean scenes requires knowledge of the modulation of the radar backscatter by long ocean waves. By using the conventional composite surface theory, the modulation of the radar backscatter can be related to a single parameter of the ocean wave field, such as slope or orbital velocity of the long waves. Usually, this relationship is assumed to be linear and described by a linear modulation transfer function (MTF). Consequently, most investigations of the modulation of the ocean radar backscatter concentrate on the calculation of this linear MTF. However, the coherence between the variations of the backscattered radar power and those of the orbital velocity associated with the long waves is generally low at C and X bands. In this paper we make the hypothesis that the low coherence is due to a nonlinear relationship between the radar backscatter and the long ocean waves (rather than to the neglect of other possible modulation sources). In order to quantify this nonlinearity, we estimate linear and quadratic MTFs from time series of the backscattered radar power and the Doppler shift of the backscattered radar signal. The latter is directly related to the orbital velocity of the long waves. The data were acquired by a scatterometer mounted on a sea-based platform in the North Sea. We define the coherence as the fraction of the output spectrum that can be related to the input spectrum by a measured transfer function, which can be either linear or linear and quadratic. We find that the coherence between the backscattered radar power and the Doppler shift can be increased by up to 0.14 by using linear and quadratic MTFs instead of a linear one. However, at X band the coherence never becomes larger than 0.5. Thus we conclude that the modulation process cannot be described fully by a quadratically nonlinear system which relates the radar backscatter to the orbital velocity of the long waves. Therefore a higher order of nonlinearity has to be considered. ¿ 1998 American Geophysical Union