Both linear and nonlinear interactions between oblique whistler, electrostatic, quasi-upper hybrid mode waves, and an electron beam are studied by linear analyses and electromagnetic particle simulations. In addition to a background cold plasma we assumed a hot electron beam drifting along a static magnetic field (B0). Growth rates of the oblique whistler, oblique electrostatic, and quasi-upper hybrid instabilities were first calculated. We found that there are four kinds of unstable mode waves for parallel and oblique propagations. They are the electromagnetic whistler mode wave (WW1), the electrostatic whistler mode wave (WW2), the electrostatic mode wave (ESW), and the quasi-upper hybrid mode wave (UHW). When the angle &thgr; between the wave vector k and B0 is small enough (≤10¿), the electrostatic instability is dominant compared with the whistler mode instability. When &thgr; is around 30¿, the growth rates of whistler (both electrostatic and electromagnetic) mode waves and electrostatic mode waves are of the same order. When &thgr; increases to 60¿, the WW2 mode will be the most unstable mode wave. For a very large &thgr;, (~80¿), the WW2 instability still has positive growth rates, and the UHW instability begins to have positive growth rates. Electromagnetic particle simulations were performed for parallel and three oblique cases, &thgr;=0¿, 30¿, 60¿, and 80¿. When &thgr;=0¿, whistler mode waves can hardly grow from the thermal fluctuation level because the electron beam which is supposed to provide free energy to the whistler mode waves is quickly diffused in the velocity space by much faster growing ESW.

The ESW can lead to a secondary electrostatic instability. With &thgr;=30¿, both electrostatic and whistler mode waves grow simultaneously. Also, the electrons diffused by the whistler mode instability to higher v∥ velocity regions can lead to a secondary electrostatic instability. When &thgr; is 60¿, diffusion of the electron beam is controlled mainly by the WW2 instability. For &thgr;=80¿, both WW2 and UHW grow despite their small growth rates. The simulations agree with linear analyses on the ESW growth rate for &thgr;=0¿, but from our simulation data, growth rates of oblique whistler mode, electrostatic mode, and quasi-upper hybrid mode waves are usually smaller than those predicted by linear analyses. The electrostatic and whistler mode instabilities affect each other through their interactions with the electron beam. While the most intense ESW is generated for parallel propagation, the most intense whistler mode wave is observed at an oblique direction. Modulations between different electrostatic waves are found for &thgr;=0¿ after the electric field reaches its saturation level. Modulations between whistler waves (&thgr;=60¿, 80¿) are also found while their magnetic fields increase nonlinearly and reach their saturation levels. A possible mechanism is proposed to explain the satellite observations of whistler mode chorus and accompanied electrostatic waves, whose amplitudes are sometimes modulated at the chorus frequency. ¿ American Geophysical Union 1993