The Kelvin-Helmholtz instability in the magnetopause-boundary layer region is studied on the basis of an idealized model which consists of three uniform plasma regions: the magnetosheath, the boundary layer, and the magnetosphere. There are two unstable modes in the magnetopause-boundary layer region: one is excited at the magnetopause (the magnetopause mode) and the other is excited at the inner surface of the boundary layer (the inner mode). The inner mode is found to be unstable most of the time, while the excitation of the magnetopause mode depends on the magnetic field in the magnetosheath. The observed variation of the boundary layer thickness can be attributed to the unstable inner mode. Possible relationships between the Pc 3-5 geomagnetic pulsations and the surface waves excited on the magnetospheric boundary are also discussed.