Cluster observations in the near-Earth plasma sheet boundary layer (PSBL) region have shown the presence of ion shell distributions related to velocity-dispersed ion structures (VDIS) coincident with electrostatic emissions. We have examined ion shell instabilities in the presence of a cold ion and electron background using linear theory and particle in cell simulations. Linear theory shows that the shell instability is only excited when a cold ion background is present, generating a broad range of ion cyclotron harmonics. Numerical simulations confirm that ion Bernstein modes are preferentially excited transverse to the ambient magnetic field, along with a lower level of wave power at oblique angles. The background ions are heated primarily in the transverse direction because of a linear nonstochastic ion cyclotron heating mechanism, and overall saturation of the instability occurs because of thermalization of the shell combined with heating of the background ions and electrons. Comparison with Cluster observations shows that observed electrostatic waves with a spectrum from a few Hz to several hundred Hz is in good agreement with that expected from the shell instability. The cold background ions are observed to have a temperature anisotropy with T$perp$ > T$parallel$, while electrons are observed to have T$parallel$ > T$perp$, also in qualitative agreement with the shell instability. The wave-particle effects due to the shell instability in the near-Earth PSBL could have important consequences for auroral potential structure at lower altitudes and may cause the gaps in VDIS structure leading to beamlets.