The heliopause, a surface separating the tenuous hot heliosheath flow and the dense, magnetized interstellar flow, is subject to instabilities of the Rayleigh-Taylor and Kelvin-Helmholtz types. The dynamic properties of this discontinuity is of considerable importance for understanding the neutral atom and cosmic ray filtration at the interface. Here we investigate the stability of the upwind segment of the heliopause in the presence of charge exchange collisions using both an analytic (dispersion relation) approach and a numerical model that includes the interstellar magnetic field. Linear analysis yields dispersion relations that admit imaginary solutions for a range of wave numbers, implying that the stagnation point on the heliopause is Rayleigh-Taylor unstable to small perturbations propagating parallel to the discontinuity surface. Effects of interstellar and heliosheath atoms are analyzed separately. We confirm our analytic results by performing time-dependent numerical simulations of the nonlinear development of this instability using a multifluid MHD-neutral approach. For typical solar wind and LISM conditions we obtain cyclical evolution of the upwind heliopause with a period of about 100 years. The amplitude of these oscillations is found to be dependent on the presence of hot heliosheath neutrals. We discuss the effect of a strong LISM magnetic field on the heliopause stability and possible implications of the obtained instability on the X-ray emission and cosmic ray transport in the outer heliosphere.