In this paper a unified computer-aided approach based on the vector finite element method is proposed to study the propagation of optical waves in dielectric nonlinear waveguides. Most of the existing methods are based on the scalar Helmholtz wave equation and cannot adequately describe the vectorial properties of fields in nonlinear optical structures. To take into account vectorial field properties and properties of the medium without limitations, a new nonlinear hybrid vector finite element method based on the vectorial wave equation is developed. Several numerical examples are given to demonstrate the applicability of this method. ¿ American Geophysical Union 1996