The problem of diffraction of TE polarized EM waves by an open dielectric waveguide with a nonlinear inhomogeneous filling is under consideration. In such problems a nonlinear eigenvalue problem for a system of differential equations is arised. For the numerical solution a method based on the solution of the auxillary Cauchy problem is proposed. Numerical results are presented.