The propagation of monochromatic nonlinear symmetric hybrid waves in a cylindrical nonlinear dielectric waveguide is considered. The physical problem is reduced to solving a transmission eigenvalue problem for a system of ordinary differential equations. Spectral parameters of the problem are propagation constants of the waveguide. Numerical results are found with the modification of the shooting method. The method allows us to determine approximate eigenvalues with any prescribed accuracy. The comparison with known exact solutions (for particular values of parameters) are made. The approach described in this paper can be applied to other problems, e.g., to multilayered inhomogeneous waveguides.