The propagation of hybrid waves in a cylindrical anisotropic nonlinear metal-dielectric waveguide is considered. The physical setting is reduced to a transmission eigenvalue problem for a system of ordinary differential equations which is new type of nonlinear eigenvalue problem where spectral parameters are the wave propagation constants. For the numerical solution, a method is proposed based on solving an auxiliary Cauchy problem (a version of the shooting method). As a result of comprehensive numerical modeling, new propagation regimes are discovered.