We consider propagation of surface TE waves in a circular metal-dielectric waveguide filled with nonlinear (Kerr nonlinearity) multilayered inhomogeneous medium. Analysis is reduced to solving a nonlinear transmission eigenvalue problem for an ordinary differential equation; eigenvalues of the problem correspond to propagation constants of the waveguide. For the numerical solution, a method is proposed based on solving an auxiliary Cauchy problem (a version of the shooting method). As a result of comprehensive numerical modeling, new propagation regimes are discovered.