Analysis of the wave propagation in open metal-dielectric waveguides constitutes an important class of vector electromagnetic problems. In the case of hollow shielded waveguides (filled with homogeneous dielectric) the spectral parameter enters the equations and not the transmission conditions, ending up with an eigenvalue problem for a self-adjoint operator. However, a general setting for a metal-dielectric waveguide yields non-self-adjoint boundary eigenvalue problems for the systems of Helmholtz equations with piecewise constant coefficients, the transmission conditions, and the conditions at infinity containing the spectral parameter; the transmission conditions are stated on the discontinuity lines (surfaces) of the permittivity and the resulting problem becomes non-self-adjoint.