Background and Objectives: Interpersonal violence is a social and public health problem globally, and though it is related to poor health outcomes across all genders, most research has been directed towards violence against women. As a result, the health consequences of men’s victimization may be underreported and unaddressed. The purpose of this study was to assess the relationship between interpersonal violence and the psychological health outcomes of self-reported stress, anxiety, and depression among men. Materials and Methods: The study used data from the male sample (n = 2597) of the 2018 Health on Equal Terms Survey conducted in Gävleborg County in East-Central Sweden. Regression analysis was carried out to study the relationship between interpersonal violence and self-reported stress, anxiety, and depression. Results: The bivariate analysis showed that there was a statistically significant association between interpersonal violence and self-reported stress (OR 2.35; CI 1.45–3.81), anxiety (OR 1.54; CI 1.06–2.25), and depression (OR 2.30; CI 1.48–3.57). Controlling for other variables in the multivariate analysis removed the statistically significant relationship and reduced the odds ratios for stress (OR 1.46; CI 0.57–3.74), anxiety (OR 0.86; 0.40–1.84), and depression (OR 1.40; CI 0.67–3.32) respectively. Conclusions: The study found that interpersonal violence among men was associated with stress, anxiety and depression which was largely explained by demographic, socioeconomic, and health/behavior-related factors. The findings suggest the need for longitudinal studies to assess causal links between male victimization and psychological health outcomes at the county level.