This article presents a theoretical framework for analysing the findings derived from the Diversity Barometer, a longitudinal study that collected data on immigration and diversity-related topics in Swedish society from 2005 to 2022. This article examines attitudes towards immigrants and migration in Sweden and how the historical context of Swedish race relations and structural racism shapes these attitudes. Specifically, the article focuses on attitudes regarding immigrants’ social rights and responsibilities, workplace diversity, cultural diversity, and spatial segregation. Our findings and analysis demonstrate how structural racism manifests itself through attitudes towards individuals with immigrant backgrounds in Sweden. Research has also revealed the pervasive nature of racial discrimination across various aspects of Swedish society, contributing to a divided society based on racial markers. However, the findings also show complexity in how Swedes’ attitudes toward immigrants are to be understood when they do not follow a simple horizontal division between Swedes and immigrants. Instead, it appears as an intricate vertical division where immigrants, their culture and their labour are accepted, albeit in a subordinate manner.