Background: Sedentary behavior (SED) has generally increased during the COVID-19 pandemic in people working from home, and physical activity (PA) has therefore decreased. However, it is unclear whether the pandemic has affected office workers in different countries in the same way. We aimed to compare the 24-hour time-use compositions of physical behaviors between Brazilian and Swedish office workers at working and non-working days during the pandemic. Methods: Physical behaviors were monitored over 7 days using thigh-worn accelerometer in 73 Brazilian and 202 Swedish workers. Daily time-use compositions were exhaustively described in terms of SED in short (<30 min) and long (≥30 min) bouts, light PA (LPA), moderate-to-vigorous PA (MVPA), and time-in-bed (TIB). Following a compositional data analysis, isometric log-ratios (ilr) were calculated to express the ratio of TIB to time spent awake, SED (short and long bouts) relative to LPA and MVPA, SED in short relative to long bouts, and LPA relative to MVPA. We examined differences between countries using MANOVA, followed by univariate post-hoc tests of pairwise differences. Results: Both groups spent most of their time SED and in bed. On working days, Brazilian workers spent 294 min in SED in short bouts, 477 min in SED in long bouts, 157 min in LPA, 50 min in MVPA and 461 min TIB; Swedes spent 274, 365, 257, 86 and 458 min, respectively. During non-working days, results were 279, 359, 237, 61 and 504 min among Brazilians and 263, 251, 305, 93 and 529 min among Swedes.  Brazilians and Swedes differed significantly in the set of ilrs as a whole during working (p < 0.001) and non-working days (p < 0.001) and in all pairwise comparisons, except for the ratio of TIB to time spent awake during non-working days. Conclusions: During the COVID-19 pandemic Brazilian and Swedish office workers behaved differently. Whether this relates to restrictions being different or to differences even before the pandemic is not clear.