Barriers in cities, such as administrative boundaries, natural obstacles, railways or major roads are thought to induce segregation. However, the empirical knowledge about this phenomenon is limited. Here, we present a network science framework to assess barriers to urban mobility along their hierarchy, across residential areas and visited amenities. Using GPS mobility data, we construct a network of blocks from the sequence of individual stays in a major European city. A community detection algorithm allows us to partition this network into non-overlapping areas of dense mobility clusters, in which the effect of transportation hubs can be tuned with a parameter. We apply the Symmetric Area Difference index to quantify the overlap between these mobility clusters and the polygons of urban area separated by barriers. Reducing the effect of transportation hubs results in smaller scale mobility clusters that fit better to lower rank administrative or road barriers compared to their higher rank pairs. We find that characteristic urban barriers can replace each other in dividing mobility clusters of different scales. Next, we define the Barrier Crossing Ratio, the fraction of barrier crossings that link mobility clusters. An event study of bridge closure demonstrates that the Barrier Crossing Ratio can capture barrier effects even on small spatial scales. The decomposition of this indicator by origins and destinations suggests a significantly higher impact of barriers on those who live closer to the city center and smaller impact on visits to complex amenities. These results contribute to the ongoing discourse on urban segregation, emphasizing the importance of barriers to urban mobility in shaping interactions and mixing.