Tolerating others' opinions, even if disliked, is a cornerstone of liberal democracy. At the same time, there are limits to political tolerance as tolerating extremists and groups who use violence would threaten the foundations of tolerance itself. We study people's willingness to set limits to tolerance in case of violence and extremism (scope of tolerance)—under different conditions regarding ideological groups (left-wing, right-wing, religious) and offline/online contexts of free speech. Using data from a large-scale survey experiment conducted in Germany, we show that citizens clearly set limits to tolerance of different groups, especially if the latter have violent intentions, and that people tend to be more tolerant online than offline. Moreover, we find that citizens are more tolerant toward groups that are closer to their own ideological stance. However, violence disrupts such an ideological bias as respondents across the ideological spectrum exhibit low levels of tolerance toward violent groups—irrespectively of their political stance. Our findings highlight the importance of situational factors as foundations of judgments on the limits to tolerance.