Fluency assessment is part of many official language tests (e.g., TOEFL iBT) which evaluate non-native speakers’ language development. In order to operationalise the notion of fluency, the different factors that influence fluency judgments are to be disentangled. In a previous study the authors found a primary role in L2 fluency perception for pause characteristics of speech. Therefore, the present experiment was designed to zoom in on the contribution of silent pauses to fluency perception, both in native and non-native speech. Speech fragments from turns in simulated discussions were recorded and digitally manipulated. The manipulations resulted in three conditions: NoPauses (pauses >250ms excised); ShortPauses (pauses >250ms received an altered duration between 250-500ms); LongPauses (pauses >250ms received an altered duration between 750-1000ms). These manipulated native and non-native speech fragments were rated on oral fluency by untrained raters using a Latin Square design. Linear Mixed Models of the subjective fluency ratings demonstrated that i) non-native speech was rated as significantly less fluent than native speech, ii) the NoPauses condition was rated significantly more fluent than the other conditions, iii) the effect of silent pauses did not differ in L2 speech relative to L1 speech. Despite the clear difference in fluency level of native vs. non-native speakers, it is concluded that both the number and duration of silent pauses have an equally strong effect on fluency perception in native and non-native speech. These results suggest that, at least with respect to pauses, the notion of fluency is constant across native and non-native speech.