Speech is a continuous stream. Listeners can only make sense of speech by identifying the components that comprise it - words. Segmenting speech into words is an operation which has to be learned very early, since it is how infants compile even their initial vocabulary. Evidence from new behavioural and electrophysiological studies of infant speech perception illustrates this learning process. Infants’ relative success at achieving speech segmentation in fact turns out to be a direct predictor of language skills during later development. Adult listeners segment speech so efficiently, however, that they are virtually never aware of the operation of segmentation. In part they achieve this level of efficiency by exploiting accrued knowledge of relevant structure in the native language. Amassing this language-specific knowledge also starts in infancy. However, some relevant features call on more advanced levels of language processing ability; the continuous refinement of segmentation efficiency is apparent in that (as revealed by adult listening studies across a dozen or so languages) these structural features are exploited for segmentation too, even if applying them means overturning constraints used, perhaps universally, by infants.