The evolution of a wide range of morphological and physiological characters in plants is believed to have been driven by interactions with insect herbivores. Yet, the effects of insect herbivory on population dynamics and evolutionary trajectories of putative defence traits in natural plant populations are little known. Leaf hairs, trichomes, can reduce herbivore damage, but may also contribute to tolerance against drought and UV-radiation. We conducted a four-year field experiment to determine whether insect herbivory reduces population growth and affects selection on trichome production in a natural population of the outcrossing, perennial herb Arabidopsis lyrata, which is polymorphic for trichome production and occurs in a glabrous and a trichome-producing form. In control plots, glabrous plants were more damaged by insect herbivores and had a lower population growth rate than trichome-producing plants due to a lower probability of remaining in the reproductive stage. Herbivore removal increased population growth rate, in particular in the glabrous morph. In plots from which insect herbivores had been removed, glabrous and trichome-producing plants had similar growth rates. Herbivore removal enhanced the growth rate of the glabrous morph mainly through increasing the probability of remaining in the reproductive stage, and in the trichome-producing morph mainly through increasing the probability for vegetative plants to enter the reproductive stage. The results show that insect herbivory may reduce population growth and result in selection for trichome production in A. lyrata. Spatial and temporal variation in insect herbivory could thus contribute to the maintenance of the polymorphism in trichome production.