We study, in the setting of algebraic varieties, finite-dimensional spaces of functions $V$ that are invariant under a ring $\Dc^V$ of differential operators, and give conditions under which $\Dc^V$ acts irreducibly. We show how this problem, originally formulated in physics \cite{kamran-Milson-Olver:invariant,turbiner:bochner}, is related to the study of principal parts bundles and Weierstrass points \cite{EGA4,laksov-thorup}, including a detailed study of Taylor expansions. Under some conditions it is possible to obtain $V$ and $\Dc^V$ as global sections of a line bundle and its ring of differential operators. We show that several of the published examples of $\Dc^V$ are of this type, and that there are many more - in particular, arising from toric varieties.