On the variety of invariant subspaces of a finite-dimensional linear operator

Abstract

If V V is a finite-dimensional vector space over R \mathbf {R} or C \mathbf {C} and A ∈ Hom ( V ) A \in {\operatorname {Hom}}(V) , the set S A ( k ) {S_A}(k) of k k -dimensional A A -invariant subspaces is a compact subvariety of the Grassmann manifold G k ( V ) {G^k}(V) , but it need not be a Schubert variety. We study the topology of S A ( k ) {S_A}(k) . We reduce to the case where A A is nilpotent. In this case we prove that S A ( k ) {S_A}(k) is connected but need not be a manifold. However, the subset of S A ( k ) {S_A}(k) consisting of those subspaces with a fixed cyclic structure is a regular submanifold of G k ( V ) {G^k}(V) .