In the seventies, László Babai has classified all finite groups isomorphic to Euclidean symmetry groups of vertex transitive polytopes. In the same paper, Babai asked for a related classification of the affine symmetry groups of orbit polytopes. The present dissertation introduces an algebraic theory of "generic symmetries" of group representations which is capable not only to reprove Babai's classical result, but also to answer Babai's question.