The only thing I can think of is the the dispersion direction of the prism and grating are not aligned in such a way that the path differences with a fully illuminated entrance pupil and a partially illuminated one results in more dispersion from the prism in the fully illuminated case.
Clutching at straws!
Regards Andrew