What a puzzle.

Andrew Smith

The only thing I can think of is the the dispersion direction of the prism and grating are not aligned in such a way that the path differences with a fully illuminated entrance pupil and a partially illuminated one results in more dispersion from the prism in the fully illuminated case.

Clutching at straws!

Regards Andrew