I am not too sure how you “polynomial fit to the radial distortion in the lens”. Is it a one-off calibration done back in the distant past by, say, imaging a sheet of graph paper?
Alternatively, do you apply a rough calibration then iterate (plate-solve, polynomial-fit) until convergence?
Further, is the calibration polynomial assumed to depend only on the radial coordinate? If so, have you considered a full (x,y) polynomial fit? The distortion *should* be angle-independent but if you haven’t located the coordinates of the centre pixel (centre of the lens, not the centre of the detector) accurately enough you are likely to have problems.