found the problem

Robin Leadbeater

Thanks for the files.  I now see the problem.  When ISIS does the geometric correction for smile, this shifts the X positions of the lines so the  entered X reference position for the line that ISIS works out the other lines from is not correct and it cannot lock on.  This does not give a problem if the lines are close to vertical, for example in my case the shift is only 2 pixels. If there is a significant slant though the shift can be significant (+15 pixels in your case)  If you add 15 pixels to the reference position, I think you should find the calibration runs correctly.  Can you try it and see what happens?  (It ran for me giving an RMS of 0.3A but I have not tried it combined with the Balmer lines using the wizard)

The slant can be corrected in the ALPY by rotating the grism to make the lines perpendicular to the dispersion direction. (It is simple to do. I did this with my ALPY when I received it and I will dig out the instructions)  but I am a bit surprised ISIS does not take this shift into account. I will check with Christian Buil via the ARAS forum