Hi John,
Are you using the calibration assistant method described here ?
http://www.astrosurf.com/buil/isis/guide_alpy/tuto_en.htm
If the rms errors are large in the wavelength calibration. It is usually because ISIS is not finding the correct lines in the lamp spectrum. To get this to work correctly the entered pixel size is critical. Because of tolerances in the ALPY this is not identical to the actual pixel size. You can calculate the pixel size to be entered by measuring the distance in pixels between H alpha and H beta in a hot star. The pixel size is then 3123/no of pixels. You may still need to do some fine tuning though to get ISIS to lock onto the lines correctly. Also the lamp spectrum needs to be well exposed and in good focus.
If you are having problems it can be worthwhile taking a step back and not using the calibration assistant but following the instructions here which has more information on the automatic wavelength calibration function and includes information on troubleshooting problems (method is 2 for the calibration module but the whole thing is worth reading to understand what is going on)
http://www.astrosurf.com/buil/isis/guide_alpy/resume_calibration.htm
If fact I generally use this method rather than the calibration assistant
Once set up, I find the automatic calibration reliable and much easier than finding the lines manually but it can be fiddly initially to get the pixel size correct.
Cheers
Robin