Yes that is the right reference line. ISIS should definitely work with the camera you have and the method I suggested, I use it all the time. (The high errors means ISIS is failing to find and lock onto the correct lines)
Can you check that the smile correction has worked correctly please. Run the calibration on the lamp spectrum itself instead of the target star, ticking sky background not removed. (you will need to put a 1 on the end of the file name so ISIS sees it as the first of a series) If you then examine the image (“display image”) in the page you get after pressing go, the lines should be vertical and straight.
Can you also post the original image as a zip file please and I will see if I can get it to work this end.