Hi Bill,
You wrote..
> In the basic library with Visual Spec the spectrum of Vega, for example has a sharp cut off then some oscillating components.
There may be something wrong here. The 5A/pixel Pickles A0v spectrum in the Vspec library (file a0v.dat) shows a smooth decline in the continuum at the blue end as the Balmer lines merge towards the Balmer jump and a relatively flat continuum beyond that.
You can see it here for example where I use it to calculate the instrument response of my star analyser setup
http://www.threehillsobservatory.co.uk/astro/Spectroscopy_BAA_VSS_workshop_2008.pdf
in slide 38
Are you using a different spectrum at very high resolution perhaps ?
Note that correcting just using the CCD (or photdiode) response will not give a good result as the response of the grating (and the atmosphere) is also significant here
Robin