Nick.
The SkyX dome setup is fairly well described in the manual (with one exception, see below) but here are a few tips from my experience using the smaller 2.2m Pulsar dome, the current Rigel dome controller and ACP/MaxIm with The SkyX and a Paramount.
Firstly, the dome driver (ASCOM or X2) does very little other than drive to a commanded AZ angle and open/close the shutter, the actual calculations for the position of the slot relative to the OTA are all handled by the observatory control program, in your case TheSkyX.
While the ASCOM driver for the Rigel dome controller works well in both ACP and MaxIm I found that it was unreliable and rather erratic when used with TheSkyX. I think this is a SkyX issue and nothing to do with the Rigel ASCOM driver and you will find TheSkyX X2 plugin for the Rigel dome controller works more reliably. I use the X2 driver when operating the dome via TheSkyX, for instance when building a T-Point or PEC model for the Paramount, but when running the dome normally under ACP/MaxIm I use the Rigel ASCOM driver where it performs well (in this configuration the dome is controlled by ACP and not TheSkyX).
You can download the X2 driver for the Rigel dome controller via the RTI-Zone website here.
Install the X2 driver as described in TheSkyX manual and select it in the dome setup tab, the geometry settings you already have remain intact.
Measuring accurately when configuring the dome geometry is vital for good OTA-slot alignment. You have to measure to within +/-5mm the mount and optical axis offsets and be sure to set the offset value signing correctly. Beside the mount and optical axis offsets the dome home-sensor offset setting must produce a real AZ value for the slot centre line that is within a degree of the displayed AZ value. To determine an accurate north reference I found using a compass bearing with lots of buried rebar in the dome concrete pad was useless with at least a 7 degree declination from true north at my location. For both pier and dome I used the local noon solar transit shadow method to determine true north, marked that on the dome wall and based my measurements for the dome home-sensor offset from that.
In TheSkyX dome setup there is one poorly explained setting that is critical for alignment between the OTA and the dome slot. In Dome Setup, Preferences, you will find the setting “Minimum angle from slit edge to telescope position” and it is not explained in the documentation how to determine the angle.
The starting point for calculating that setting is the formula: Arctan((L-D)/(2R)) where L = Width of slit in mm, D = Effective diameter of the OTA in mm (including guide scope if applicable) and R = radius of the dome in mm. Enter your calculated value in the box for “Minimum angle from slit edge to telescope position” and that should be a good starting value to ensure that the OTA optical axis always remains close to the centre on the slot. If you find any remaining overlap between OTA and slot then increase that value a degree at a time until the overlap is no longer present. For my current 100mm OTA = the calculated value was 8.58 deg and the actual value required was 12 deg to ensure that OTA and separate guide scope always remained aligned with the slot on both sides of the meridian. If that angle value is set too high or too low then the OTA will overlap either edge of the slot on both sides of the meridian.
Lastly, when driving the dome from TheSkyX the Rigel controller should be set to 0.00 sidereal rate on the control panel otherwise TheSkyX and the Rigel controller will argue over who is in control and the dome will oscillate continually. When driven from TheSkyX the dome does not continually track at sidereal speed but it moves in a series of incremental updates, the larger the diameter of the OTA relative to the size of the dome slot then the incremental updates occur more frequently. For my current 100mm OTA and 560mm width slot the dome increments about every three minutes, when I use a 350mm diameter OTA the dome increments every thirty seconds.
When setup correctly TheSkyX and the Rigel dome controller work well (using the X2 driver rather than the ASCOM driver) and I do not experience the overlap issues that you are reporting.
Hopefully there is something above that helps.
William.