Hi James,
The first challenge is to capture the sodium tail then later attempt to pull out detail if you can. I would go for the 174 based camera to maximise the SNR as it has the larger pixels and even employ a focal reducer to further improve SNR. I believe you need long accumulated exposures.
The main issue with the 174 is the relatively high read noise so another possibility is to use the 585 and bin the pixels 2×2. The 585 has a much lower read noise and with 2×2 binning the pixel size is almost exactly the same as the 174 (5.8um v 5.86um).
The expert on this is the BAA’s own Chris Hooker and I would seek advice from him – maybe you and he will be at the Winchester Weekend. There is a short section on imaging the ion tail in the BAA Mercury handbook but I think Chris who wrote the handbook has gathered much more experience since writing that section; https://britastro.org/wp-content/uploads/2020/04/Guide-to-observing-Mercury-v2.6.pdf
Good luck
Martin