Martin.
The image size from the Atik 414 is small and your test with the large WASP-12b data set was successful so it can’t be a memory issue.
I tried Nebulosity in demo form many years ago but recall struggling with image calibration and correct pedestal values, often ending up with negative pixel values in the calibrated image so this problem may be related but without being able to inspect a sample of the data and calibration frames that is just a guess.
Can you try capturing data and calibration frames with different acquisition software, the ATIK capture software distributed with the camera should be ok for photometry work.
William.