I’m impressed that you manage to achieve 0.1 mag accuracy!
I have also played around the background subtraction and flat-fielding with my meteor camera, which I use to get as many stars as possible when calibrating the pointing and lens abberations using astrometry.net. But the stars have a tendency to be horribly saturated (which isn’t an issue for me).
Presumably you have to tweak the gain manually for each target, to place the star optimally within the very limited dynamic range you have? I’ve noticed that stars often have quite evident “ringing” artifacts on either side of them, due to a analogue signal losing its higher frequency components, which would be another noise source. The limited number of pixels is another factor that I would imagine might making accurate photometry difficult.