- This topic has 10 replies, 3 voices, and was last updated 4 years, 2 months ago by
.
-
Posts
-
I have been investigating using wavelets and multi spectral analysis to denoise my flare star spectra which are necessarily noisy.
The attached show the results using Matlab Wavelet Analyzer. The residuals show what looks like pure noise showing how effective it can be. This is from the HIP25953 flare on 2019-12-14.
I think it show the potential of cleaning the data before making measurements. Regards Andrew
Attachments:
Fascinating!
Have you tried good old Savitzky-Golay smoothing? It was specifically designed for smoothing (and differentiating) spectra. If so, how does it compare with the wavelet approach?
No as I had not heard about it. I will investigate. What I like about the multi spectral approach is you can clearly see what is happening at the various levels.
Regards Andrew
I like that the residuals follow the expected shape, noisy at the ends of the spectrum where the sensitivity is lower. It gives confidence that you are removing noise, not signal
Had a look at the S-G smoothing and like many filtering approaches it requires you to set parameters which I can find no obvious method of fixing other than by eyeballing the results.
While you need to set the number of layers in the wavelet multi spectral approach it uses noise modelling to set the thresholds on the filters at each layer. This together with the visualisation makes it simpler for me to assess the results.
It does seem to be the modern approach, see Astronomical Image and Data Analysis by Starck and Murtagh.
Regards Andrew
Two standard approaches, one lazy and the other more rigorous.
Lazy: guess the width of the signal peaks and set the smoothing window to be around 3 times that. Use degree 2 or 3 polynomials unless the window is very large (over 20 say) when you could choose degree 4 or 5.
Rigorous: Compute residuals for a range of window widths and polynomial degrees. Perform statistical tests on them and choose the filter which best passes your acceptance requirements of high noise to signal ratio (that way round because you are trying to remove all the signal from the noise). The Durbin-Watson statistic, which measures autocorrelatIon at lag 1, is a simple test which tends to work well, but a fuller autocorrelation at a variety of lags might be more appropriate.
Wikipedia has good articles on S-G, D-W and several other tests for signal in the presence of noise. Easy to use implementations are all over the place. I tend to use R as it is free and portable, unlike Matlab and Mathematica.
Thanks Paul, I have sent you the data I used in the example as requested by email and I look forward to what you discover. I looked at the Wiki on S-G and will look up D-W.
I am looking for something I can reliably program as I can have a hundred or more spectra per flare to process. The noise profile changes with observing conditions e.g.moon light so as simple and reliable as possible is good.
I used Matlab as I had it from a previous project not tried R but will look at it.
Regards Andrew
Thanks for prompting me to re-visit the S-G algorithm. As a result I found https://cran.r-project.org/web/packages/ADPF/ADPF.pdf
which describes an implementation which automatically optimizes the polynomial order. The window width still needs to be chosen by hand.
Hi Robin, I had a play with you DN Gem spectra. Filtering confirms the lines you picked out. The noise residuals look good.
Sorry Andrew, I missed this
Impressive. I have just updated my copy of Visual Spec and I see Valerie has added a wavelets feature
Yes Robin, I have used it. Valerie, was kind enough to respond to my questions on what some of the numbers were. It gives similar results but you have to watch the extremes of the data set as when you have low frequency wavelets you need a way of managing the boundary. Visual Spec clearly does this in a simple way and you can get artifacts.
Regards Andrew
- You must be logged in to reply to this topic.