Filtering Spectra

Forums Spectroscopy Filtering Spectra

Viewing 11 posts - 1 through 11 (of 11 total)
  • Author
    Posts
  • #574736
    Dr Andrew Smith
    Participant

    I have been investigating using wavelets and multi spectral analysis to denoise my flare star spectra which are necessarily noisy.

    The attached show the results using Matlab Wavelet Analyzer. The residuals show what looks like pure noise showing how effective it can be. This is from the HIP25953 flare on 2019-12-14.

    I think it show the potential of cleaning the data before making measurements. Regards Andrew

    #583137
    Dr Paul Leyland
    Participant

    Fascinating!

    Have you tried good old Savitzky-Golay smoothing? It was specifically designed for smoothing (and differentiating) spectra. If so, how does it compare with the wavelet approach?

    #583140
    Dr Andrew Smith
    Participant

    No as I had not heard about it. I will investigate. What I like about the multi spectral approach is you can clearly see what is happening at the various levels.

    Regards Andrew 

    #583141
    Robin Leadbeater
    Participant

    I like that the residuals follow the expected shape, noisy at the ends of the spectrum where the sensitivity is lower. It gives confidence that you are removing noise, not signal

    #583142
    Dr Andrew Smith
    Participant

    Had a look at the S-G smoothing and like many filtering approaches it requires you to set parameters which I can find no obvious method of fixing other than by eyeballing the results.

    While you need to set the number of layers in the wavelet multi spectral approach it uses noise modelling to set the thresholds on the filters at each layer. This together with the visualisation makes it simpler for me to assess the results.

    It does seem to be the modern approach, see Astronomical Image and Data Analysis  by Starck and Murtagh.

    Regards Andrew 

    #583143
    Dr Paul Leyland
    Participant

    Two standard approaches, one lazy and the other more rigorous.

    Lazy: guess the width of the signal peaks and set the smoothing window to be around 3 times that. Use degree 2 or 3 polynomials unless the window is very large (over 20 say) when you could choose degree 4 or 5.

    Rigorous: Compute residuals for a range of window widths and polynomial degrees. Perform statistical tests on them and choose the filter which best passes your acceptance requirements of high noise to signal ratio (that way round because you are trying to remove all the signal from the noise).  The Durbin-Watson statistic, which measures autocorrelatIon at lag 1, is a simple test which tends to work well, but a fuller autocorrelation at a variety of lags might be more appropriate.

    Wikipedia has good articles on S-G, D-W and several other tests for signal in the presence of noise. Easy to use implementations are all over the place. I tend to use R as it is free and portable, unlike Matlab and Mathematica.

    #583144
    Dr Andrew Smith
    Participant

    Thanks Paul, I have sent you the data I used in the example as requested by email and I look forward to what you discover. I looked at the Wiki on S-G and will look up D-W.

    I am looking for something I can reliably program as I can have a hundred or more spectra per flare to process. The noise profile changes with observing conditions e.g.moon light so as simple and reliable as possible is good.

    I used Matlab as I had it from a previous project not tried R but will look at it.

    Regards Andrew

    #583145
    Dr Paul Leyland
    Participant

    Thanks for prompting me to re-visit the S-G algorithm. As a result I found https://cran.r-project.org/web/packages/ADPF/ADPF.pdf

    which describes an implementation which automatically optimizes the polynomial order.  The window width still needs to be chosen by hand.

    #583152
    Dr Andrew Smith
    Participant

    Hi Robin, I had a play with you DN Gem spectra. Filtering confirms the lines you picked out. The noise residuals look good.

    #583219
    Robin Leadbeater
    Participant

    Sorry Andrew, I missed this

    Impressive.  I have just updated my copy of Visual Spec and I see Valerie has added a wavelets feature

    #583220
    Dr Andrew Smith
    Participant

    Yes Robin, I have used it. Valerie, was kind enough to respond to my questions on what some of the numbers were. It gives similar results but you have to watch the extremes of the data set as when you have low frequency wavelets you need a way of managing the boundary. Visual Spec clearly does this in a simple way and you can get artifacts.

    Regards Andrew 

Viewing 11 posts - 1 through 11 (of 11 total)
  • You must be logged in to reply to this topic.