AstroImageJ Data Processing in Batch Mode

[George Fleming]

For the past year I have been using Richard Lee’s excellent tutorial on processing variable stars data with AstroImageJ and then using his spreadsheet to get the data ready for upload to the VSS database. With just a couple of targets this worked well – however as my ambitions grew and the amount of data grew exponentially –  I need a faster method of processing. Especially as urgent requests for monitoring another star is part of life attached to BAA VSS Alert system and yet it was taking weeks to get to the results – not just the lack of clear nights!

Attached are a PDF describing a new method of using AstroImageJ Data Processor in batch mode and an example macro which shows how to do this. Hope some members find this useful to start developing their own macros. (Please note the upload system needs a .doc extension – ImageJ macros use a .txt extension)

Attachments:

1. Using-AstroImageJ-Data-Processor-in-Macros.pdf
2. BatchDP.doc

[Dr Andrew Smith]

Thanks could be very useful. Regards Andrew 

[Dr Paul Leyland]

Thank you.  I’ve been using APT in batch mode to process many thousands of images from high-cadence photometry runs, at least in part because it is so easy to automate compared with AIJ.  Time for me to re-evaluate that decision.

Now to see whether your macro works in LibreOffice.  The ones supplied by the BAA-VSS do not, another reason why APT has been the utility of choice.

Paul

[George Fleming]

Are you trying to invoke AIJ from within the LibreOffice environment? At the moment this macro is designed to work once AIJ has been started. Once the macro gets setup for your system it will run a long time but the results are well worth it. I am still developing the macro to handle the Multi Aperture analysis automatically as well. After that I going to look at invoking a version of Richard Lee’s spreadsheet to process the output Measurement file to produce the VSS input data file. A complete workflow automated once the radec file is setup for the comparison stars

This morning I noticed I had topped 25,000 images plate solved in the last 12 months – even with Northern UK weather! Like you I really had to develop a batch system. I’ll post the next version here soon.

George

[Dr Paul Leyland]

I tried all combinations I could think of.  The macros (i.e. the ones supplied by the VSS) did not then work with Libre Office.

The bulk of the images I processed were high-cadence photometry of V3721 Oph.  I’m the L in CHL, the others being Phil Charles and Kevin Hills.

[George Fleming]

LibreOffice appears to be incompatible to Excel – the platform for the VSS macros? However the AIJ macro as it stands only needs your file structure to create a set of calibrated images ready for further processing. The next version will need a radec file which is an AIJ published format and can be generated by hand (with care) or use AIJ to mark the comparison stars and then “Export RA/Dec…”  is a File menu item on the Image display. Once you have that the new macro will produce a Measurement file which needs to be processed. This type of  text file is, I suspect, something similar to what you are handling now to produce the VSS input file?

[Dr Paul Leyland]

I can provide my VS pipeline to anyone who may find it interesting and/or useful.  The stages run as follows, where the bolded terms are my scripts, most of them in Perl, one in bash:

• Download a CCD photometry HTML file from AAVSO’s VSP page, then run it through CCD2APT which produces two files, one a source list for APT and the other a comparisons list for APT2VSS (described below).  This needs be done only once per variable.
• (Optional but recommended) Check the images for plausibility — correct field, not trailed, no inconvenient satellite tracks, etc.  I use the ds9 viewer.
• Put a WCS on the FITS files with solve-field from a local installation of astrometry.net.  For efficiency reasons I store a pre-parameterized call to solve-field held in a trivial shell-script.  That for SV Ari reads:

solve-field -O -p -L 0.25 -H 1.5 -u app -8 pos -3 03:25:03.34 -4 +19:49:52.9 -5 2.0 -z 4 *.fit

• Remove all the extraneous crud that solve-field leaves behind with clean_solve.
• Co-add if desired, using SWarp with COMBINE-T = AVERAGE
• Use APT for aperture photometry, either interactively or scripted.  APT is very easy to script.
• APT2VSS converts APT’s CVS-format output into a TSV file ready for submission directly to the BAA-VSS database.
• (Optional, every so often) Accumulate all the individual TSV files for a given variable with merge_phottsv into a multi-record file suitable for BAA-VSS submission.
• (Optional, once a month) Run all the BAA-VSS files created in a given month through VSS2TA to produce a file in the format which The Astronomer prefers.

Scripting the core of the pipeline is itself trivial.

[Author]

Posts