HOPS software installation

[Roger Dymock]

The ARIEL Space Mission page at https://britastro.org/sites/default/files/ARIEL%20Space%20Mission%20V7.1_0.pdf has been updated to include a comprehensive guide to installing Python/HOPS software. A number of us have struggled with this installation and i must thank William Bristow for providing this update.

[Martin Smith]

Hi Roger,

I’ve installed HOPS, and reinstalled following all the steps in the guide above (including checking Python).

HOPS opens OK and I can fill in the fields. When run, the reduction frames process OK but the program becomes unresponsive when it’s the turn of the observation frames, the dialogue reporting 0.0% processing. I tried a subset of the observations and some other frames from another run to no effect. 

Do you have any insight into this?

Many thanks,

Martin

[William Bristow]

Hi Martin.

When you install Anaconda and Python it is important that both programs are installed in a root folder who’s name contains no white space. If you installed both Anaconda and Python in the default ‘user’ folder and that folder is named ‘Martin Smith’, with a space between Martin and Smith, then my experience has been that the Python scripts that HOPS runs will fail to run properly on Windows 10.

If this applies to you, reinstall Anaconda and Python in a dedicated folder at the root of the C: drive and not in the default user directory.

When running a dataset, the enclosing folder should not be write protected and works ok if placed inside the usual ‘My Documents’ location.

While awaiting a response from Roger, carry out a test with a known good data set to determine if the problem is with the source data or with the HOPS installation.

The sample data set I recommend is the WASP-12b series available from here:

https://www.astro.louisville.edu/software/astroimagej/examples/

Select and download the raw calibration folders and uncalibrated image folder in either .zip or .tar.gz to suit your OS.

The total download size is ~4.6Gb and if experiencing problems keeping the server link active recommend using a download manager.

The raw image data coordinates for the WASP-12b data set are in decimal format, to use this set with HOPS the coordinates to be used are RA: 6  30 43.057 and DEC: +29  40  32.20.

When this particular sample data set is used, to identify and match the target and calibration stars to the SIMBAD view click the ‘Flip FOV’ button in HOPS.

Selected filter to be used is ‘R’.

Lastly, the final fitting stage in HOPS is both CPU and memory intensive and for a large data set it may appear that nothing is happening for quite a while. The Wasp-12b set linked to above takes about five minutes to process on an iMac Pro with 12 core CPU, fast SSD and 64Gb RAM, on my small Windows laptop with an Intel i3 CPU standard HD and 8Gb RAM the same data set takes over forty minutes to process. If you are using similar hardware then allow at least fifteen to thirty minutes before assuming that HOPS has stalled.

HTH

William.

[Martin Smith]

Thank you William, the test data is a good idea and will give it a go. The white space issue was covered in the installation guide and should not be a problem. I thought it might be a matter of giving enough time for processing, but I have tried being patient. Also the progress bar stays firmly at 0%. Another long time series planned for tonight so I’ll have time to try your suggestions.

Thanks again,

Martin

[Martin Smith]

I was able to process the sample data with no problem so presumably there is an issue with my raw image files. I’ve attached the FITS header from one of the files, it contains less information than the sample files. Is something critical missing?

[William Bristow]

Hi Martin.

I can’t see anything obvious wrong with the FITS header, I don’t think HOPS has specific metadata dependencies.

Is the target in the HOPS database and are the target parameters auto loading at run time or are you manually entering the parameters for a new target previously unknown to HOPS?

Assuming the target parameters are loading/loaded correctly, try skipping the calibration frames and just run with a small set of the data frames, perhaps the data frames are being compromised during calibration, if so, running an uncalibrated set would show if that is the problem.

If HOPS still fails to solve, even when uncalibrated frames are used, upload a small set of frames to a file share service (Dropbox, Google Drive, One Drive, etc) and post a share link back here and I can try running them on the mac version of HOPS.

Two of each calibration frames and half a dozen good quality, random, data frames plus the target coordinates and target name would be enough to test.

William.

[Martin Smith]

It’s a known target, KELT-8b.

Using 5 images alone reduction is swift, if I add the bias and dark frames it reduces OK too. With the images and flat frames only it takes a while but works. With all four the progam becomes unresponsive, after half an hour there is 20% progress with an unreponsive program warning. 

[Martin Smith]

It works if the observation files are reduced separately, in the absence of an alternative this is probably the way to go.

[William Bristow]

Martin.

The image size from the Atik 414 is small and your test with the large WASP-12b data set was successful so it can’t be a memory issue.

I tried Nebulosity in demo form many years ago but recall struggling with image calibration and correct pedestal values, often ending up with negative pixel values in the calibrated image so this problem may be related but without being able to inspect a sample of the data and calibration frames that is just a guess.

Can you try capturing data and calibration frames with different acquisition software, the ATIK capture software distributed with the camera should be ok for photometry work.

William.

[Roger Dymock]

Hi Martin,

William knows far more about HOPS/Python than I ever will so I am sure his advice will provide the answer to your problem.

Regards

[Martin Smith]

OK I’ll try that. I not had problems with other software using images captured by Nebulosity, but it is an annoying program in some respects.

Thanks William for all your help.

Martin

[Martin Smith]

After a couple more sessions I’ve managed to get a light curve. HOPS still stalled when calibrating the images even though they were acquired using the Atik software and not Nebulosity. I reduced the frames in AstroimageJ. So here’s my attempt, I failed to remove some dodgy images but HOPS even seems to take care of that.

Thanks again for your help William.

Martin

[William Bristow]

Hi Martin.

Good to hear that you have made some progress with HOPS.

I can’t explain why HOPS will not calibrate your images, with the test images I have it works with quite large data sets and I have not experienced any hangs during calibration.

As your other processing programs are able to calibrate successfully the same data volume without problems I can’t really guess what might be going wrong when you try to run calibration in HOPS.

There are just a couple of items I can think of to double check:

With HOPS, the reduction and science files must be in the same source folder, no sub-folders are allowed. Hops does not identify and sort frame types from the fits header but reads frame type from the file name and therefore relies on the user to create unique file names for each of the reduction frame types and the reduction frames must not have the same identifying name as the science frames.

A long shot – in Windows 10, type “Control Panel” in the search box on the bottom toolbar and open the “Control Panel App” to bring up the classic Windows Control Panel > Click on the “System” icon > In the System page click “Advanced system settings”, the “System Properties” window will open > Click the “Advanced” tab,  at the top of the “Advanced ” tab click the “Settings” button under the “Performance” option, the Performance Options windows will open > Click the ‘”Advanced” tab in the “Performance options” window and under “Virtual memory” click the “Change” button, The “Virtual Memory” tab will open > Deselect the option “Automatically manage paging file size for all drives” and select the Custom size button > If you have more than one drive you can define a page file to be used as virtual memory on each drive and Windows performance will improve if you are able to define page files on multiple fixed drives but for this walk-through assume only one drive is present > Select the first available drive (C:) and set the “Initial size (MB)” for the page file, the recommendation here is to set at least 420 MB > Set the “Maximum size (MB)” to be at least a third of the available free space remaining on the drive, (the free space remaining is displayed just above the Custom size boxes) ideally, to assure adequate resources are available during post processing the Maximum size should be at least 20000MB (~20Gb) > With those figures entered click the “Set” button > Repeat for additional fixed drives if present and then click “Apply” and “OK” your way out of all open windows then reboot the computer for those changes to take effect.

If that solves the problem during calibration then it was a page file too-small issue and Windows had run out of available disk space for swap files, if the problem is not helped by this then revert the Page File control to Automatic.

If I can find anything else relevant I will update the thread.

William.

[Martin Smith]

Thank you William. I have all the files in a single folder so that’s not an issue. I’ll work through the Windows tweaking and see if that helps. 

All the best,

Martin

[Tim Haymes]

Thanks for the correspondence – yes I also encounter this problem.  I will check the installation in the light of what has been discussed.  I have a data set for Qatar-1b that could be submitted if HOPS completes successfully.  I thought it was caused by poor data quality.  I will investigate further.

[Tim Haymes]

Ive checked my installation I think my problem is different.  The fact i can get to a preliminary light curve suggest the install is OK. When I type at the anaconda prompt: echo %PATH%  the path includes  C:Anaconda3. So im guessing that’s correct (It is the correct dir although not the one in the ARIEL notes, and python is there also).  The problem I have in the work flow, is the FITTING routine.  I get this message displayed for 1 second, then nothing.  What does this imply?

[Tim Haymes]

Sorry there is  problem editing my message above.  I will make contact via email
Basically the FITTING does not work. Although anaconda installation looks OK

It turns out that my data is causing a problem in W7 and W10 installations, while the WASP-12b test data is OK.

I hope to try another transit with a different camera.

Tim

[Dr Paul Leyland]

Everything went well with the installation, though sudo was needed to complete the installation as expected.

However, when run the code crashes with

ImportError: cannot import name ‘ImageTk’ from ‘PIL’ (/usr/lib/python3/dist-packages/PIL/__init__.py)

There was lots more diagnostic output which can be provided on request.

Anyone any ideas?  I’ve contacted the Exoclock team but without any response so far.

[Author]

Posts