Have I made an error with Oiii data from CHI-1

[Liam Quinn]

Hi All

I'm not quite sure if this has been processed correctly in PixInsight. 
Just for a bit of background - I'm new to processing in general, been putting myself through crash courses in Pixinsight and am enjoying it. I normally do OSC stuff with my gear. 
Been inspired with all the faint Oiii things being found by others, so I used my credits with CHI-1 to look at NGC 253 here - I've 23 x 5 min Oiii subs and some blue filter ones too (this image has not had continuum Subtraction done yet) 
I was having a few issues with WBPP and think I solve it as it was running through fine - I had all the calibration files in and selected but the master output was not too solid. 
I kept seeing the master flat file issues on it. 
So I tried doing manual steps instead - with image registration, I used cosmetic correction using the Master Dark file - then Image Integration.
As you can see there is this outer ring on this image (hopefully attached) while I wish it was Oiii data I suspect its more a flat fielding problem. 
I did specific dithering on these subs via Adv requests too if that needs to be considered somewhere. 

Am I allowed to share my data? - just wondering if I missed something on the processing side?  
Is there anything special you need to do to process Oiii only data in PixInsight with WBPP? 

Thanks

Liam

[ERNESTO GUIDO]

Hi Liam, 

We do an automatic calibration of the frames you find in the archive. It is possible that this halo in the OIII files is an artifact caused by this automatic calibration. Have you tried doing manual calibration using the raw frames and calibration files? 

 

You can download the raw frames using the Calibration filter in the MY DATA ARCHIVE section (see attached screenshot).

Please consider that raw frames are compressed and PI is unable to read it. So you need to convert it before using the raw frames in PI. For example you can open the raw frames with free software SIRIL and then save it. After that PI will be able to read it. Otherwise you can follow the info shared in this post below:

 

 

Clear Skies,

Ernesto

[Liam Quinn]

On 12/12/2023 at 11:45 AM, ERNESTO GUIDO said:

Hi Liam, 

We do an automatic calibration of the frames you find in the archive. It is possible that this halo in the OIII files is an artifact caused by this automatic calibration. Have you tried doing manual calibration using the raw frames and calibration files? 

 

You can download the raw frames using the Calibration filter in the MY DATA ARCHIVE section (see attached screenshot).

Please consider that raw frames are compressed and PI is unable to read it. So you need to convert it before using the raw frames in PI. For example you can open the raw frames with free software SIRIL and then save it. After that PI will be able to read it. Otherwise you can follow the info shared in this post below:

 

 

Clear Skies,

Ernesto

Hi Ernesto

thank you for responding - and the useful info - I've not tried raw processing but its now on my to do list!
 

[Reggie Jones]

Hi Liam.  I use PI exclusively and I agree that this is a flat field problem and in my experience, this happens sometimes with TL data.  The solution I turn to when I encounter this problem is to continue pre processing through integration.  After that, I create and apply a synthetic flat for the affected integrated master and that usually takes care of the problem.

Thanks

Reggie

[Liam Quinn]

3 hours ago, Reggie Jones said:

Hi Liam.  I use PI exclusively and I agree that this is a flat field problem and in my experience, this happens sometimes with TL data.  The solution I turn to when I encounter this problem is to continue pre processing through integration.  After that, I create and apply a synthetic flat for the affected integrated master and that usually takes care of the problem.

Thanks

Reggie

Interesting Idea Reggie - Is there a general video posted anywhere touching upon this technique?
I've a rough idea but might take me a go or 3 to implement that 

[Liam Quinn]

On 12/12/2023 at 11:45 AM, ERNESTO GUIDO said:

Hi Liam, 

We do an automatic calibration of the frames you find in the archive. It is possible that this halo in the OIII files is an artifact caused by this automatic calibration. Have you tried doing manual calibration using the raw frames and calibration files? 

 

You can download the raw frames using the Calibration filter in the MY DATA ARCHIVE section (see attached screenshot).

Please consider that raw frames are compressed and PI is unable to read it. So you need to convert it before using the raw frames in PI. For example you can open the raw frames with free software SIRIL and then save it. After that PI will be able to read it. Otherwise you can follow the info shared in this post below:

 

 

Clear Skies,

Ernesto

Hi Ernesto

So I got the raw Oiii files from my Advanced Request and sorted the compression issue by following your advice with SIRIL.
Still having a right problem getting a proper output from WBPP - hopefully some screenshots aid in explaining what I do (hopefully something is amiss as I cannot explain the output differences) 

For Subframe weighting I follow Adam Blocks advice with picking PSF Scale SNR - sadly I am finding my finished output after WBPP is not very good. 
I'll include my WBPP screen and two Oiii images from the master folder - the two output images here are with identical settings with one single change - if I untick 'Calibration Settings - Dark'  highlighted in Red I get a much better image without all the dust ring issues. 
I've seen the same problems on L,R,G,B,Ha,Sii as well 

Am I missing something obvious? 
 

[Liam Quinn]

On 12/12/2023 at 11:45 AM, ERNESTO GUIDO said:

Hi Liam, 

We do an automatic calibration of the frames you find in the archive. It is possible that this halo in the OIII files is an artifact caused by this automatic calibration. Have you tried doing manual calibration using the raw frames and calibration files? 

 

You can download the raw frames using the Calibration filter in the MY DATA ARCHIVE section (see attached screenshot).

Please consider that raw frames are compressed and PI is unable to read it. So you need to convert it before using the raw frames in PI. For example you can open the raw frames with free software SIRIL and then save it. After that PI will be able to read it. Otherwise you can follow the info shared in this post below:

 

 

Clear Skies,

Ernesto

Hi Ernesto

So I got the raw Oiii files from my Advanced Request and sorted the compression issue by following your advice with SIRIL.
Still having a right problem getting a proper output from WBPP - hopefully some screenshots aid in explaining what I do (hopefully something is amiss as I cannot explain the output differences) 

For Subframe weighting I follow Adam Blocks advice with picking PSF Scale SNR - sadly I am finding my finished output after WBPP is not very good. 
I'll include my WBPP screen and two Oiii images from the master folder - the two output images here are with identical settings with one single change - if I untick 'Calibration Settings - Dark'  highlighted in Red I get a much better image without all the dust ring issues. 
I've seen the same problems on L,R,G,B,Ha,Sii as well 

Am I missing something obvious? 
 

[Liam Quinn]

Possibly Similar Issue
This thread I found was interesting in that the resulting images are not right - "So my conclusion is: the wrong offset values for the dark frames and flat-darks (offset 30 instead of 50) resulted in a too large numerator in the flat field correction, causing the overcorrection."

Also "images taken with same parameters, but different acquisition SW, differ so much and are (more or less) incompatible."    Meaning I think APT versus NINA for that example. 

I'm struggling to see any other reasons unless I messed up majorly.  Still testing every possibly solution I'm aware of inside PixInsight in the mean time. 

 

[Liam Quinn]

I even put together a little test 7z file for people to play with - can you get this through WBPP properly - I am more convinced there is an issue with the Master Dark and/or Master Flat

Test O3 Data with Relevant Master Files 

[Reggie Jones]

Hi Liam.  Here is a tutorial on how to create and apply a synthetic flat.

 

 

[Liam Quinn]

3 hours ago, Reggie Jones said:

Hi Liam.  Here is a tutorial on how to create and apply a synthetic flat.

 

 

Thanks Reggie - will look that over!

[Scotty Bishop]

While I generally stack in APP and do some of my initial stuff in it too I would try stacking it with both the raw calibration frames (latest Oiii flats, along with the latest darks and bias), then try with the pre-calibrated master flat, dark, and bias. I have found both will give me slightly different results. If that doesn't work I would definitely go for a synthetic flat per the tutorial Reggie posted.

If you have APP too I would give it a shot through that because APP has some powerful normilization and background gradient removal tools built in and I've been able to stack things that PI runs from. One more thing I have seen people get away with is editing the fits header to "fool" a stacking program into thinking things matched.

[Liam Quinn]

6 hours ago, Scotty Bishop said:

While I generally stack in APP and do some of my initial stuff in it too I would try stacking it with both the raw calibration frames (latest Oiii flats, along with the latest darks and bias), then try with the pre-calibrated master flat, dark, and bias. I have found both will give me slightly different results. If that doesn't work I would definitely go for a synthetic flat per the tutorial Reggie posted.

If you have APP too I would give it a shot through that because APP has some powerful normilization and background gradient removal tools built in and I've been able to stack things that PI runs from. One more thing I have seen people get away with is editing the fits header to "fool" a stacking program into thinking things matched.

Thank you very much for looking at my dataset Scotty its appreciated - Looks like I need to buy APP now as well. Always heard good things on its mosaic potential and I do a lot of that. 
I will need to do more checking of the FITS file header certainly from now on - I know about the '60' issue with PI and having to rewrite that in the FITS header. 
I'll try a few more paths with PI you've suggested - they really need to improve a little esp with handling mosaics as I have to usually image solve then MosaicByCoOrd to get anywhere. 
Did the dataset go fairly OK with App ?  Any great issues? 
Also big respect for the Gowron avatar!  - Glory to you and your house!  😄

Edited December 21, 2023 by Liam Quinn

[Scotty Bishop]

19 hours ago, Liam Quinn said:

Thank you very much for looking at my dataset Scotty its appreciated - Looks like I need to buy APP now as well. Always heard good things on its mosaic potential and I do a lot of that. 
I will need to do more checking of the FITS file header certainly from now on - I know about the '60' issue with PI and having to rewrite that in the FITS header. 
I'll try a few more paths with PI you've suggested - they really need to improve a little esp with handling mosaics as I have to usually image solve then MosaicByCoOrd to get anywhere. 
Did the dataset go fairly OK with App ?  Any great issues? 
Also big respect for the Gowron avatar!  - Glory to you and your house!  😄

The dataset was pretty easy to do in APP. I imagine you were passed along the master lights and the HSOB stack I did with them as well. I think the master lights will combine in PI pretty straightforward. Ohh, and APP is probably the easiest program to use to stack and to do mosaics, and it is very good. In fact using the calibrated lights from TL you can simply extract them all into one folder, and then select them all and load them as lights without assigning a session or filter to them. APP will read the fits header and assign them as the proper filter (though when combining them in the RGB combine tool it always sets the Ha filed from here as custom so you need to select hydrogen alpha).

Good luck on the other things you are going to try with PI. And you can get a free and fully functional trial of APP, plus I have plenty of stuff in the articles section showing how to use it. And glory to your house!

[Liam Quinn]

21 minutes ago, Scotty Bishop said:

The dataset was pretty easy to do in APP. I imagine you were passed along the master lights and the HSOB stack I did with them as well. I think the master lights will combine in PI pretty straightforward. Ohh, and APP is probably the easiest program to use to stack and to do mosaics, and it is very good. In fact using the calibrated lights from TL you can simply extract them all into one folder, and then select them all and load them as lights without assigning a session or filter to them. APP will read the fits header and assign them as the proper filter (though when combining them in the RGB combine tool it always sets the Ha filed from here as custom so you need to select hydrogen alpha).

Good luck on the other things you are going to try with PI. And you can get a free and fully functional trial of APP, plus I have plenty of stuff in the articles section showing how to use it. And glory to your house!

Appreciate the APP guidance there - will deffo pick that program up as differing processing tools is always a bonus and very handy when things are difficult. 
Love the mosaic features I've seen so far with it. 

Just to finish the thread I did get it working in PixInsight by the way - following your suggestion of just raw calibration frames I did just that - ignored the masterFiles (bias, dark, flats) that were available entirely - and finally got somewhere once I did that. I was previously trying every combo of setting using just the masters!

Anyway appreciate the time and effort put in to help - I'll be checking your articles on APP usage for sure. 

Here is my Oiii master Light...