Processing NGC7635 Bundle

[Keith Parker]

I have downloaded the NGC7635 bundle and processed passed the master HSO registration point and both OIII and SII have some major either gradient or background issues.  I was just wondering if anyone else has come across this issue? Any suggestions on how to correct these issues? 

Thank you,

KAP

OIII_registered.xisf SII_registered.xisf

[Scotty Bishop]

Hi Keith,

Go get the raw uncalibrated frames and use those with the raw bias, flats, and darks for those frames. Don't use the master calibration frames. You can get the uncalibrated light frames here: https://app.telescope.live/archive/my?showCalibratedImages=false&telescopeIds=61&targetName=7635

Go to the calibration frames after that and get your matching ones. The darks need to be 120s each and from as close to the same time as the lights were shot. The flats also need to be as close as you can get to when the lights were shot. The bias doesn't matter so get them all. https://app.telescope.live/archive/calibration?searchRadius=15&imageTypes=2,3,4&filters=(HSO),Halpha,SII,OIII&telescopeIds=61

You will also need to make sure you have your stuff in list view and set to 100 because 100 is the maximum you can download at a time using this. The cause of this is that there was some kind of calibration error in the calibrated light so you need to do it like you would with raw and uncalibrated files.

Doing that has gotten me some good results from the SPA-2-CMOS stuff. Here are a couple examples https://www.astrobin.com/iucyav/  https://www.astrobin.com/aj6rrz/

 

[Scotty Bishop]

Here are screenshots of the stretched master frames on this.

Sii



Oiii

Ha

[Keith Parker]

Thanks Scotty. I really appreciate the assistance.  I will download the files and get started on the processing. 

[Keith Parker]

Any suggestions on these errors? Trying to load them into Weighted Batch Processing. 

 

=== 0 of 172 light frames were added ===
Unable to open input file: F:/Astrophotography/Telescope Live/NGC7635_SPA-2-CMOS_272_Bubble Nebula - Processed/Bubble Nebula Darks Bias Lights/SPA-2-CMOS_2023-09-08T01-58-30_NGC7635_Halpha_120s_ID385173_raw.fits
Unable to open input file: F:/Astrophotography/Telescope Live/NGC7635_SPA-2-CMOS_272_Bubble Nebula - Processed/Bubble Nebula Darks Bias Lights/SPA-2-CMOS_2023-09-08T02-00-37_NGC7635_Halpha_120s_ID385174_raw.fits
Unable to open input file: F:/Astrophotography/Telescope Live/NGC7635_SPA-2-CMOS_272_Bubble Nebula - Processed/Bubble Nebula Darks Bias Lights/SPA-2-CMOS_2023-09-08T02-02-42_NGC7635_Halpha_120s_ID385175_raw.fits

[ERNESTO GUIDO]

Hi Keith, 

Raw frames are compressed and PI is unable to read it. So you need to convert it before using the raw frames in PI. For example you can open the raw frames with free software SIRIL and then save it. After that PI will be able to read it. Otherwise you can follow the info shared in this post below:
 

 

In particular there is a tool developed by @Jarmo Ruuth , you'll find all the info in the post above. 

 

Clear Skies,

Ernesto

[Keith Parker]

Is this the same for the Darks and Bias, PI will not load them as well?

And thanks again for the continuing support.

Keith

[Scotty Bishop]

MB-IG_0.0-E_0.0s-notAvailable-2394x1595-all_channels-session_1.fits MD-IG_0.0-E120.0s-notAvailable-2394x1595-all_channels-session_1.fits MF-IG_0.0-E_7.2s-notAvailable-2394x1595--Oxygen_III-session_1.fits

[Scotty Bishop]

MF-IG_0.0-E_69.6s-notAvailable-2394x1595--Hydrogen-alpha-session_1.fits MF-IG_0.0-E_108.4s-notAvailable-2394x1595--Sulphur_II-session_1.fits

[Scotty Bishop]

Give these master calibration frames a shot.

[Keith Parker]

I'll let you know how it turns out.

 

[Scotty Bishop]

2 hours ago, Keith Parker said:

I'll let you know how it turns out.

 

I can't wait to see. I posted my result in the gallery yesterday.

[Keith Parker]

Well, I finally figured out how to convert the raw files as well as the flats, darks and bias in Siril and am now running WBPP.  Crossing my fingers here.

Thanks again. 

 

 

[Ken Abremski]

Hi Scotty,

I followed your suggestions and used the uncalibrated raw files and the raw calibration frames with WBPP in PixInsight. Resulting masters and final image turned out well. Posted it to the gallery yesterday.

Thanks,

Ken

[Craig Schultz]

Scotty, I've seen you post about recalibrating the data yourself a couple of times now and I'm wondering why this is necessary at all.  Why would reprocessing the data using the raw bias, darks, flats and lights give different results compared to the calibrated telescope.live light frames unless the calibrated TL light frames are being generated incorrectly?  If there is a significant difference, all the raw files should be available to every subscription level, not just silver and above.

[Marek Idec]

Hi All,

I tried to process this data set and, first of all, light frames are calibrated incorrectly. It's clearly visible, that something is wrong when you look at a calibrated frame. It's an example with the OIII filter.

When you convert it to RGB and replace all zero pixels to red it becomes more visible that in most of the pixels there is no data at all.

Why is it like that? An obvious mistake is too low pedestal set during calibration. Why a pedestal is needed? A simple answer is because there is very low signal in those areas. The data sat is taken with narrow band filters, good narrow band filters with bandwidth 3nm. Considering the darks sky and the fact that  the SPA-2 is the "darkest" telescope from all Telescope Live telescope with the focal ratio f/8, the background in those frames is virtually indistinguishable from a dark frame.

Dark frames contain only noise. To be more precise, different type of noise combined. The noise is not random, and if you compare, for instance hot pixels in a raw uncalibrated frame and hot pixels in the master dark frame with the same exposure time, the mostly match. Which is why subtracting the master dark from from a light frame removes most of the hot pixels. However, there is a certain variance in the noise. If there is no light in the background, subtracting the master dark may result in negative values, that are clipped to zero. Hence we need a pedestal added to light frames to avoid clipped pixels. What should be the pedestal value? You can find it by trying various values. For this particular data set I set 100 the pedestal. This is one of the calibrated frames.

There are still some clipped pixels scattered all over the frame, but not as many as in the calibrated frame from Telescope Live. Is this value sufficient? If I increased the pedestal to 200, I would likely get less clipped pixels.

Now, an answer to Craig's question, why we need to reprocess master calibration frames. To be honest, I didn't know I had to do this, and I used master calibration frames downloaded from TL. I couldn't get satisfactory results. I tried various combinations, and my calibrated light frames were either under- or over-corrected with flats. I think, I know the answer now, and if I'm wrong, please anybody correct me.

I loaded the master dark and master bias frames in PI, and I found out that the master bias is already subtracted from the master dark. See the median value from both.

The master bias median is much higher than the master dark. When I processed the set in WBPP, first I used both dark and bias, and I checked the master dark optimization.

The calibration diagram shows that, in this case, the master bias is subtracted from the master dark and from the light frame, then dark is optimized and subtracted from the light frame.

Remember, the master dark median is much lower than bias, and effectively the bias would be subtracted twice. It can't be correct, right. I the master dark optimization is off, the master bias is not use at all in calibration.

Since only the master dark is subtracted from the light frame, the master flat cannot do a good job, because the background value in the light frame is too high.

To be honest, I have no idea, how the master dark from TL was created. The one I created from raw dark frames has very similar median to the master bias. So, adding master dark and bias from TL to get a new master dark would not work either. The only solution for me was generating master calibration frames myself from raw files.

There is another thing to remember while processing this data set or a similar one. Those clipped pixels in calibrated light frames, if not removed, can cause troubles later. This is a fragment of an example frame with clipped pixels marked in red.

After registration, those pixels grow.

They look like holes punched in the image. They are larger than one pixel and not clipped. There is some value there, although much lower than the background. They come from interpolation. Single hot pixels also grew into a larger shapes, but they will be easily removed during integration. Especially in this set with around 90 frames per channel. I tried to reduced those "holes" and most of them were removed with cosmetic correction. Not only them, but most of the hot pixels, too.

Nevertheless, I was not able to get rid all them, and some pattern similar to a "walking noise" was visible on integrated images, when they were zoomed out. I tried various types of interpolations and the best results were with bilinear. Stars were slightly larger than with the standard Lanczos-3, but not much, FWHMx was around 4.2 instead of 3.9.

After all experiments, I think, there is no way to improve the quality of the image more. There is simply no signal in the background. Increasing exposure time could help, I guess. I will try to do similar experiments with another set from SPA-2 with 300s exposure.

Summarizing what I learned from this set:
1. Narrowband TL calibrated frames from SPA-2 are unusable and we need to calibrated them ourselves, unless TL recalibrates them with a higher pedestal value.

2. Master calibration frames from TL can't be used in PI. At least I don't know how to use them, so we need to create master calibration files ourselves from raw files.

3. A pedestal value must be set while calibrating narrow band light frames.

4. Cosmetic correction with a reasonable hot and cold pixel removal is necessary to avoid artifacts after frames registration.

I will appreciate more any advice what could be done better.

Best regards,

Marek

[Astro Newbie]

Thank you Marek for your detailed explanation regarding this particular narrowband images problem. I think this may not just apply to that telescope when it comes to very faint targets and insufficient integration time. 

A partial solution might be to up the exposure times to 600 seconds to improve the signal. This was something originally suggested last May by John Ebersole in the 'Unveiling of TL4' post within the Platform Update section, originally posted November 2023. 

 

 

[Craig Schultz]

Marek, all I can say is WOW!  Thank you so much for going to such lengths to troubleshoot and explain what you are seeing with this data set and perhaps more within TL.  Hopefully, TL will respond to your post and outline how they generate the master calibration frames and subsequently the master frames with a flow diagram and critical process settings.  I'm hopeful this isn't a widespread issue affecting many data sets and that TL will regenerate all the master frames for those data sets that are affected.

At a minimum, it seems like having access to the raw data files for bias, darks, flats and lights would be a requirement to get the most from TL data and thus should be made available to all subscribers.

Thanks again for your considerable work on this.