TPRDSS

TPRDSS

A tool to find matching darks for lights and to group them
by temperature, iso and exposure time in a file list for Deep Sky Stacker

Version 0.4beta

Special thanks to Michael for beta testing.

Motivation
Knock-out criterion
How the program works
Implementation notes
Contact
Installation notes
History
Download
The GUI
Legal stuff

Motivation

Common practise to take dark frames is to take them at the same ambient temperature as the light frames.
This practise has one drawback: the camera sensor heats up from one exposure to the next until the temperature reaches a maximum. If you keep in mind that dark current noise increases with temperature (it almost doubles every 6�C) the first light frames of an exposure series contain less dark current noise and the last light frames contain more.

The dark frames are subject to the the same behaviour. The first frames are cooler than the last ones: here also exists an increase of dark current noise from frame to frame.
The dark frames are averaged with a stacker program to produce a master dark frame. This reduces stochastic noise to 1/sqrt( n) in the master dark frame (n = number of dark frames).
But: dark frames not only contain stochastic noise.They also contain deterministic noise in form of warm and hot pixels (the light frames obviously too). With the temperature, warm and hot pixel vary in strength: the higher the temperature the more lucid those pixels. And the warm and hot pixel reside at the same position in the pictures (that's why they are deterministic) more often than not. Therefore its strength cannot be reduced by averaging several darks.They remain in the master dark frame. This is a wanted nature of master dark frames. It helps eliminating hot pixel and amplifier glow in the lights.

But: Because the light frames are different in deterministic noise due to their different temperature and the master dark frame is constant in this respect, the dark frame calibration of the light frames leaves artefacts in the results. Cooler light frames are over-calibrated, warmer light frames are under-calibrated.
The following two pictures of M17 - the Omega Nebula - show this effect:

too_cool.gif (336861 Byte)

too_warm.gif (340353 Byte)

TPRDSS was inspired by the Dark Library, introduced on Cloudy Nights in this thread and by the impacts of the Dark Library illustrated on Astronomy Forum. But the Dark Library is not able to handle groups in Deep Sky Stacker.

To overcome these issues TPRDSS was created.

Knock-out criterion

If your camera does not read out the temperature and write it into the EXIF data there is no way to use this program.
Check your camera out if it supports temperature. The EXIF tag is called "Camera Temperature". Use IrfanView for example to get access to the EXIF data of your frames.

How the program works

Your dark frames
TPRDSS needs to know where your dark frames reside and what EXIF data they contain. With TPRDSS you create a list of all frames that reside in a folder and all its subfolders. This list is saved in a file called "dark list" here. You may save this dark list where you want.
The file extension your frame files have is entered in the preferences dialog. Whithout this extension TPRDSS won't work at all.
The dark list will contain one line per frame file in the format: temperature space iso space exposure_time space absolute_path_of_frame_file LF
You create this dark list only once. If you've taken new dark frames and want TPRDSS to use them too, simply generate a second dark list and add their entries manually to the initial dark list (at the top, at the bottom or in between doesn't matter).
TPRDSS will use this dark list to find the matching dark frames for your light frames.

With TPRDSS you may arrange your dark frame folders. This will copy your dark frame files to a new folder structure and optionally rename them there. Each folder will contain all dark frame files of a specific temperature, iso and exposure time (or exposure time range if the exposure times vary slightly and you specify a variant >0). This arrangement is optional. If you don't want to use it leave it. But keep in mind, Deep Sky Stacker will create master dark frames on a folder basis. If you don't want to arrange your dark frames, you should activate the separation of them later when generating the file list for Deep Sky Stacker (see below).
If you arrange your dark frames, Deep Sky Stacker will generate a master dark frame for each folder in use and will reuse it in future sessions even if its file list contains individual dark frames and no master dark frame. Deep Sky Stacker recognises a master dark frame in a folder that contains the individual dark frames and favors it over the individual dark frames if the actual stacking method for the dark frames is the same that was used to create the master dark frame. If you have added new dark frames as described above and Deep Sky Stacker sees these new dark frames in a folder which contains a master dark frame, Deep Sky Stacker will indeed create a new master dark frame but it will not save it in this folder (at least the file date is not changed). In those cases I delete the affected master dark frames and associated Description text files before I start stacking with the newly added dark frames.
Rearranging will save a lot of computation time in future stacking sessions. If you don't use the arrangement but instead the separation described below, the master dark is created prior calibration in every future stacking session.
If you've taken new dark frames, you already rearranged your old ones and want TPRDSS to use the new ones too, simply generate a second dark list, but for the destination folder specify the folder previously used in rearrangement. Afterwards add the entries of this second dark list manually to the initial dark list.

Your light frames of an astronomic object
Just as well with the dark frames TPRDSS needs to know where your light frames of an object reside and what EXIF data they contain. The same manner you created the dark list you create a light list in respect of the list file (there is no need to arrange the light frames etc.)

The Deep Sky Stacker file list
TPRDSS uses the light list and the dark list you created before and extends a given file list from Deep Sky Stacker. The file list from Deep Sky Stacker may contain bias frames and flat frames only or just bias frames (if you use flat frames, the use of bias frames is strongly recommended). Instead of individual flat and bias frames you may use the master version of them of course.
So, create a file list with Deep Sky Stacker that contains your calibration frames and save it (either with the extension .txt or .dssfilelist) where you like. Don't create new groups in Deep Sky Stacker. Otherwise Deep Sky Stacker will use the calibration frames in a group for this group only.

Then

specify the path to your dark list you create before if it is not specified automatically (defaults to the preferences entry)
specify the path to your DSS file list you created before with Deep Sky Stacker
specify the path to your light list you create before if it is not specified automatically
specify the path to the Output DSS file list which will be generated
optionally activate the separation of dark and light frames into individual folders according to temperature (or temperature range if you specify a variant >0), iso and exposure time (or exposure time range if you specify a variant >0) of the light frames and specify the path to the folder that will receive these individual folders.
With the two fields "Exp. time variance %" and "Temp. variance �C" you can specify a value so that the exposure time and temperature of the dark frames act as a range. This will find more dark frames, especially when the exposure time of the light frames and/or dark frames vary slightly. A too big variance for the temperature however will make this tool useless.
press the Generate button

TPRDSS will log all activities in black color and errors in red color into the "Log" panel and writes what it generates as a copy into the "Generated" panel.

Now, load the generated Output DSS file list in Deep Sky Stacker. In the Registering and Stacking view you will see one ore more additional groups below the file panel right to the Main Group according how your lights vary in temperature, iso and exposure time.

Make adjustments you like and stack.

Implementation notes

ExifTool
TPRDSS uses ExifTool by Phil Harvey version 8.61 to extract camera temperature, iso, exposure time and date time original from picture files. If your camera is not supported by ExifTool yet, wait for a newer version, copy its executable to the folder script of TPRDSS installation directory and call it exiftoolexe.exe.

Performance
TPRDSS uses threads to extract EXIF information from picture files. The number of threads is determinded by the number of cores + 1. In the log panel you may see file copy logs that do not match to directory creations when arranging dark frames. This is due to the asynchronous communication between the worker threads, which among other things create the directories, and the file copy thread.

Contact

Troubles, suggestions etc: mailto:markus@laus-hiel.de in english or german.

Installation notes

There is no installer program. Extract the downloaded tgz-file to a folder you like.
TPRDSS is written in Java. You should have a Java 6 Runtime installed in order to run TPRDSS.
To run TPRDSS double click the file tprdss.cmd in the folder you extracted the tgz-file. If it doesn't work drag tprdss.cmd to your desktop and try it there.
TPRDSS will create a folder called TPRDSS in your home directory. In this folder the application stores three files:

preferences.xml: contains the entries of the preferences dialog
prefs.xml: contains recently used folders, files, project, window position and dimension
rootlogger.log: the log file of this application. On misbehaviour please email me this file with a failure description.

History

0.4beta
- some files were written with UTF-8 but read with default charset. Now, all files except dss-filelist are written and read with UTF-8.

0.3beta
- removed check button for disabling the Arrange darks pane in the preferences dialog that does not exists since 0.2beta anymore
- code for reading out exif data when generating a dark list failed; this was due to test code which was not removed before creating the release

0.2beta
- localized to German
- tooltips for every text input and output field
- generating dark list combined with arranging darks
- validating fields before generating
- recently used projects (max. 10) added to project menu
- redesign of recently used files and folders
- errors during generating are written in red to the log panel
- more detailed error logging when exif information missing
- bug fixes

0.1beta
- first release

Download

0.4beta

The GUI (right click show picture to see the picture in its original size)

	The first time you start TPRDSS you are asked to enter the file extension of your pictures files.
	CR2 is the extension of the files taken with a Canon EOS 1000D for example. The extension is not case sensitive and it doesn't matter if you specify a dot or not.
	In the Generate light list pane you create a file - the New light list - that will contain the temperature, iso, exposure time and path of the light frames of an object you want to stack in Deep Sky Stacker.
	In the Generate DSS file list pane you create a file - the New DSS file list - that will contain the DSS file list plus the groups with the light frames and matching dark frames.
	In the Generate dark list pane you create a file - the New dark list - similar to the Light list but from all your dark frames. This Dark list will be used in all future projects. How to extend this dark list with new darks taken see Your dark frames. It is recommended that you rearrange your darks into a new directory structure under the Destination folder. Specifiy an exposure time variance (10 should be enough) to ensure that slightly variing dark frames will be copied into the same folder. Your darks will be copied and not moved. File creation times will be preserved. You also may rename your dark frames if you like to.
	The new directory structure after a rearrangement will look like the Explorer picture to the right. The directory name consists of the temperature, iso and exposure time. When renamed and used the date and time in the file names are taken from the EXIF data and not from the file date.
	If you have disabled the panel called "Generate dark list" in the Preferences dialog the window will look like this:

Legal stuff

TPRDSS is free software.
The author of TPRDSS and owner of this homepage is not liable for damages caused by this software.