The on-line Data Acquisition (DA) system reads out each CCD chip in the imaging camera simultaneously, and groups together data (e.g. quartiles) from all the images, which are sometimes referred to as frames, into a Gang file. Since it takes about 10 minutes for a star to drift across the focal plane and trail over each CCD chip, the data in a Gang file correspond to the same read-out time, rather than to the same area of sky. The SSC re-arranges this data and produces a new Fang file, which looks very much like a Gang file, but has information for the same area of sky, a field, in each passband all together. Additionaly, the SSC pipeline is used to read raw photometric images from tape, and to write them as idR files to disk.
The downstream photometric pipeline is divided into two smaller pipelines, the Postage Stamp Pipeline (PSP), and Frames Pipeline. Frames Pipeline operates on a frame by frame basis (the photometric data stream from each CCD in the photometric array is cut into frames of 2048 x 1362 pixels). The quantities such as the point spread function, the sky brightness, and the flat field and bias vectors which are not local quantities, but are determined as a function of time, are determined by the PSP.
The PSP uses information form Fang files to calculate the sky brightness, and the flat field and bias vectors. To determine the point spread function (PSF), it uses small regions, called postage stamps, which are cut out from large image frames by the SSC pipeline, and also written to Fang files. Various types of postage stamps, and details of the cutting procedure are described in section Algorithms .
There are two main ways one can run the SSC: simply to read images from
tape and write onto disk, or to read and process Gang files,
images, etc. If you set the writePhotomOnly keyword in the scPlan.par file to 1, then the SSC will do
nothing but read photometric images from tape and write them to disk.
In that case, the only outputs are FITS images
idR*.fit.
Assuming that writePhotomOnly is set to 0, the SSC produces
the following output:
As already said in Introduction, the main SSC pipeline tasks
are to rearrange data from files corresponding to a given read-out time
to files containing data for a given area on sky, and to produce postage stamps.
This section describes these steps in more details. Names printed in typeset
font are input parameters specified by user in scPlan.par
and scParam.par files.
The on-line DA system reads out each CCD chip in the imaging camera simultaneously, and groups together data (e.g. quartiles) from all the images read out at one time into a Gang file. Since it takes about 10 minutes for a star to drift across the focal plane and trail over each CCD chip, the data in a Gang file do not correspond to the same area of sky. For example, the layout of the imaging camera shows that a star which appears in camera column 2 will first drift across the leading astrometric chip (number 72), then the r'-band chip (12), then the i'-, u'-, z'- and g'-band chips (22-52), and finally leave by moving across the trailing astrometric chip (82). The way the on-line DA arranges things, this star would appear in 7 different Gang files; for example,
The leading astrometric detection in Gang file 0020 The r'-band detection in Gang file 0023 The i'-band detection in Gang file 0025 The u'-band detection in Gang file 0027 The z'-band detection in Gang file 0029 The g'-band detection in Gang file 0031 The trailing astrometric detection in Gang file 0033
One of the primary purposes of the SSC is to re-arrange the information gathered by the DA so that all the data pertaining to some area of the sky is collected into a single file. This Fang file therefore looks very much like a Gang file, but has information on stars in each passband (and astrometric chip) all together. To continue the example shown above, the Fang file for field 23 would contain data from each of the detections of the same star.
1) Find stars appearing on both astrometric chips.
The positions are taken from astrometric Gang files. Matching is controlled
by am_match_diff parameter which specifies a maximal distance
in pixels along the row and column directions (not a true geometric distance)
for two astrometric chip detections to be considered a match.
2) Discard stars which are too close to defects.
Every star closer than bc_distance to a known chip defect
is ignored.
3) Find the above stars on photometric images. For each chip in a camera column (i.e. for each filter), the coordinates of stamps are calculated from the positions in the leading astrometric chip and known chip-to-chip astrometric transformations.
4) Cut out stamps from photometric images.
Normally, the pipeline cuts out postage stamps for stars in the
photometric chips based solely upon predictions of their positions
from the astrometric chips; however, if the user sets
st_recenter_boxsize to a value greater than 0, a
stamp is cut out at the predicted position, then its intensity-weighted
first moment of pixel values is calculated (for pixels in a box with
st_recenter_boxsize half-length, centered at the stamp),
and then the stamp is re-cut based on the new position.
5) Discard stars which are too close to defects.
Every star closer than bc_distance to a known chip defect
is ignored.
6) Measure star parameters.
In order to avoid any systematic differences between the
positions of astrometric detections (which are measured
by the on-line DA) and instances of the same stars on
the photometric chips (which are measured by SSC),
the SSC links in a routine called
atDACentroidFind from
ASTROTOOLS
to perform the calculations of the center, width, peak, and integrated
brightness of stars.
It convolves the stamp with a gaussian of width st_conv_sigma
before calculating those quantities. The results aren't
very sensitive to the value of st_conv_sigma, but it should
be comparable to the width of a gaussian model of the PSF.
SSC can perform these calculations on postage stamps which are
ff parameters are used to control the method by which
photometric postage stamps are flat-fielded before having their
parameters measured. Note that the flatfielded versions are
not saved and passed into SSC output files. Instead, the original,
raw versions are sent to the downstream pipelines. However,
the SSC output does contain the flatfield values (if any) which
were used to perform the flatfielding, in case the later pipelines
wish to use the same values.
The sk group of parameters control the manner in which the
sky value is calculated for each photometric postage stamp. Only
pixels lying within sk_edge_npix pixels of an edge of the
stamp are considered; that set of values is subjected to
sk_clip_niter iterations of a sigma-clipping technique, in
which the mean and standard deviation are calculated, and values more
than sk_clip_nsigma stdev from the mean are discarded. The
mean value of the final iteration is taken to be the sky for that
stamp.
7) Mark stars with "bad" parameters.
The ds group of parameters serve to dis-qualify certain
photometric stamps. Those whose measured parameters fail to fall
within the given ranges for radius or peak value, are marked as "bad".
If the user sets the ds_xxx_delete parameter(s) to 1, then
stars which fail the xxx test will be deleted from the list of valid
postage stamps. Stamps are checked only in passbands listed in
ds_filterlist, so by leaving this list empty,
one allows all stars to pass through.
The wing and frame stars are so bright that they ought to have perceptible wings extending outwards for many arcseconds or even arcminutes from their centers. The photometric pipelines use them to derive the shape and amplitude of such extended wings, relative to the central peak of a star. The SSC uses pre-existing stellar catalogs to predict the positions of wing and frame stars on the photometric images (magnitude ranges are specified in scParam.par). Once a wing stamp is cut out at the predicted position, its intensity-weighted first moment of pixel values is calculated, and then the stamp is re-cut based on the new position. No recentering is attempted for frame stamps.
The selection of wing stars is controlled by the ws group of
input parameters. If a frame has more than one qualifying star, none are
selected. The user must set the magnitude limits appropriately to
acquire the desired number of stars per run. Stars which are more than
ws_pix_offset from the center of a frame are ignored.
The fs group of parameters are used to select frame
stars. These stars are even brighter than wing stars, and are intended
to help characterize the ghost structure of the PSF. If a frame has
more than one qualifying star, none are selected. The user must set
the magnitude limits appropriately to acquire the desired (tiny) number
of stars per run. Stars which are more than fs_pix_offset
from the center of a frame are ignored.
The SSC can handle raw, photometric images files in several ways:
first, if ignorePhotomData is set to 1, the SSC ignores
all photometric images (and Gang files) completely. If
ignorePhotomData is set to 0, then the SSC will look
for photometric images either on disk or on tape:
fieldInputDevice to
disk, then the SSC looks in the fieldInputDir
directory for images.
fieldInputDevice
to tape, then the SSC expects to find the images on a tape
(e.g. the first processing of DLT tapes at Fermilab).
The tape must be mounted in a drive with device name given
by tapeDeviceName. The tape must have an ASCII label at
its very beginning, with length exactly tapeLabelLength
bytes. The SSC will compare the first word in this label with the
tapeLabel value, and halt execution if the two do not match.
Following the tape label should be individual photometric images, written
one after another, field by field, 5 images per field (that is, the
re-arranging of image data analogous to Gang - Fang processing has already
been done by the DA). The SSC will check to make sure that the keywords
RUN FILTER FRAME CAMROW CAMCOLin the FITS header have the values expected for the image.
In addition, if the SSC was reading images from a tape, and
you set the keepPhotomImages parameter to 1,
then it will place very large raw photometric image files,
idR*.fit, into the fieldInputDir
directory (they are simply copied from the tape, and not modified in
any way).
If the ignorePhotomData parameter is set to 1,
then all the photometric data is ignored; this means that no
scFrame, scWing,
or idR
files will be created.
Moreover, the output scFang
files will contain only the parameters for bright stars measured
by the DA system.
$SSC_DIR/bin/sscproducing a response like so (here, '>' is the Unix shell prompt).
> $SSC_DIR/bin/ssc
Executing commands in /u/products/astrotools/v1_9/etc/astrotoolsStartup.tcl
Executing commands in /u/products/dervish/v5_0/etc/dervishStartup.tcl:
Executing commands in /u/richmond/.dervish:
Executing commands in /peyton/scr/lucifer0/richmond/ssc/etc/sscStartup.tcl:
ssc>
Now, one can either change directories to a directory containing the scPlan.par file, and then run the pipeline:
ssc> cd /data/dp1.4/sscdata/input ssc> run_ssc_pipeline
or, supply the name of the file (including a relative or absolute path) as an argument to the pipeline script:
ssc> run_ssc_pipeline /data/dp1.4/sscdata/input/scPlan.par
In either case, a large number of messages appear on your screen as the SSC works its way through the data:
ssc> run_ssc_pipeline
Entering proc init_discard_bad_photom_stars_2
entering init_calc_flats
.
.
etc.
.
Entering clean_up_process
done with clean_up_process
{ELAPSED 33.735} {CPU {{OVERALL 32.190} {UTIME 29.470} {STIME 2.720} {CUTIME 0.000} {CSTIME 0.000}}}
0
ssc>
If all goes well, the last thing printed by the SSC will be its return code, 0. If not, some sort of error occurred, and the last few lines of the SSC process describe the problem.
Examining output
If all went well, the SSC should produce two-and-a-half sorts of output.
First, the half-output: if the SSC was reading photometric
images from tape, and keepPhotomImages was set to 1,
then the directory fieldInputDir will have FITS
versions of all the photometric images which were processed.
In such cases, this will probably be much larger than
the real outputs described below.
The first real type of output will be in the output directory specified in the scPlan.par file; it should contain one or more Fang files. A typical size for a Fang file is 400-700 kB, for a camera column containing photometric chips (camCol 1-6), or about 40 kB, for a camera column with astrometric chips only (camCol 7-11). One can use the standard Dervish tools, such as fits2Schema and chainPage to examine the contents of the Fang file, one HDU at a time.
Sometimes it's nice to look at the postage stamps cut out by SSC, since the eye is so good at picking out cruddy data in a crowd. One can use fits2Schema to read one of the HDUs from a Fang file that contains postage stamps, specifying a data type of SCUSTAMP, like so: assuming that the output directory is $SSCDATA_DIR/output, one can type:
ssc> cd $SSCDATA_DIR/output
ssc> set hdr [hdrNew]
h2
ssc> set ch [fits2Schema scFang-000581-1-0026.fit SCUSTAMP $hdr -hdu 1]
h3
ssc>
And then convert that CHAIN of SCUSTAMPs into a large REGION,
consisting of a mosaic of each SCUSTAMP's data:
ssc> make_ustamp_mosaic $ch
h4
ssc> saoDisplay h4
The mosaic can be displayed or otherwise manipulated as desired.
The output directory will also contain FITS images containing any
wing stars or frame stars
that were cut out during the run.
Each file is a single FITS image, which should be either a sub-frame
centered on the star (wing stars), or
a complete copy of the raw frame (frame stars).
One can use the standard DERVISH commands for reading and displaying
images (i.e. REGIONs) to examine each file individually. For example,
assuming that one wing star is in file scWing-000505-12-0039.fit.
ssc> cd /data/dp1.4/sscdata/input ssc> set reg [regReadAsFits [regNew] scWing-000505-12-0039.fit] h1 ssc> saoDisplay $reg 1 ssc>
Finally, one can examine the
diagnostic output file
to see how many stars were detected and matched in each field, etc.
Since the file has a relatively simple ASCII format, it can be browsed
directly.
However, it may be more convenient to use the ssc_results
procedure to create a 4-panel graph showing the behavior
of various diagnostics as a function of field number.
This is probably the quickest way to find a big problem.
The SSC pipeline works in a rather straightforward way and its flow chart is in essence a linear list of executed procedures.
Initialise Modules
Read Plan/Parameter Files
Read Star Catalog
{if needed: read flat-field vectors}
Foreach field {
Locate matching astrometric Gang files
Locate matching photometric Gang files
Locate photometric images (either on disk or tape)
Find catalog stars on photometric images
Cut out wing and frame stars and write to disk
Read astrometric keywords from Gang file headers
Read quartiles from Gang files
{if needed: calculate flat-field vectors}
Find stars appearing in both astrometric Gang files
Discard stars which are too close to defects
Find the surviving stars on photometric images
Cut out stamps from photometric images
Discard stars which are too close to defects
Measure parameters for the surviving stars
Mark stars with "bad" parameters
{if needed: discard stars with "bad" parameters}
Write surviving stamps to Fang file
{if needed: write diagnostics}
}
Finish wing and frame stars log files
#################################################################
# This is the Plan file for the Serial Stamp Collecting Pipeline
#
# Required Generic Plan Fields
action ssc # Plan type
version v1.0 # Software version to use
parameters scParam.par # Tunable parameters file
#
# Optional Generic Plan Fields
priority 1 # Execution priority
oid "23.45.34.23" # Database object identifier
targetDate "1999/04/18" # Target date (local time zone)
created "1965/04/18 05:40:00" # Creation time (UT)
authors "User Abuser" # Authors
comment "example" # Comment
diagnostics 1 # if > 0, do create a file with
# diagnostic output from each field
diagFile scDiag # prefix of the file into which
# diagnostic output will be placed.
# Name will include run and camCol.
#
# filters order
ccdrow_r 1
ccdrow_i 2
ccdrow_u 3
ccdrow_z 4
ccdrow_g 5
ccdrow_o 6
ccdrow_l 7
ccdrow_t 8
ccdrow_s 9
#################################################################
# List of paths to various required files
# the param file given above as 'parameters' may be found here
parametersDir $SSCDATA_DIR/input
# directory with the CCD parameters file ...
configDir $SSCDATA_DIR/input
# ... and file names
ccdConfig opConfig-51259.par # amplif. config
ccdBC opBC-51259.par # bad column file
ccdECalib opECalib-51259.par # readnoise/gain file
ccdCamera opCamera-51259.par # camera layout
# directory with astrometric gangs
aGangInputDir $SSCDATA_DIR//Gangs
# directory with photometric gangs
pGangInputDir $SSCDATA_DIR//Gangs
# directory with photometric images
fieldInputDir $SSCDATA_DIR//idR
# info about input device
fieldInputDevice disk # disk or tape
tapeDeviceName /dev/nrmt3h # name of tape drive
tapeLabel VOL1JG0001 # if "tape", name of volume label
tapeLabelLength 80 # if "tape", length of tape label
tapeSkipFiles 50 # skip this many images at the
# very start of the tape (does not
# include the tape label)
tapeLogFile idTapeLog-JG0001.par # if "tape", name of idTapeLog file
# Directory with catalog of known objects
koDir $SSCDATA_DIR//input
koVersion v1_0 # Version of known object catalog to use.
koFileBase koCat # prefix of the catalog file
wingFile scWing # prefix for the file with the list of
# "wing stars". Name includes run and
# camcol. Ex: scWing-000581-1.par
frameFile scFrame # prefix for the file with the list of
# "frame stars". Name includes run and
# camcol. Ex: scWing-000581-1.par
# all the rest of the input comes from here
inputDir $SSCDATA_DIR//input
# put the output here:
outputDir $SSCDATA_DIR/output
#################################################################
# Processing Parameters
# which data to process
run 581 # Imaging run number.
camCol 1 # Camera column
startField 10 # start pipeline with this field
# Gang files must start with
# a number <= startField - 3
endField 66 # stop with this (inclusive)
# Gang files must finish with
# a number >= endField + 10
# what to do with the data
keepPhotomImages 1 # if 1, keep photometric images on disk
# if 0, delete them after use
# this is probably 1 if using "disk"
# but possibly 0 if using "tape"
writePhotomOnly 0 # if 1, read photometric images off
# tape, write to disk, do nothing else
ignorePhotomData 0 # if 1, ignore any photometric data,
# working only on astrometric data
# aux stuff
verbose 1 # verbose > 0 means print a lot
display 0 # display on/off; shows various images
# as the pipeline progresses, waiting
# for user input after displaying each one
#
# Software parameters for SSC
#
# matching stars
am_xpos_field rowCentroid # name of field (from SCPARAMBS structure) used
# to compare astrometric stars in one direction
am_ypos_field colCentroid # name of field (from SCPARAMBS structure) used
# to compare astrometric stars in other direction
am_match_diff 8 # two astrometric stars with centers which are
# this close (or closer), in pixels in each
# direction (not in true geometric distance)
# are considered a match
# defining flat-fielding method
ff_none 0 # perform no bias-subtraction or flatfielding
ff_quartiles 1 # use a flatfield vector based on the quartiles
ff_fixed 0 # use a fixed flatfield vector in a disk file
ff_filename scFlat_ # base of the filename inside which the flatfield
# vector for each chip lives (as a FITS image).
# Ex: scFlat_42.fit, 42=camera row and column
ff_maxncol 2128 # max number of cols in any photometric chip
# discarding bad data
bc_distance 10 # if a star is this close (pixels) to
# a known chip defect, ignore it
# calculating sky
sk_edge_npix 10 # use pixels within this many of edge
# of a postage stamp to calc sky
sk_clip_nsigma 3.0 # clip values this many sigma from mean
# when calculating sky value in stamp
sk_clip_niter 2 # make this many iterations in clipping
# when calculating sky value in stamp
st_recenter_boxsize 11 # if > 0, calc center of a star using
# first moments inside a box with this
# half-length at center of stamp, and
# re-cut stamp at the new position.
# if == 0, do not re-cut stamp; keep stamp
# based on position in the astrometric chips
st_conv_sigma 1.2 # width (pix) of gaussian with which to
# convolve stamp before measuring its properties
# discarding instructions
ds_statusfit_delete 0 # if 1, discard stars which have "statusfit"
# field != 0
ds_filterlist r g # test stars only in these filters for valid
# parameters
ds_minrmajor 0.9 # rMajor must be > this to be OK
ds_maxrmajor 1.6 # rMajor must be < this to be OK
ds_minrminor 0.9 # rMinor must be > this to be OK
ds_maxrminor 1.6 # rMinor must be < this to be OK
ds_minpeak 100.0 # peak must be > this
ds_maxpeak 10000.0 # peak must be < this
ds_radius_bit 1 # set this bit in the "sscstatus"
# field of SCPARAMBS if any radius
# length is outside allowed range
ds_radius_delete 0 # if 1, discard stars which fail radius tests
ds_peak_bit 2 # set this bit in the "sscstatus" field of SCPARAMBS
# if peak is outside allowed range
ds_peak_delete 0 # if 1, discard stars which fail peak tests
# "wing" stars parameters
ws_pix_offset 400 # max number of pixels away from center
# of its CCD chip for a "wing star"
ws_min_mag 8.0 # minimum mag allowed for "wing star"
# (in whatever passband is in catalog)
ws_max_mag 14.0 # maximum mag allowed for "wing star"
ws_box_size 200 # size of box, in pixels, which we cut
# "frame" stars parameters
fs_pix_offset 300 # max number of pixels away from center
# of its CCD chip for a "frame star"
fs_min_mag 1.0 # minimum mag allowed for "frame star"
# (in whatever passband is in catalog)
fs_max_mag 8.0 # maximum mag allowed for "frame star"
Please send any comments or suggestions to Zeljko Ivezic