The on-line Data Acquisition (DA) system reads out each CCD chip in the imaging camera simultaneously, and groups together data (e.g. quartiles) from all the images, which are sometimes referred to as frames, into a Gang file. Since it takes about 10 minutes for a star to drift across the focal plane and trail over each CCD chip, the data in a Gang file correspond to the same read-out time, rather than to the same area of sky. The SSC re-arranges this data and produces a new Fang file, which looks very much like a Gang file, but has information for the same area of sky, a field, in each passband all together. Additionaly, the SSC pipeline is used to read raw photometric images from tape, and to write them as idR files to disk.
The downstream photometric pipeline is divided into two smaller pipelines, the Postage Stamp Pipeline (PSP), and Frames Pipeline. Frames Pipeline operates on a frame by frame basis (the photometric data stream from each CCD in the photometric array is cut into frames of 2048 x 1362 pixels). The quantities such as the point spread function, the sky brightness, and the flat field and bias vectors which are not local quantities, but are determined as a function of time, are determined by the PSP.
The PSP uses information form Fang files to calculate the sky brightness, and the flat field and bias vectors. To determine the point spread function (PSF), it uses small regions, called postage stamps, which are cut out from large image frames by the SSC pipeline, and also written to Fang files. Various types of postage stamps, and details of the cutting procedure are described in section Algorithms .
There are two main ways one can run the SSC: simply to read images from
tape and write onto disk, or to read and process Gang files,
images, etc. If you set the writePhotomOnly
keyword in the scPlan.par file to 1, then the SSC will do
nothing but read photometric images from tape and write them to disk.
In that case, the only outputs are FITS images
idR*.fit.
Assuming that writePhotomOnly
is set to 0, the SSC produces
the following output:
As already said in Introduction, the main SSC pipeline tasks
are to rearrange data from files corresponding to a given read-out time
to files containing data for a given area on sky, and to produce postage stamps.
This section describes these steps in more details. Names printed in typeset
font
are input parameters specified by user in scPlan.par
and scParam.par files.
The on-line DA system reads out each CCD chip in the imaging camera simultaneously, and groups together data (e.g. quartiles) from all the images read out at one time into a Gang file. Since it takes about 10 minutes for a star to drift across the focal plane and trail over each CCD chip, the data in a Gang file do not correspond to the same area of sky. For example, the layout of the imaging camera shows that a star which appears in camera column 2 will first drift across the leading astrometric chip (number 72), then the r'-band chip (12), then the i'-, u'-, z'- and g'-band chips (22-52), and finally leave by moving across the trailing astrometric chip (82). The way the on-line DA arranges things, this star would appear in 7 different Gang files; for example,
The leading astrometric detection in Gang file 0020 The r'-band detection in Gang file 0023 The i'-band detection in Gang file 0025 The u'-band detection in Gang file 0027 The z'-band detection in Gang file 0029 The g'-band detection in Gang file 0031 The trailing astrometric detection in Gang file 0033
One of the primary purposes of the SSC is to re-arrange the information gathered by the DA so that all the data pertaining to some area of the sky is collected into a single file. This Fang file therefore looks very much like a Gang file, but has information on stars in each passband (and astrometric chip) all together. To continue the example shown above, the Fang file for field 23 would contain data from each of the detections of the same star.
1) Find stars appearing on both astrometric chips.
The positions are taken from astrometric Gang files. Matching is controlled
by am_match_diff
parameter which specifies a maximal distance
in pixels along the row and column directions (not a true geometric distance)
for two astrometric chip detections to be considered a match.
2) Discard stars which are too close to defects.
Every star closer than bc_distance
to a known chip defect
is ignored.
3) Find the above stars on photometric images. For each chip in a camera column (i.e. for each filter), the coordinates of stamps are calculated from the positions in the leading astrometric chip and known chip-to-chip astrometric transformations.
4) Cut out stamps from photometric images.
Normally, the pipeline cuts out postage stamps for stars in the
photometric chips based solely upon predictions of their positions
from the astrometric chips; however, if the user sets
st_recenter_boxsize
to a value greater than 0, a
stamp is cut out at the predicted position, then its intensity-weighted
first moment of pixel values is calculated (for pixels in a box with
st_recenter_boxsize
half-length, centered at the stamp),
and then the stamp is re-cut based on the new position.
5) Discard stars which are too close to defects.
Every star closer than bc_distance
to a known chip defect
is ignored.
6) Measure star parameters.
In order to avoid any systematic differences between the
positions of astrometric detections (which are measured
by the on-line DA) and instances of the same stars on
the photometric chips (which are measured by SSC),
the SSC links in a routine called
atDACentroidFind from
ASTROTOOLS
to perform the calculations of the center, width, peak, and integrated
brightness of stars.
It convolves the stamp with a gaussian of width st_conv_sigma
before calculating those quantities. The results aren't
very sensitive to the value of st_conv_sigma
, but it should
be comparable to the width of a gaussian model of the PSF.
SSC can perform these calculations on postage stamps which are
ff
parameters are used to control the method by which
photometric postage stamps are flat-fielded before having their
parameters measured. Note that the flatfielded versions are
not saved and passed into SSC output files. Instead, the original,
raw versions are sent to the downstream pipelines. However,
the SSC output does contain the flatfield values (if any) which
were used to perform the flatfielding, in case the later pipelines
wish to use the same values.
The sk
group of parameters control the manner in which the
sky value is calculated for each photometric postage stamp. Only
pixels lying within sk_edge_npix
pixels of an edge of the
stamp are considered; that set of values is subjected to
sk_clip_niter
iterations of a sigma-clipping technique, in
which the mean and standard deviation are calculated, and values more
than sk_clip_nsigma
stdev from the mean are discarded. The
mean value of the final iteration is taken to be the sky for that
stamp.
7) Mark stars with "bad" parameters.
The ds
group of parameters serve to dis-qualify certain
photometric stamps. Those whose measured parameters fail to fall
within the given ranges for radius or peak value, are marked as "bad".
If the user sets the ds_xxx_delete
parameter(s) to 1, then
stars which fail the xxx test will be deleted from the list of valid
postage stamps. Stamps are checked only in passbands listed in
ds_filterlist
, so by leaving this list empty,
one allows all stars to pass through.
The wing and frame stars are so bright that they ought to have perceptible wings extending outwards for many arcseconds or even arcminutes from their centers. The photometric pipelines use them to derive the shape and amplitude of such extended wings, relative to the central peak of a star. The SSC uses pre-existing stellar catalogs to predict the positions of wing and frame stars on the photometric images (magnitude ranges are specified in scParam.par). Once a wing stamp is cut out at the predicted position, its intensity-weighted first moment of pixel values is calculated, and then the stamp is re-cut based on the new position. No recentering is attempted for frame stamps.
The selection of wing stars is controlled by the ws
group of
input parameters. If a frame has more than one qualifying star, none are
selected. The user must set the magnitude limits appropriately to
acquire the desired number of stars per run. Stars which are more than
ws_pix_offset
from the center of a frame are ignored.
The fs
group of parameters are used to select frame
stars. These stars are even brighter than wing stars, and are intended
to help characterize the ghost structure of the PSF. If a frame has
more than one qualifying star, none are selected. The user must set
the magnitude limits appropriately to acquire the desired (tiny) number
of stars per run. Stars which are more than fs_pix_offset
from the center of a frame are ignored.
The SSC can handle raw, photometric images files in several ways:
first, if ignorePhotomData
is set to 1, the SSC ignores
all photometric images (and Gang files) completely. If
ignorePhotomData
is set to 0, then the SSC will look
for photometric images either on disk or on tape:
fieldInputDevice
to
disk, then the SSC looks in the fieldInputDir
directory for images.
fieldInputDevice
to tape, then the SSC expects to find the images on a tape
(e.g. the first processing of DLT tapes at Fermilab).
The tape must be mounted in a drive with device name given
by tapeDeviceName
. The tape must have an ASCII label at
its very beginning, with length exactly tapeLabelLength
bytes. The SSC will compare the first word in this label with the
tapeLabel
value, and halt execution if the two do not match.
Following the tape label should be individual photometric images, written
one after another, field by field, 5 images per field (that is, the
re-arranging of image data analogous to Gang - Fang processing has already
been done by the DA). The SSC will check to make sure that the keywords
RUN FILTER FRAME CAMROW CAMCOLin the FITS header have the values expected for the image.
In addition, if the SSC was reading images from a tape, and
you set the keepPhotomImages
parameter to 1,
then it will place very large raw photometric image files,
idR*.fit, into the fieldInputDir
directory (they are simply copied from the tape, and not modified in
any way).
If the ignorePhotomData
parameter is set to 1,
then all the photometric data is ignored; this means that no
scFrame, scWing,
or idR
files will be created.
Moreover, the output scFang
files will contain only the parameters for bright stars measured
by the DA system.
$SSC_DIR/bin/sscproducing a response like so (here, '>' is the Unix shell prompt).
> $SSC_DIR/bin/ssc Executing commands in /u/products/astrotools/v1_9/etc/astrotoolsStartup.tcl Executing commands in /u/products/dervish/v5_0/etc/dervishStartup.tcl: Executing commands in /u/richmond/.dervish: Executing commands in /peyton/scr/lucifer0/richmond/ssc/etc/sscStartup.tcl: ssc>
Now, one can either change directories to a directory containing the scPlan.par file, and then run the pipeline:
ssc> cd /data/dp1.4/sscdata/input ssc> run_ssc_pipeline
or, supply the name of the file (including a relative or absolute path) as an argument to the pipeline script:
ssc> run_ssc_pipeline /data/dp1.4/sscdata/input/scPlan.par
In either case, a large number of messages appear on your screen as the SSC works its way through the data:
ssc> run_ssc_pipeline Entering proc init_discard_bad_photom_stars_2 entering init_calc_flats . . etc. . Entering clean_up_process done with clean_up_process {ELAPSED 33.735} {CPU {{OVERALL 32.190} {UTIME 29.470} {STIME 2.720} {CUTIME 0.000} {CSTIME 0.000}}} 0 ssc>
If all goes well, the last thing printed by the SSC will be its return code, 0. If not, some sort of error occurred, and the last few lines of the SSC process describe the problem.
Examining output
If all went well, the SSC should produce two-and-a-half sorts of output.
First, the half-output: if the SSC was reading photometric
images from tape, and keepPhotomImages
was set to 1,
then the directory fieldInputDir
will have FITS
versions of all the photometric images which were processed.
In such cases, this will probably be much larger than
the real outputs described below.
The first real type of output will be in the output directory specified in the scPlan.par file; it should contain one or more Fang files. A typical size for a Fang file is 400-700 kB, for a camera column containing photometric chips (camCol 1-6), or about 40 kB, for a camera column with astrometric chips only (camCol 7-11). One can use the standard Dervish tools, such as fits2Schema and chainPage to examine the contents of the Fang file, one HDU at a time.
Sometimes it's nice to look at the postage stamps cut out by SSC, since the eye is so good at picking out cruddy data in a crowd. One can use fits2Schema to read one of the HDUs from a Fang file that contains postage stamps, specifying a data type of SCUSTAMP, like so: assuming that the output directory is $SSCDATA_DIR/output, one can type:
ssc> cd $SSCDATA_DIR/output ssc> set hdr [hdrNew] h2 ssc> set ch [fits2Schema scFang-000581-1-0026.fit SCUSTAMP $hdr -hdu 1] h3 ssc>And then convert that CHAIN of SCUSTAMPs into a large REGION, consisting of a mosaic of each SCUSTAMP's data:
ssc> make_ustamp_mosaic $ch h4 ssc> saoDisplay h4The mosaic can be displayed or otherwise manipulated as desired.
The output directory will also contain FITS images containing any
wing stars or frame stars
that were cut out during the run.
Each file is a single FITS image, which should be either a sub-frame
centered on the star (wing stars), or
a complete copy of the raw frame (frame stars).
One can use the standard DERVISH commands for reading and displaying
images (i.e. REGIONs) to examine each file individually. For example,
assuming that one wing star is in file scWing-000505-12-0039.fit
.
ssc> cd /data/dp1.4/sscdata/input ssc> set reg [regReadAsFits [regNew] scWing-000505-12-0039.fit] h1 ssc> saoDisplay $reg 1 ssc>
Finally, one can examine the
diagnostic output file
to see how many stars were detected and matched in each field, etc.
Since the file has a relatively simple ASCII format, it can be browsed
directly.
However, it may be more convenient to use the ssc_results
procedure to create a 4-panel graph showing the behavior
of various diagnostics as a function of field number.
This is probably the quickest way to find a big problem.
The SSC pipeline works in a rather straightforward way and its flow chart is in essence a linear list of executed procedures.
Initialise Modules Read Plan/Parameter Files Read Star Catalog {if needed: read flat-field vectors} Foreach field { Locate matching astrometric Gang files Locate matching photometric Gang files Locate photometric images (either on disk or tape) Find catalog stars on photometric images Cut out wing and frame stars and write to disk Read astrometric keywords from Gang file headers Read quartiles from Gang files {if needed: calculate flat-field vectors} Find stars appearing in both astrometric Gang files Discard stars which are too close to defects Find the surviving stars on photometric images Cut out stamps from photometric images Discard stars which are too close to defects Measure parameters for the surviving stars Mark stars with "bad" parameters {if needed: discard stars with "bad" parameters} Write surviving stamps to Fang file {if needed: write diagnostics} } Finish wing and frame stars log files
################################################################# # This is the Plan file for the Serial Stamp Collecting Pipeline # # Required Generic Plan Fields action ssc # Plan type version v1.0 # Software version to use parameters scParam.par # Tunable parameters file # # Optional Generic Plan Fields priority 1 # Execution priority oid "23.45.34.23" # Database object identifier targetDate "1999/04/18" # Target date (local time zone) created "1965/04/18 05:40:00" # Creation time (UT) authors "User Abuser" # Authors comment "example" # Comment diagnostics 1 # if > 0, do create a file with # diagnostic output from each field diagFile scDiag # prefix of the file into which # diagnostic output will be placed. # Name will include run and camCol. # # filters order ccdrow_r 1 ccdrow_i 2 ccdrow_u 3 ccdrow_z 4 ccdrow_g 5 ccdrow_o 6 ccdrow_l 7 ccdrow_t 8 ccdrow_s 9 ################################################################# # List of paths to various required files # the param file given above as 'parameters' may be found here parametersDir $SSCDATA_DIR/input # directory with the CCD parameters file ... configDir $SSCDATA_DIR/input # ... and file names ccdConfig opConfig-51259.par # amplif. config ccdBC opBC-51259.par # bad column file ccdECalib opECalib-51259.par # readnoise/gain file ccdCamera opCamera-51259.par # camera layout # directory with astrometric gangs aGangInputDir $SSCDATA_DIR//Gangs # directory with photometric gangs pGangInputDir $SSCDATA_DIR//Gangs # directory with photometric images fieldInputDir $SSCDATA_DIR//idR # info about input device fieldInputDevice disk # disk or tape tapeDeviceName /dev/nrmt3h # name of tape drive tapeLabel VOL1JG0001 # if "tape", name of volume label tapeLabelLength 80 # if "tape", length of tape label tapeSkipFiles 50 # skip this many images at the # very start of the tape (does not # include the tape label) tapeLogFile idTapeLog-JG0001.par # if "tape", name of idTapeLog file # Directory with catalog of known objects koDir $SSCDATA_DIR//input koVersion v1_0 # Version of known object catalog to use. koFileBase koCat # prefix of the catalog file wingFile scWing # prefix for the file with the list of # "wing stars". Name includes run and # camcol. Ex: scWing-000581-1.par frameFile scFrame # prefix for the file with the list of # "frame stars". Name includes run and # camcol. Ex: scWing-000581-1.par # all the rest of the input comes from here inputDir $SSCDATA_DIR//input # put the output here: outputDir $SSCDATA_DIR/output ################################################################# # Processing Parameters # which data to process run 581 # Imaging run number. camCol 1 # Camera column startField 10 # start pipeline with this field # Gang files must start with # a number <= startField - 3 endField 66 # stop with this (inclusive) # Gang files must finish with # a number >= endField + 10 # what to do with the data keepPhotomImages 1 # if 1, keep photometric images on disk # if 0, delete them after use # this is probably 1 if using "disk" # but possibly 0 if using "tape" writePhotomOnly 0 # if 1, read photometric images off # tape, write to disk, do nothing else ignorePhotomData 0 # if 1, ignore any photometric data, # working only on astrometric data # aux stuff verbose 1 # verbose > 0 means print a lot display 0 # display on/off; shows various images # as the pipeline progresses, waiting # for user input after displaying each one
# # Software parameters for SSC # # matching stars am_xpos_field rowCentroid # name of field (from SCPARAMBS structure) used # to compare astrometric stars in one direction am_ypos_field colCentroid # name of field (from SCPARAMBS structure) used # to compare astrometric stars in other direction am_match_diff 8 # two astrometric stars with centers which are # this close (or closer), in pixels in each # direction (not in true geometric distance) # are considered a match # defining flat-fielding method ff_none 0 # perform no bias-subtraction or flatfielding ff_quartiles 1 # use a flatfield vector based on the quartiles ff_fixed 0 # use a fixed flatfield vector in a disk file ff_filename scFlat_ # base of the filename inside which the flatfield # vector for each chip lives (as a FITS image). # Ex: scFlat_42.fit, 42=camera row and column ff_maxncol 2128 # max number of cols in any photometric chip # discarding bad data bc_distance 10 # if a star is this close (pixels) to # a known chip defect, ignore it # calculating sky sk_edge_npix 10 # use pixels within this many of edge # of a postage stamp to calc sky sk_clip_nsigma 3.0 # clip values this many sigma from mean # when calculating sky value in stamp sk_clip_niter 2 # make this many iterations in clipping # when calculating sky value in stamp st_recenter_boxsize 11 # if > 0, calc center of a star using # first moments inside a box with this # half-length at center of stamp, and # re-cut stamp at the new position. # if == 0, do not re-cut stamp; keep stamp # based on position in the astrometric chips st_conv_sigma 1.2 # width (pix) of gaussian with which to # convolve stamp before measuring its properties # discarding instructions ds_statusfit_delete 0 # if 1, discard stars which have "statusfit" # field != 0 ds_filterlist r g # test stars only in these filters for valid # parameters ds_minrmajor 0.9 # rMajor must be > this to be OK ds_maxrmajor 1.6 # rMajor must be < this to be OK ds_minrminor 0.9 # rMinor must be > this to be OK ds_maxrminor 1.6 # rMinor must be < this to be OK ds_minpeak 100.0 # peak must be > this ds_maxpeak 10000.0 # peak must be < this ds_radius_bit 1 # set this bit in the "sscstatus" # field of SCPARAMBS if any radius # length is outside allowed range ds_radius_delete 0 # if 1, discard stars which fail radius tests ds_peak_bit 2 # set this bit in the "sscstatus" field of SCPARAMBS # if peak is outside allowed range ds_peak_delete 0 # if 1, discard stars which fail peak tests # "wing" stars parameters ws_pix_offset 400 # max number of pixels away from center # of its CCD chip for a "wing star" ws_min_mag 8.0 # minimum mag allowed for "wing star" # (in whatever passband is in catalog) ws_max_mag 14.0 # maximum mag allowed for "wing star" ws_box_size 200 # size of box, in pixels, which we cut # "frame" stars parameters fs_pix_offset 300 # max number of pixels away from center # of its CCD chip for a "frame star" fs_min_mag 1.0 # minimum mag allowed for "frame star" # (in whatever passband is in catalog) fs_max_mag 8.0 # maximum mag allowed for "frame star"
Please send any comments or suggestions to Zeljko Ivezic