The Serial Stamp Collecting Pipeline


Table of Contents

  • Introduction
  • Inputs
  • Outputs
  • Algorithms
  • Running the SSC
  • Flow charts
  • This file is somewhat abridged compilation of documents written by Michael Richmond, the main developer of this pipeline. More detailed descriptions can be found in html files distributed with the code (in $SSC/doc/www, where $SSC is the installation directory for the SSC pipeline).

    Introduction to the Serial Stamp Collecting Pipeline

    Two main jobs of the SSC are

    The on-line Data Acquisition (DA) system reads out each CCD chip in the imaging camera simultaneously, and groups together data (e.g. quartiles) from all the images, which are sometimes referred to as frames, into a Gang file. Since it takes about 10 minutes for a star to drift across the focal plane and trail over each CCD chip, the data in a Gang file correspond to the same read-out time, rather than to the same area of sky. The SSC re-arranges this data and produces a new Fang file, which looks very much like a Gang file, but has information for the same area of sky, a field, in each passband all together. Additionaly, the SSC pipeline is used to read raw photometric images from tape, and to write them as idR files to disk.

    The downstream photometric pipeline is divided into two smaller pipelines, the Postage Stamp Pipeline (PSP), and Frames Pipeline. Frames Pipeline operates on a frame by frame basis (the photometric data stream from each CCD in the photometric array is cut into frames of 2048 x 1362 pixels). The quantities such as the point spread function, the sky brightness, and the flat field and bias vectors which are not local quantities, but are determined as a function of time, are determined by the PSP.

    The PSP uses information form Fang files to calculate the sky brightness, and the flat field and bias vectors. To determine the point spread function (PSF), it uses small regions, called postage stamps, which are cut out from large image frames by the SSC pipeline, and also written to Fang files. Various types of postage stamps, and details of the cutting procedure are described in section Algorithms .


    Serial Stamp Collecting Pipeline Inputs

    The relevant inputs to the SSC are: SSC finds the location of all input files by reading scPlan.par file. A detailed description of all input files is given as a part of Survey Interface File Formats Web page. These files include (an * stands for a string describing run, filter, camera row and frame sequence number):

    Serial Stamp Collecting Pipeline Outputs

    There are two main ways one can run the SSC: simply to read images from tape and write onto disk, or to read and process Gang files, images, etc. If you set the writePhotomOnly keyword in the scPlan.par file to 1, then the SSC will do nothing but read photometric images from tape and write them to disk. In that case, the only outputs are FITS images idR*.fit. Assuming that writePhotomOnly is set to 0, the SSC produces the following output:


    Serial Stamp Collecting Pipeline Algorithms

    As already said in Introduction, the main SSC pipeline tasks are to rearrange data from files corresponding to a given read-out time to files containing data for a given area on sky, and to produce postage stamps. This section describes these steps in more details. Names printed in typeset font are input parameters specified by user in scPlan.par and scParam.par files.

    Producing Fang files from Gang files

    The on-line DA system reads out each CCD chip in the imaging camera simultaneously, and groups together data (e.g. quartiles) from all the images read out at one time into a Gang file. Since it takes about 10 minutes for a star to drift across the focal plane and trail over each CCD chip, the data in a Gang file do not correspond to the same area of sky. For example, the layout of the imaging camera shows that a star which appears in camera column 2 will first drift across the leading astrometric chip (number 72), then the r'-band chip (12), then the i'-, u'-, z'- and g'-band chips (22-52), and finally leave by moving across the trailing astrometric chip (82). The way the on-line DA arranges things, this star would appear in 7 different Gang files; for example,

       The leading  astrometric detection in Gang file 0020
       The r'-band              detection in Gang file 0023
       The i'-band              detection in Gang file 0025
       The u'-band              detection in Gang file 0027
       The z'-band              detection in Gang file 0029
       The g'-band              detection in Gang file 0031
       The trailing astrometric detection in Gang file 0033
    

    One of the primary purposes of the SSC is to re-arrange the information gathered by the DA so that all the data pertaining to some area of the sky is collected into a single file. This Fang file therefore looks very much like a Gang file, but has information on stars in each passband (and astrometric chip) all together. To continue the example shown above, the Fang file for field 23 would contain data from each of the detections of the same star.

    Cutting postage stamps

    Stamps, which are small regions cut out from large frames, can have three "flavors". The smallest (65x65 pixels) stamps contain a bright object detected by both leading and trailing astrometric chips. There are about 10-100 such stamps per frame. "Wing" stamps are somewhat larger (200x200 pixels) and contain very bright stars (those that saturate CCDs, about 8-14 mag) whose list is provided separately. These stars are used to constrain the outer "wing" part of the PSF. The third type of stamps, "frame" stars, are indeed full frames and contain extremely bright stars (brighter than ~8 mag) which are used to characterize the ghost structures. There are typically several "wing" stars and about one "frame" star per run. The cutting of small stamps, those whose positions are based on astrometric chips detections, includes the following steps:

    1) Find stars appearing on both astrometric chips. The positions are taken from astrometric Gang files. Matching is controlled by am_match_diff parameter which specifies a maximal distance in pixels along the row and column directions (not a true geometric distance) for two astrometric chip detections to be considered a match.

    2) Discard stars which are too close to defects. Every star closer than bc_distance to a known chip defect is ignored.

    3) Find the above stars on photometric images. For each chip in a camera column (i.e. for each filter), the coordinates of stamps are calculated from the positions in the leading astrometric chip and known chip-to-chip astrometric transformations.

    4) Cut out stamps from photometric images. Normally, the pipeline cuts out postage stamps for stars in the photometric chips based solely upon predictions of their positions from the astrometric chips; however, if the user sets st_recenter_boxsize to a value greater than 0, a stamp is cut out at the predicted position, then its intensity-weighted first moment of pixel values is calculated (for pixels in a box with st_recenter_boxsize half-length, centered at the stamp), and then the stamp is re-cut based on the new position.

    5) Discard stars which are too close to defects. Every star closer than bc_distance to a known chip defect is ignored.

    6) Measure star parameters. In order to avoid any systematic differences between the positions of astrometric detections (which are measured by the on-line DA) and instances of the same stars on the photometric chips (which are measured by SSC), the SSC links in a routine called atDACentroidFind from ASTROTOOLS to perform the calculations of the center, width, peak, and integrated brightness of stars. It convolves the stamp with a gaussian of width st_conv_sigma before calculating those quantities. The results aren't very sensitive to the value of st_conv_sigma, but it should be comparable to the width of a gaussian model of the PSF.

    SSC can perform these calculations on postage stamps which are

    There are 2 sets of input parameters specified in parameter file scParam.par which control calculations of star parameters. The ff parameters are used to control the method by which photometric postage stamps are flat-fielded before having their parameters measured. Note that the flatfielded versions are not saved and passed into SSC output files. Instead, the original, raw versions are sent to the downstream pipelines. However, the SSC output does contain the flatfield values (if any) which were used to perform the flatfielding, in case the later pipelines wish to use the same values.

    The sk group of parameters control the manner in which the sky value is calculated for each photometric postage stamp. Only pixels lying within sk_edge_npix pixels of an edge of the stamp are considered; that set of values is subjected to sk_clip_niter iterations of a sigma-clipping technique, in which the mean and standard deviation are calculated, and values more than sk_clip_nsigma stdev from the mean are discarded. The mean value of the final iteration is taken to be the sky for that stamp.

    7) Mark stars with "bad" parameters. The ds group of parameters serve to dis-qualify certain photometric stamps. Those whose measured parameters fail to fall within the given ranges for radius or peak value, are marked as "bad". If the user sets the ds_xxx_delete parameter(s) to 1, then stars which fail the xxx test will be deleted from the list of valid postage stamps. Stamps are checked only in passbands listed in ds_filterlist, so by leaving this list empty, one allows all stars to pass through.

    Cutting wing and frame stamps

    The wing and frame stars are so bright that they ought to have perceptible wings extending outwards for many arcseconds or even arcminutes from their centers. The photometric pipelines use them to derive the shape and amplitude of such extended wings, relative to the central peak of a star. The SSC uses pre-existing stellar catalogs to predict the positions of wing and frame stars on the photometric images (magnitude ranges are specified in scParam.par). Once a wing stamp is cut out at the predicted position, its intensity-weighted first moment of pixel values is calculated, and then the stamp is re-cut based on the new position. No recentering is attempted for frame stamps.

    The selection of wing stars is controlled by the ws group of input parameters. If a frame has more than one qualifying star, none are selected. The user must set the magnitude limits appropriately to acquire the desired number of stars per run. Stars which are more than ws_pix_offset from the center of a frame are ignored.

    The fs group of parameters are used to select frame stars. These stars are even brighter than wing stars, and are intended to help characterize the ghost structure of the PSF. If a frame has more than one qualifying star, none are selected. The user must set the magnitude limits appropriately to acquire the desired (tiny) number of stars per run. Stars which are more than fs_pix_offset from the center of a frame are ignored.

    A note on handling photometric image files

    The SSC can handle raw, photometric images files in several ways: first, if ignorePhotomData is set to 1, the SSC ignores all photometric images (and Gang files) completely. If ignorePhotomData is set to 0, then the SSC will look for photometric images either on disk or on tape:

    In addition, if the SSC was reading images from a tape, and you set the keepPhotomImages parameter to 1, then it will place very large raw photometric image files, idR*.fit, into the fieldInputDir directory (they are simply copied from the tape, and not modified in any way).

    If the ignorePhotomData parameter is set to 1, then all the photometric data is ignored; this means that no scFrame, scWing, or idR files will be created. Moreover, the output scFang files will contain only the parameters for bright stars measured by the DA system.


    Running the Serial Stamp Collecting Pipeline

    This document assumes that the SSC pipeline and all other required SDSS products have been properly installed. For details of this procedure please refer to other SDSS documentation. Assuming that SSC's home directory is $SSC_DIR, the SSC pipeline is executed by
    $SSC_DIR/bin/ssc
    
    producing a response like so (here, '>' is the Unix shell prompt).
      > $SSC_DIR/bin/ssc
        Executing commands in /u/products/astrotools/v1_9/etc/astrotoolsStartup.tcl
        Executing commands in /u/products/dervish/v5_0/etc/dervishStartup.tcl: 
        Executing commands in /u/richmond/.dervish: 
        Executing commands in /peyton/scr/lucifer0/richmond/ssc/etc/sscStartup.tcl: 
      ssc> 
    

    Now, one can either change directories to a directory containing the scPlan.par file, and then run the pipeline:

      ssc> cd /data/dp1.4/sscdata/input
      ssc> run_ssc_pipeline
    

    or, supply the name of the file (including a relative or absolute path) as an argument to the pipeline script:

      ssc> run_ssc_pipeline /data/dp1.4/sscdata/input/scPlan.par
    

    In either case, a large number of messages appear on your screen as the SSC works its way through the data:

      ssc> run_ssc_pipeline
        Entering proc init_discard_bad_photom_stars_2 
        entering init_calc_flats
          .
          . 
          etc.
          .
        Entering clean_up_process
        done with clean_up_process 
        {ELAPSED 33.735} {CPU {{OVERALL 32.190} {UTIME 29.470} {STIME 2.720} {CUTIME 0.000} {CSTIME 0.000}}}
        0
      ssc>
    
    

    If all goes well, the last thing printed by the SSC will be its return code, 0. If not, some sort of error occurred, and the last few lines of the SSC process describe the problem.

    Examining output

    If all went well, the SSC should produce two-and-a-half sorts of output.

    First, the half-output: if the SSC was reading photometric images from tape, and keepPhotomImages was set to 1, then the directory fieldInputDir will have FITS versions of all the photometric images which were processed. In such cases, this will probably be much larger than the real outputs described below.

    The first real type of output will be in the output directory specified in the scPlan.par file; it should contain one or more Fang files. A typical size for a Fang file is 400-700 kB, for a camera column containing photometric chips (camCol 1-6), or about 40 kB, for a camera column with astrometric chips only (camCol 7-11). One can use the standard Dervish tools, such as fits2Schema and chainPage to examine the contents of the Fang file, one HDU at a time.

    Sometimes it's nice to look at the postage stamps cut out by SSC, since the eye is so good at picking out cruddy data in a crowd. One can use fits2Schema to read one of the HDUs from a Fang file that contains postage stamps, specifying a data type of SCUSTAMP, like so: assuming that the output directory is $SSCDATA_DIR/output, one can type:

      ssc> cd $SSCDATA_DIR/output
      ssc> set hdr [hdrNew]
        h2
      ssc> set ch [fits2Schema scFang-000581-1-0026.fit SCUSTAMP $hdr -hdu 1]
        h3
      ssc>
    
    And then convert that CHAIN of SCUSTAMPs into a large REGION, consisting of a mosaic of each SCUSTAMP's data:
      ssc> make_ustamp_mosaic $ch
        h4
      ssc> saoDisplay h4
    
    The mosaic can be displayed or otherwise manipulated as desired.

    The output directory will also contain FITS images containing any wing stars or frame stars that were cut out during the run. Each file is a single FITS image, which should be either a sub-frame centered on the star (wing stars), or a complete copy of the raw frame (frame stars). One can use the standard DERVISH commands for reading and displaying images (i.e. REGIONs) to examine each file individually. For example, assuming that one wing star is in file scWing-000505-12-0039.fit.

       ssc> cd /data/dp1.4/sscdata/input
       ssc> set reg [regReadAsFits [regNew] scWing-000505-12-0039.fit]
       h1
       ssc> saoDisplay $reg
       1
       ssc>
    

    Finally, one can examine the diagnostic output file to see how many stars were detected and matched in each field, etc. Since the file has a relatively simple ASCII format, it can be browsed directly. However, it may be more convenient to use the ssc_results procedure to create a 4-panel graph showing the behavior of various diagnostics as a function of field number. This is probably the quickest way to find a big problem.


    The SSC Pipeline Flow Chart

    The SSC pipeline works in a rather straightforward way and its flow chart is in essence a linear list of executed procedures.

     
    Initialise Modules
    Read Plan/Parameter Files
    Read Star Catalog
    {if needed: read flat-field vectors}
    
    Foreach field {
        Locate matching astrometric Gang files
        Locate matching photometric Gang files
        Locate photometric images (either on disk or tape)
        Find catalog stars on photometric images 
        Cut out wing and frame stars and write to disk
        Read astrometric keywords from Gang file headers 
        Read quartiles from Gang files
        {if needed: calculate flat-field vectors}
        Find stars appearing in both astrometric Gang files
        Discard stars which are too close to defects
        Find the surviving stars on photometric images 
        Cut out stamps from photometric images 
        Discard stars which are too close to defects
        Measure parameters for the surviving stars
        Mark stars with "bad" parameters
        {if needed: discard stars with "bad" parameters}       
        Write surviving stamps to Fang file
        {if needed: write diagnostics}        
    }
    
    Finish wing and frame stars log files
    
    

    Appendix


    scPlan.par: plan file for the SSC Pipeline

    This is an example of scPlan.par file which is used to supply all logistical information to the SSC pipeline (parameters which control SSC algorithms are supplied in scParam.par. All data is entered in form keyword - value. In case of multiple entries the last one counts. This example assumes that the data is stored in directory $SSCDATA_DIR.
    
    
    ################################################################# 
    # This is the Plan file for the Serial Stamp Collecting Pipeline 
    
    #
    # Required Generic Plan Fields
    action              ssc                   # Plan type
    version             v1.0                  # Software version to use
    parameters          scParam.par           # Tunable parameters file
    
    #
    # Optional Generic Plan Fields
    priority            1                     # Execution priority
    oid                 "23.45.34.23"         # Database object identifier
    targetDate          "1999/04/18"          # Target date (local time zone)
    created             "1965/04/18 05:40:00" # Creation time (UT)
    authors             "User Abuser"  	  # Authors
    comment             "example"             # Comment
    diagnostics         1                     # if > 0, do create a file with
                                              # diagnostic output from each field
    diagFile            scDiag                # prefix of the file into which 
                                              # diagnostic output will be placed.
                                              # Name will include run and camCol. 
    #                                       
    # filters order
    ccdrow_r            1
    ccdrow_i            2
    ccdrow_u            3
    ccdrow_z            4
    ccdrow_g            5
    ccdrow_o            6
    ccdrow_l            7
    ccdrow_t            8
    ccdrow_s            9
    
    #################################################################
    # List of paths to various required files
    
    # the param file given above as 'parameters' may be found here
    parametersDir 	    $SSCDATA_DIR/input
    
    # directory with the CCD parameters file ...
    configDir           $SSCDATA_DIR/input
    # ... and file names
    ccdConfig           opConfig-51259.par    # amplif. config
    ccdBC      	    opBC-51259.par	  # bad column file
    ccdECalib 	    opECalib-51259.par	  # readnoise/gain file 
    ccdCamera           opCamera-51259.par    # camera layout
    
    # directory with astrometric gangs 
    aGangInputDir       $SSCDATA_DIR//Gangs               
    
    # directory with photometric gangs  
    pGangInputDir       $SSCDATA_DIR//Gangs       
    
    # directory with photometric images  
    fieldInputDir      $SSCDATA_DIR//idR               
    
    # info about input device
    fieldInputDevice    disk                  # disk or tape
    tapeDeviceName      /dev/nrmt3h           # name of tape drive
    tapeLabel           VOL1JG0001            # if "tape", name of volume label
    tapeLabelLength     80                    # if "tape", length of tape label 
    tapeSkipFiles       50                    # skip this many images at the 
                                              # very start of the tape (does not 
                                              # include the tape label)
    tapeLogFile         idTapeLog-JG0001.par  # if "tape", name of idTapeLog file
    
    # Directory with catalog of known objects
    koDir               $SSCDATA_DIR//input 
    koVersion           v1_0                # Version of known object catalog to use. 
    koFileBase          koCat               # prefix of the catalog file  
    wingFile            scWing              # prefix for the file with the list of
                                            # "wing stars". Name includes run and 
                                            # camcol. Ex: scWing-000581-1.par
    frameFile           scFrame             # prefix for the file with the list of
                                            # "frame stars". Name includes run and 
                                            # camcol. Ex: scWing-000581-1.par 
    
    # all the rest of the input comes from here
    inputDir         $SSCDATA_DIR//input     
    
    # put the output here:
    outputDir 	    $SSCDATA_DIR/output
     
    #################################################################
    # Processing Parameters
    
    # which data to process
    run                 581                 # Imaging run number. 
    camCol   	      1		        # Camera column
    startField           10                 # start pipeline with this field
                                            # Gang files must start with 
                                            # a number <= startField - 3 
    endField             66                 # stop with this (inclusive)
                                            # Gang files must finish with 
                                            # a number >= endField + 10
    
    # what to do with the data
    keepPhotomImages      1                 # if 1, keep photometric images on disk
                                            # if 0, delete them after use
                                            # this is probably 1 if using "disk"
                                            # but possibly 0 if using "tape"
    writePhotomOnly       0                 # if 1, read photometric images off
                                            # tape, write to disk, do nothing else
    ignorePhotomData      0                 # if 1, ignore any photometric data,
                                            # working only on astrometric data
      
    # aux stuff
    verbose               1                 # verbose > 0 means print a lot
    display		      0                 # display on/off; shows various images  
                                            # as the pipeline progresses, waiting 
                                            # for user input after displaying each one
    
    

    scParam.par: parameter file for the SSC Pipeline

    This is an example of scParam.par file which specifies all parameters for controlling the SSC algorithms. All data is entered in form keyword - value.
    
    #
    # Software parameters for SSC
    #
    
    # matching stars
    am_xpos_field        rowCentroid   # name of field (from SCPARAMBS structure) used 
                                       # to compare astrometric stars in one direction
    am_ypos_field        colCentroid   # name of field (from SCPARAMBS structure) used 
                                       # to compare astrometric stars in other direction
    am_match_diff           8          # two astrometric stars with centers which are
                                       # this close (or closer), in pixels in each 
                                       # direction (not in true geometric distance)
                                       # are considered a match
    
    # defining flat-fielding method
    ff_none                 0          # perform no bias-subtraction or flatfielding
    ff_quartiles            1          # use a flatfield vector based on the quartiles                                  
    ff_fixed                0          # use a fixed flatfield vector in a disk file
    ff_filename         scFlat_        # base of the filename inside which the flatfield 
                                       # vector for each chip lives (as a FITS image).
                                       # Ex:  scFlat_42.fit, 42=camera row and column
    ff_maxncol             2128        # max number of cols in any photometric chip  
     
    # discarding bad data
    bc_distance              10        # if a star is this close (pixels) to 
                                       # a known chip defect, ignore it
    
    # calculating sky
    sk_edge_npix             10        # use pixels within this many of edge
                                       # of a postage stamp to calc sky
    sk_clip_nsigma          3.0        # clip values this many sigma from mean
                                       # when calculating sky value in stamp
    sk_clip_niter             2        # make this many iterations in clipping
                                       # when calculating sky value in stamp
    st_recenter_boxsize      11        # if > 0, calc center of a star using
                                       # first moments inside a box with this 
                                       # half-length at center of stamp, and 
                                       # re-cut stamp at the new position.
                                       # if == 0, do not re-cut stamp; keep stamp 
                                       # based on position in the astrometric chips
    st_conv_sigma           1.2        # width (pix) of gaussian with which to 
                                       # convolve stamp before measuring its properties
    
    # discarding instructions
    ds_statusfit_delete       0        # if 1, discard stars which have "statusfit" 
                                       # field != 0
    ds_filterlist           r g        # test stars only in these filters for valid 
                                       # parameters
    ds_minrmajor            0.9        # rMajor must be > this to be OK
    ds_maxrmajor            1.6        # rMajor must be < this to be OK
    ds_minrminor            0.9        # rMinor must be > this to be OK
    ds_maxrminor            1.6        # rMinor must be < this to be OK
    ds_minpeak            100.0        # peak must be > this
    ds_maxpeak          10000.0        # peak must be < this
    ds_radius_bit             1        # set this bit in the "sscstatus"
                                       # field of SCPARAMBS if any radius
                                       # length is outside allowed range
    ds_radius_delete          0        # if 1, discard stars which fail radius tests
    ds_peak_bit               2        # set this bit in the "sscstatus" field of SCPARAMBS 
                                       # if peak is outside allowed range
    ds_peak_delete            0        # if 1, discard stars which fail peak tests
    
    # "wing" stars parameters
    ws_pix_offset           400        # max number of pixels away from center
                                       # of its CCD chip for a "wing star"
    ws_min_mag              8.0        # minimum mag allowed for "wing star"
                                       # (in whatever passband is in catalog)
    ws_max_mag             14.0        # maximum mag allowed for "wing star"
    ws_box_size             200        # size of box, in pixels, which we cut
    
    # "frame" stars parameters
    fs_pix_offset           300        # max number of pixels away from center
                                       # of its CCD chip for a "frame star"
    fs_min_mag              1.0        # minimum mag allowed for "frame star"
                                       # (in whatever passband is in catalog)
    fs_max_mag              8.0        # maximum mag allowed for "frame star"
                                          
    
    
    
    

    Top of this file
    Back to Photo Review

    Please send any comments or suggestions to Zeljko Ivezic


    ZI, February 1998