The name of this pipeline comes from "stamps", small regions cut out from large, 2048 x 1362 pixels, frames. The stamps can have three "flavors". The smallest (65x65 pixels) stamps contain a bright object detected by both leading and trailing astrometric chips. There are about 10-100 such stamps per frame. "Wing" stamps are somewhat larger (200x200 pixels) and contain very bright stars (those that saturate CCDs, about 8-14 mag) whose list is provided separately. These stars are used to constrain the outer "wing" part of the PSF. The third type of stamps, "frame" stars, are indeed full frames and contain extremely bright stars (brighter than ~8 mag) which are used to characterize the ghost structures. There are typically several "wing" stars and about one "frame" star per run. Magnitude ranges for the "wing" and "frame" stars (in whatever passband is in the catalog) are specified in scParam.par , an input file for the Serial Stamp Collecting Pipeline (SSC).
The PSP tasks include:
cellprofile_file. This file contains precalculated
PSF models to be used in the PSF characterization, and can be produced
by running sdssmake.
In addition to the above FITS files which contain data to be used by downstream pipelines, there are several additional diagnostics files (mostly in PostScript format) which help in assessing the reliability of PSP outputs:
PSP tasks can be divided into 3 parts:
When the drift scanning technique is employed, pixels along the scanning direction (rows) are integrated together, and all three above quantities depend only on the chip columns (at a given time). The bias, flat-field and sky are determined on a frame by frame basis (corresponding to intervals of about 37 sec; note that the these values are mean filtered over 55 sec exposure time and over a 14 arcmin long strip on the sky). Another simplification is that the overall bias shape as a function of column is rather stable and it is sufficient to continuosly monitor only its overall shift called the bias drift. For this purpose each chip has an overclock region, a 40 columns wide strip not exposed to the incoming light.
Two kinds of input data are used for the bias, flat-field and sky determination:
Data quartiles are determined for an imaging run. They include 25%, 50% (median), and 75% quartiles of the pixel values in each column, and are evaluated by the DA system on a frame by frame basis. These values are copied by SSC pipeline from the DA output files to scFang*.fit files which are part of the PSP input.
Bias drift for each frame is evaluated as the difference between the mean 50% quartile value in the overclock region (mean over 40 columns) determined for the imaging run, and that determined from the bias vector. Overclock region columns with negative 50% quartile values (DA system's signal that the value is unreliable) are not taken into account. If a chip has two amplifiers, the bias drift is evaluated for each one separately (`left' and `right' bias drifts).
Flat-field and sky level determination assumes that only a minority of pixels are covered by objects, so that most of them are representative of the sky level. In addition, it is assumed that the sky level does not vary significantly across the frame (about 14 arcmin) and during the exposure time (about 55 sec). With these assumptions, the flat-field correction (vector) for a given frame is determined from the variation of median pixel values for each column. The mean value of these medians is taken to be the sky level for this chip and frame. Sky level error is determined from the mean difference of the 25% and 75% quartiles.
Large objects comparable in size to the whole frame can affect the pixel values distribution. For this reason, the flat-field and bias vectors, and sky values are smoothed and interpolated over a sequence of frames. Details of this procedure and parameters from psParam.par which control its performance are described elsewhere. Currently used method is a median filter about 10 frames wide, but a mean filter and a Lorentzian weighting function are also available.
This part of PSP also calculates the dark current variance from read noise and gain (the ratio of photon to electron counts) values, and the mean gain in case of 2 amplifiers. The dark current variance might eventually be determined directly from bias frames.
There are two PSF descriptions used in Photo. The first one is used for object classification in Frames and utilizes a pre-calculated catalog of models based on a sum of circularly symmetric Gaussians. In this representation, the PSF is described by the weights of these components, and indices indicating their widths, in a model which provides the best-fit to a profile characteristic of the PSF. The fitting is done on a cell by cell basis, where a cell is a union of pixels which traces both the angular and radial structure of an object's image (an object is divided into 12 angular sectors, and each sector is further divided radially into cells, with the number of pixels in each cell increasing with the radial distance; for details see the document SDSS Profile Extraction). The second representation describes the PSF in terms of best-fit model parameters determined by fitting a sum of two Gaussians and a power-law wing to the PSF profile, and is used for subtraction of bright objects (this representation handles much larger dynamic range than the first one).
As it can happen that some frames do not have enough bright stars for the PSF determination (especially in u band), several frames are combined together. The number of frames to be considered is a free parameter specified, as well as all other free input parameters which control the PSF determination, in file psParam.par.
Images of objects whose profiles are used for the PSF determination are supplied by the SSC pipeline in input files as three types of stamps. 65x65 pixel stamps are the basic type which is necessary to run the PSP. They contain moderately bright objects, usually from several to about a hundred per frame. Bright objects needed to characterize the outer part of the PSF (i.e. the power-law wing) are supplied as 200x200 pixel stamps, usually several per camera column and run. Images of extremely bright objects are supplied as whole frames. The PSP can run without the latter two types of stamps, in which case it assumes that the index of the power-law wing is -3 (after Racine 1996), and fits only its scale.
The most complex task in determining the PSF is to select objects whose profile is representative of the PSF, from all objects provided in stamps. In essence, the algorithm is based on successive clippings of `odd' profiles until a uniform sample is obtained. Such clippings can be divided into 2 groups:
As the objects are extracted from the postage stamps (i.e. counts for each cell are determined), a set of flags is set, each indicating that something is not right with that particular object. Objects that have any of the following flags set are excluded from the subsequent consideration for the PSF determination:
Statistical rejection
From the pool of objects which survived the initial rejection, a smaller
sample is defined on a frame by frame basis. It contains all the objects
recorded on a number of frames preceeding and succeeding the
current frame. For such a sample a list of statistical clippings
is performed. Performance and turning on/off each of these steps is
controlled by parameters specified in psParam.par. Here they have the same
names as in that file and are printed in typewriter font. The
clipping algorithms include and are executed in the following order:
ncellclip*sigma from a median value,
where sigma is their
quoted error for that cell, are marked. All objects that are marked in
this way in at least nbadcell cells are rejected. In addition,
objects that deviate more than maxdev*sigma from a median
in any
cell are rejected, too. The purpose of this step is to reject obviously
deviant objects.
psf_nsigma2*sigma, where
sigma is the distribution width. Note that this step does not assume
circularly symmetric PSF (although the model forms used to describe
PSF are circularly symmetric) to allow for errors in tracking and
temporal changes of the PSF.
psf_nsigma1*sigma from a distribution
median, where sigma is a distribution width.
chisq_cutoff are rejected.
If the mean deviation of the remaining objects is larger than
chisq_max, object with the largest deviation is also rejected.
This process is repeated until none of the objects deviates from the
mean more than chisq_cutoff, and until the average deviation is
less than chisq_max (presumably chisq_max <
chisq_cutoff), or until only min_starnum objects are
left.
Once a sample of 'good' objects for a given frame is formed, the best-fit pre-calculated model is determined from their mean profile. This profile is also used to calculate several photometric corrections (e.g. the aperture correction used to convert psfCounts to the counts within some aperture, currently aperture 3, with radius 4.41 pixels XXX).
The same sample of stars is used to find a composite profile. The composite profile is formed from a sample of stars which greatly vary is brightness, and has increased dynamic range. If 'wing' and 'frame' stars are available, their profiles are added to the composite profile, and a sum of 2 Gaussians and a power-law wing is fitted to it. In the case that a composite profile does not extend far enough (i.e. 'wing' and 'frame' stars are not bright enough), it is assumed that the index of power-law wing is -3 (after Racine 1996), and only its scale is fitted. In extreme cases (corresponding to the ratio of the intensity of the last point in the profile to the central intensity greater than 0.001) the power-law wing is ignored completely, and only 2 Gaussians are fitted to the composite profile.
Photometric calibration is somewhat more involved. Its task is to determine,
for each frame and color, flux20, the number of counts corresponding to a
20th magnitude object at zero airmass whose color index is refcolor
(refcolor is currently hard-coded as 0). This is done by matching
the postage stamps (after they passed individual rejection)
to a list of objects provided by the MT calibration system (which are observed
nearly simultaneously with the main survey, more details can be found in
chapter
Photometric Calibration and the Monitor Telescope
of the SDSS NASA Proposal, aka
Black Book). Postage stamps provide counts, and the MT list provides magnitudes
and colors for the objects used in calibration.
Assuming that log(flux20) varies linearly with the color index
log(flux20) = log(flux) + 0.4*(m-20) - b*color + b*refcolor - k*z,
where
This document assumes that Photo and all other required SDSS products have been properly installed. For details of this procedure please refer to other SDSS documentation. The aim of this document is to lead a user step-by-step through an execution with a lot of diagnostic output.
Assuming that Photo's home directory is $PHOTO (in the following example $PHOTO = /u/ivezic/photo), Photo is executed by
$PHOTO/bin/photo
which produces the following response ('>' is the Unix shell prompt):
>$PHOTO/bin/photo Executing commands in /u/products/IRIX/astrotools/v3_0/etc/astrotoolsStartup.tcl Executing commands in /u/products/IRIX/dervish/v6_6/etc/dervishStartup.tcl: Executing commands in /u/ivezic/photo/startup.shiva: Executing commands in /u/ivezic/photo/startup.photo: Executing commands in /u/ivezic/photo/startup.ZI: photo>In the case of using 'display' output (SAO Image),
photo> set_mtv
should be used for launching an SAO Image Display window.
Now, one can either change directories to a directory containing the psPlan.par file (e.g. $PHOTODATA_DIR), and then run the pipeline:
photo> cd $PHOTODATA_DIR photo> run_psp_pipelineor, supply the name of the file (including a relative or absolute path) as an argument to the pipeline script:
photo> run_psp_pipeline $PHOTODATA_DIR/psPlan.parThe following example is based on observations of the Hubble Deep Field by using the 3.5 m Apache Point Observatory reflector and SPICAM camera. All
display_* keywords in psPlan.par file
have been set to 1 (except display_psf=2 and display_flat=2),
and verbose to 0. For verbose > 0 more text output is
produced, with the text amount proportional to the selected value of verbose
(on a scale 1-10). All plots described below are displayed on a monitor. However, a few
of them can be also obtained in PostScript version as a part of the PSP output
.
The following plots are shown (click on the name for an example):
psf_def_rad,
defined in psParam.par file. The lower left
corner of each stamp contains two numbers: the first number is the stamp's
index in the list of all stamps, and the second one is the frame number from
which that stamp was taken.
print_flags_info
0x802 produces: The following flags are set: BADSKY, CR.
Note that stamps are never rejected because of CR (cosmic ray) flag alone.
The main diagnostics plot: there are 12 panels which show the behavior of various relevant quantities on a frame by frame basis. The name of each plotted quantity is given on top of the corresponding panel, as well as the units (if applicable). These quantities include (moving row by row from left to right, starting in the upper left corner and ending in the lower right corner):
star1_aperature specified in
psParam.par file, and counts in the adopted
PSF profile.
Initialise Modules, Read Plan/Parameter Files
Foreach filter {
Read bias vector for this run from idB*.fit file
Read data quartiles for all frames from scFang*.fit files
Foreach frame {
/* bias drift: median over rows, mean over overscan columns */
Bias drift = mean(median(imaging run) - median(bias run))
/* flat-field: median over rows */
Foreach column {
Flat-field = median(imaging run) /* normalized */
}
/* sky: mean over data columns */
Sky = mean(flat_field)
}
Foreach frame {
Smooth and interpolate bias drift, flat-field and sky
Write flat-field to file psFF*.fit
Write bias drift and sky to file psCB*.fit
}
}
Initialise Modules, Read Plan/Parameter Files
Foreach filter {
Determine mean (over all frames) gains and dark variances
}
Retrieve information on 200x200 stamps
Foreach filter {
Foreach frame {
Read the flat-field vector (from a psFF*.fit file)
Read 65x65 stamps (from a scFang*.fit file)
Extract profiles for 65x65 stamps
If available { Read 200x200 stamps
Extract profiles for 200x200 stamps
}
}
Reject objects with any of individual flags set on
}
/* these two loops over filters are separated because the above is
coded at Tcl level, and the following one in C */
Foreach filter {
Foreach frame {
Form a subsample of objects
Clip the subsample on the deviation from a median in each cell
Clip the subsample on the extracted profile shape (QU)
Clip the subsample on the extracted profile size (2 Gaussians fit)
Clip the subsample by using a best-fit model PSF
Clip the subsample until it is uniform (measured by chi2)
Determine the mean PSF profile and the best-fit model PSF
Determine composite profile
Add bright objects' profiles to the composite profile
Calculate photometric corrections (aperture correction etc.)
}
}
Read astrometric transformation coefficients for each frame and filter
(from asTrans*.fit file)
Calculate time and airmass for each frame and filter
(by using information from scFang*.fit files headers)
Read zero points, extinction coefficients and color terms for each filter
(from exPhotom*.par file)
Read positions and magnitudes in each filter of objects detected in the
MT pipeline (from kaCalObj*.fit files, aka patch files)
Read color terms for each filter from psParam.par file
(to be used in log(flux20) = a + color_term * color_index)
Read counts in each filter for a 20th mag object
(from an old calibration file, psCT*.fit)
Setup extinction arrays for interpolation
(extinction coefficient for each filter as a function of time)
Foreach frame {
If No MT patches
Use old calibration(time) from psCT*.fit
Else {
Foreach patch file {
Match PS stars with MT stars
Determine time and calibration(filter) for this patch file
}
}
Interpolate calibration(time,filter) to each frame
}
#################################################################
# This is the Plan file for the Postage Stamp Pipeline
#
# Required Generic Plan Fields
action ps # Plan type
version v1.0 # Software version to use
parameters psParam.par # Tunable parameters file
#
# Optional Generic Plan Fields
priority 1 # Execution priority
oid "23.45.34.23" # Database object identifier
targetDate "1999/04/18" # Target date (local time zone)
created "1965/04/18 05:40:00" # Creation time (UT)
authors "User Abuser" # Authors
comment "example" # Comment
diagnostics 1 # if > 0, do create a file with
# diagnostic output from each field
diagFile psDiag # prefix of the file into which
# diagnostic output will be placed.
# Name will include run and camCol.
#
# additional info
nfilters 5 # number of filters with data in scFang files
stampsize 65 # size on a side of postage stamps
refcolor r # to which the chip to chip trans transforms
# filters order
ccdrow_r 1
ccdrow_i 2
ccdrow_u 3
ccdrow_z 4
ccdrow_g 5
ccdrow_o 6
ccdrow_l 7
ccdrow_t 8
ccdrow_s 9
#################################################################
# List of paths to various required files
# the param file given above as 'parameters' may be found here
parametersDir $PHOTODATA_DIR/input
# directory with bias data (idB*.fit files)
biasDir $PHOTODATA_DIR/idB
# directory with the CCD parameters file ...
configDir $PHOTODATA_DIR/input
# and file names
ccdConfig opConfig-51259.par # amplif. config
ccdBC opBC-51259.par # bad column file
ccdECalib opECalib-51259.par # readnoise/gain file
# the fangs, output of SSC pipeline may be found here
fangDir $PHOTODATA_DIR/fangs
# astrometric transformations for this run
transFileDir $PHOTODATA_DIR/input
transFile asTrans-000581.fit
# monitor telescope photoms
photomFileDir $PHOTODATA_DIR/input
photomFile exPhotom-51261.fit
# monitor telescope secondary patches are here
mtPatchFormat mtPatch-%08d.fit
mtPatches {95-25-1 $PHOTODATA_DIR/input}\
{90-25-1 $PHOTODATA_DIR/input}\
{95-25-2 $PHOTODATA_DIR/input}\
{90-25-2 $PHOTODATA_DIR/input}
# default calibration is here (last good output from psp)
ctFileDir $PHOTODATA_DIR/output
ctFile psCT-000581-3.fit
# put the output here:
outputDir $PHOTODATA_DIR/output
#################################################################
# Processing Parameters
# the important stuff
run 581 # Imaging run number.
camCol 1 # Camera column
startField 10 # start pipeline with this field
endField 66 # stop with this (inclusive)
filterlist u g r i z # filters to include when processing
# aux stuff
verbose 1 # verbose > 0 means print a lot
trace_memory 1
crc_nbyte_max 20000
### display requests (0: off) ###
display 0 # display on/off (-1 for w/o prompting)
# the following flags control saoimage display
display_raw 0 # show the raw PSF stamps
display_psf 0 # show the corrected PSF stamps (2 for rejects)
display_PSFstars 0 # show good and clipped PSF stars
display_average_psf 0 # display the average of PS stars
display_wing 0 # display corrected wing stamps
display_symmetry 0 # display residual after subtracting symmetric image
display_flat 0 # display flatfield (>1 for the smoothed flat-field vector)
# the following flags control pgplot display
display_bias 0 # plot bias
display_clipping 0 # plots of (sigma1,sigma2) and (Q,U) for good stars
# for >1 produces hard copies in outdir (psPlotsQU*.ps)
display_rejection 0 # display rejection statistics for all frames
# for >1 produces hard copies in outdir (psPlotsRejecStats*.ps)
display_profiles 0 # display stars' CELL_PROFs
# for >1 produces hard copies in outdir (psPlotsCellProf*.ps)
display_summary 0 # graphs at the end (also written to psPlots*.ps)
#
# Software parameters for postage stamp pipeline
#
ps_wingsize 200 # size of stellar-wing patches
# value at which a pixel is considered saturated
ps_saturation_level 64000
# software bias to prevent negative pix values after bias subtraction
ps_soft_bias 1000
# file of cell-array profiles (precalculated PSF models)
cellprofile_file $PHOTO_DIR/lib/cellprof.dat
# parameters used by flatfields modules
ps_smooth_length 0 # how to smooth the bias vectors
# must set to 0, currently not used
ff_ninterp 5 # controls the number of adjecent frames, nadj,
# to smooth over (nadj = 2 * ff_ninterp + 1)
ff_sigrej 3 # Rejection limit in sigmas
ff_beta 0 # index specifying interpolation (filtering) method
# 0 = median, 1 = mean, > 1 a weighting function
# which -> delta function for ff_beta -> inf.
left_buffer 0 # do not include last left_buffer pixels in left
# overscan region when determining bias drift
right_buffer 0 # do not include first right_buffer pixels in right
# overscan region when determining bias drift
# parameters used by the findCR function
cr_min_sigma 6.0 # CRs must be > this many sky-sig above sky
cr_min_e 150 # CRs must have > this many electrons in peak pixel
cr_cond3 3.0 # used in condition 3 for CR; see CR.c code
cr_cond32 0.6 # used in condition 3 for CR; see CR.c code
# parameters used by makeStarlist
star1_aperature 5 # Index of radial profile aperature
# to use for photometric calibration
psf_def_rad 27 # rad out to which to use data for PS stars
wing_def_rad 60 # rad out to which to use data for wing stars
psf_threshold 5 # % of peak to trigger asymmetry filter
sigma_multi 2.5 # number of sigma sky to trigger multiple flag
sigma_badsky 3.0 # number of sigma for bad sky, 3.0
second_peak 0.0001 # minimum required strength for the 2nd peak to
# set MULTIPLE flag
# parameters used for selecting good PSF stars
min_starnum 2 # min. number of good stars needed for PSF
# determination
psf_nframe 3 # interpolate psfs over nframes frames (odd number)
psf_niter 2 # number of times to iterate sigma clip
soft_err_min 0.01 # minimal error for composite profile used
# for softening before fitting the PSF
soft_err_max 0.02 # maximal error for composite profile used
# for softening before fitting the PSF
nowing_option 2 # what to do w/o wing stars:
# 1 = give up
# 2 = assume some canonical values
psf_critical_amplitude 5000 # if bad width and dgpsf->a > this,
# the stamp's bad in all bands
# cliping parameters
# cell by cell clip
ncellclip 3.0 # how many sigma to clip cells at
nbadcell 6 # how many bad cells to condemn a star
maxdev 5.0 # maximal allowed deviation from the median
# size and shape clip
psf_nsigma1 4.0 # number of sigma at which to clip sigma
psf_nsigma2 2.0 # number of sigma at which to clip QU
# clip on best_fit model PSF
clip_mod 0 # if < 1 do not clip on best-fit model
# chi2 clip
chisq_max 1.0 # max chi2 for all stars
chisq_cutoff 3.0 # max chi2 for a single star
# parameters used by photometricParameters module
fpp_refcolor_u g # Band used with color terms in the u
fpp_refcolor_g r # Band used with color terms in the g
fpp_refcolor_r g # Band used with color terms in the r
fpp_refcolor_i r # Band used with color terms in the i
fpp_refcolor_z i # Band used with color terms in the z
fpp_color_term_u .01 # log(flux20) = a + color_term * color
fpp_color_term_g .01 # log(flux20) = a + color_term * color
fpp_color_term_r .01 # log(flux20) = a + color_term * color
fpp_color_term_i .01 # log(flux20) = a + color_term * color
fpp_color_term_z .01 # log(flux20) = a + color_term * color
fpp_color_term_err_u .001 # Error in color term for u
fpp_color_term_err_g .001 # Error in color term for g
fpp_color_term_err_r .001 # Error in color term for r
fpp_color_term_err_i .001 # Error in color term for i
fpp_color_term_err_z .001 # Error in color term for z
fpp_starsep 8 # Max sep (") for match to mt stars
Please send any comments or suggestions to Zeljko Ivezic