{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Calibrating data\n",
"\n",
"In this tutorial chapter you will process data for the first time. We take the example of reducing raw science frames. This involves trimming, de-biasing, flat-fielding, illumination-correction, as well as the detection of saturated pixels, cosmic-rays and satellite tracks. Together with cold- and hot pixels these affected pixels are assigned a weight of zero in the weight map, which is also created. The results will be stored on the Astro-WISE dataservers and database.\n",
"\n",
"### Database projects and privileges\n",
"\n",
"Data in the database is organized in projects. To see in which project you are currently working:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"print(context.get_current_project())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For example the output could be:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" Project: OMEGACAM@VST (id = 36) \n",
" Description: All public data for OmegaCAM. OmegaCAM is a 1 square \n",
" degree wide field, optical, 16k X 16k pixel camera for the VLT Survey \n",
" Telescope (VST) on Paranal Observatory. \n",
" Instrument: OMEGACAM \n",
" Maximum Privileges: 4 (World) \n",
" Current Privileges: 1 (MyDB) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you are doing this tutorial as part of a course you should produce the data in a certain project: *TUTORIAL*. Select this project by typing:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"context.set_project('TUTORIAL')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you have not been asked to enter a certain project you can use the project you are in now. From now on, everything you create will be stored inside the chosen project. There will be other members of this project. You can set privileges to restrict who can view and access data within a project. A privilege of 1 means only you can view and access the data:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"context.set_privileges(1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"From here on, everything you create will be stored at privilege level 1 in the chosen project: only you can view/access the data. To allow other people in the project to see products, set privileges to 2 before you create them."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Processing science frames\n",
"\n",
"Selecting and exploring RawScienceFrames Enter the following query that selects the raw science data objects, i.e., instantiations of the class RawScienceFrame, from the database."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"query = (RawScienceFrame.instrument.name == 'OMEGACAM') & \\\n",
" (RawScienceFrame.template.start == datetime.datetime(2012, 6, 1, 8, 51, 44)) & \\\n",
" (RawScienceFrame.is_valid > 0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To see how many raw science frames are found type:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"len(query)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These are 5 OmegaCAM exposures, each consisting of 32 CCDs (32 RawScienceFrames are created for a single exposure).\n",
"We select only the data of CCD #65 for now:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"query = query & (RawScienceFrame.chip.name == 'ESO_CCD_#65')\n",
"len(query)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To print the meta-data of the last RawScienceFrame use the ``info()`` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"raw = query.max('DATE_OBS')\n",
"raw.info()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To print only the observation dates, filter names, and maximum pixel value in the frame for all of them:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for r in query: print(r.DATE_OBS, r.filter.name, r.imstat.max)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Download the FITS file of the frame we selected above:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"raw.retrieve()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that you have the file locally you can display it using your favorite image viewer.\n",
"To visually inspect a frame with the help of a few analysis tools type:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"raw.inspect()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"*Note: the following does not work in Jupyter notebook*\n",
"\n",
"This brings up a graphical window. Hover your mouse over an object in the window and hit 'w': this shows a 3D wire-frame plot of the pixel values around the mouse pointer. Hit 'q' when you hover over it to close the graphical window. For a listing of all hot-keys type:"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"help(raw.inspect)
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"Help on method inspect in module astro.main.BaseFrame:\n",
"\n",
"inspect(pixels=None, zone=None, kappa=3.0, iterations=2, cmap=None, vmin=None, vmax=None, interpolation='lanczos', width=6, ratio=None, viewer='skycat', force_figure=False, force_viewer=False, subplot_size=50, contour_levels=20, num_bins=100, extension=None, compare=False, level=0, other=None, clip=False, color=False) method of astro.main.RawFrame.RawScienceFrame instance\n",
" Optional visual inspection for quality control displays image in a\n",
" PyLab (MatPlotLib) window or optionally in an external viewer.\n",
" \n",
" pixels: optional list or array representing the image to be\n",
" inspected (can be MxN for greyscale, or MxNx3 for RGB)\n",
" zone: tuple of (x0, y0, x1, y1) representing the image\n",
" coordinates of the two oposing corners of the sub\n",
" image to consider\n",
" kappa: the factor by which the dynamic range is increased in\n",
" units of sigma (0 gives full range)\n",
" iterations: number of iterations in the kappa-sigma range clipping\n",
" cmap: PyLab color map instance\n",
" vmin: lower display range in native units (e.g. ADU)\n",
" vmax: upper display range in native units (e.g. ADU)\n",
" interpolation: type of interpolation the PyLab viewer uses (nearest,\n",
" bilinear, etc.)\n",
" width: width of the PyLab figure window (in inches)\n",
" ratio: ratio by which to scale the figure height (default:\n",
" x_dim/y_dim)\n",
" viewer: external viewer to use in case the image is too large\n",
" force_figure: always use the PyLab figure window (Be Careful!\n",
" Statistics calculations on large images can be very\n",
" time and memory consuming.)\n",
" force_viewer: always use the viewer\n",
" subplot_size: width and height in pixels of region of interest\n",
" contour_levels: number of contour levels for the contour plot of the\n",
" region of interest\n",
" num_bins: number of bins in the histogram plot\n",
" extension: extension of the filetype to save plot to (png, ps,\n",
" or eps) None disables saving\n",
" compare: compare this frame to its previous version using\n",
" difference imaging (current-previous), pixels is\n",
" ignored\n",
" level: depth of query for previous version (0 goes as deep\n",
" as possible) when compare is True\n",
" other: a second of the same type of Frame object to replace\n",
" previous when compare is True (if color is True,\n",
" other can be a list of two images)\n",
" clip: kappa-sigma clip each image prior to subtraction when\n",
" compare is True\n",
" color: use color combining (RGB) instead of differencing\n",
" when compare is True (kappa, vmin/vmax only honored\n",
" when clip is True), this image is R, other is B if\n",
" single, other is [G, B] if it is a list\n",
" (EXPERIMENTAL)\n",
" \n",
" When force_viewer is False, inspect() displays basic image\n",
" statistics (mean, stddev) and then a representation of the image\n",
" that can be zoomed and panned. Pressing various keys will give\n",
" different results described below in a region of interest described\n",
" by subplot_size:\n",
" \n",
" q - closes the most recent plot window when pressed in the\n",
" main window\n",
" [space] - displays the X and Y coordinate (FITS standard unit\n",
" indexed) and the count level\n",
" a - performs aperture photometry on brightest feature in the\n",
" region of interest (NOT YET IMPLEMENTED)\n",
" c - displays a contour plot of the region of interest (see\n",
" contour_levels)\n",
" h - displays a histogram of the pixel values of the region of\n",
" interest (see num_bins)\n",
" r - displays a radial plot of the brightest feature in the\n",
" region of interest\n",
" w - displays a wireframe plot of the region of interest\n",
" p - displays profile plots in both X and Y dimensions versus\n",
" intensity (count level)\n",
"\n",
" NOTE: None of the commands above work in the subplots.\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"*Proceed here (both awe and jupyter)*\n",
"\n",
"\n",
"Reduce the RawScienceFrames that you found above.\n",
"See [this HOW-TO](http://doc.astro-wise.org/man_howto_calibrationpipelines.html) for an overview of how to process data in Astro-WISE. You will use standard recipes to reduce the RawScienceFrames. It is straightforward to adapt these or to write your own recipes, but this is beyond the scope of this tutorial.\n",
"\n",
"In the Astro-WISE environment you may choose the compute cluster (the ``Distributed Processing Unit`` or ``DPU``) where the data reduction processes run. In that case no results are stored on your local machine. Created data (FITS files) are stored on the dataserver while meta-data is committed to the database.\n",
"\n",
"To de-bias and flatfield the above raw science data using the DPU you can use the``Reduce`` recipe:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"filenames = [raw.filename for raw in query]\n",
"dpu.run('Reduce', instrument='OMEGACAM', raw_filenames=filenames,\n",
" commit=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The ``commit=True`` switch ensures that your data is committed. Choosing ``commit=False`` (or omitting the option, as below) will perform a \"dry run\": everything will be done except committing your results at the end."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By specifying the template start date, you can do the query and reduction at once:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"dpu.run('Reduce', instrument='OMEGACAM', template='2012-06-01T08:51:44',\n",
" chip='ESO_CCD_#65')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can view the status of your DPU job via"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"dpu.get_status()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"or by browsing the DPU server's webpage. See the links found on the [Processing Grid page page](http://www.astro-wise.org/portal/aw_prompt.shtml) (in particular the [DPU Groningen](https://dpu.hpc.rug.astro-wise.org/)).\n",
"You can see how the processing went by retrieving the processing logs from the DPU."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"dpu.get_logs()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This command will not return anything until the process has been completed. The returned lines are also written to a single (\"date+time\".log) file in your local directory per awe-prompt session.\n",
"To cancel DPU processing jobs use the ``cancel_job()`` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for jobid in dpu.get_jobids(): dpu.cancel_job(jobid)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"While it is faster to calibrate RawScienceFrames in parallel on a DPU it can sometimes be convenient to use your local machine. In this case the results are left in your current working directory. To de-bias and flatfield your raw science data found above using using local CPU you can use the same Reduce recipe as follows:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"task = ReduceTask(instrument='OMEGACAM', raw_filenames=filenames, commit=True)\n",
"task.execute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"or..."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"task = ReduceTask(instrument='OMEGACAM', template='2012-06-01T08:51:44', chip='ESO_CCD_#65', commit=True)\n",
"task.execute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Inspect the results: ReducedScienceFrame\n",
"\n",
"The results of the above process are de-biased, flatfielded science frames, called ``ReducedScienceFrames``. To select one of the ones you just created type:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"qred = (ReducedScienceFrame.raw == raw)\n",
"red = qred.project_only().user_only().max('creation_date')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The first command queries for all ReducedScienceFrames which were created from the RawScienceFrame object query[0] in all projects you have access to. The second command narrows the selected ReducedScienceFrames to only those created within the project you are currently in. The third command zooms in on the subset of those that were created by you. The last command selects the ReducedScienceFrame that you created most recently. To do all this in a single command for all RawScienceFrames in query:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"qred = [(ReducedScienceFrame.raw==raw).project_only().user_only().max('creation_date') for raw in query]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Inspect the ReducedScienceFrame like you did earlier:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"red.inspect()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Determine which calibration frames were used. As you did not create the bias frames and flatfields, the Astro-WISE environment selected those for you. To get general information on which ones were used:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"red.bias.info()\n",
"red.flat.info()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and to inspect one of the RawBiasFrames that were used to create the bias frame:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"red.bias.raw_bias_frames[0].inspect()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have not configured anything beyond specifying the input frames in the previous examples. Find out which other parameters can be configured. \n",
"\n",
"See the [configuration HOW-TO](http://doc.astro-wise.org/man_howto_configure.html) for more details. Calling the ``help()`` function on the Task gives a description of its possible arguments. These are mostly query parameters which determine which input frames (RawScienceFrames in this case) will be reduced. In addition an argument ``pars`` can be specified which contains the processing parameters."
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"help(ReduceTask)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"Help on class ReduceTask in module astro.recipes.Reduce:\n",
"\n",
"class ReduceTask(astro.recipes.mods.Task.Task, astro.recipes.mods.Task.ImagePipe\n",
"lineMixin)\n",
" | --------------------------------------------------------------------------\n",
" | \n",
" | CATEGORY : data processing, image pipeline.\n",
" | \n",
" | PURPOSE : making reduced science frames (flat-fielded, de-biased)\n",
" | \n",
" | --------------------------------------------------------------------------\n",
" | \n",
" | This task is used to create ReducedScienceFrame's.\n",
" | \n",
" | The task must be instantiated with the appropriate keyword arguments\n",
" | described below, and is executed using its 'execute' method.\n",
" | \n",
" | --------------------------------------------------------------------------\n",
" | \n",
" | Example 1 of use from the awe prompt:\n",
" | \n",
" | awe> task = ReduceTask(instrument='OMEGACAM', chip='ESO_CCD_#65',\n",
" | filter='OCAM_i_SDSS', date='2015-03-01',\n",
" | object='KIDS_186.5_-2.5')\n",
" | awe> task.execute()\n",
" | \n",
" | Example 2 of use from the awe prompt:\n",
" | \n",
" | awe> task = ReduceTask(instrument='WFI',\n",
" | raw_filenames=['wfi60757_1.fits'])\n",
" | awe> task.execute()\n",
" | \n",
" | \n",
" | See the section below for an overview of the possible constructor\n",
" | keywords.\n",
" | \n",
" | --------------------------------------------------------------------------\n",
" | \n",
" | Two sets of query keywords can be provided in the constructor of the\n",
" | task:\n",
" | \n",
" | 1) one list of filenames.\n",
" | 2) a set of global query parameters (like date, filter and chip).\n",
" | \n",
" | The first of these sets has precedence over the second.\n",
" | \n",
" | The first set of possible keywords is one list of filenames:\n",
" | \n",
" | raw_filenames (filenames of the raw science frames, list of strings)\n",
" | \n",
" | The second set of possible keywords is:\n",
" | \n",
" | date (string)\n",
" | instrument (string)\n",
" | chip (string)\n",
" | filter (string)\n",
" | template (string)\n",
" | object (string or using wildcards like M33_?_?, M33_V*)\n",
" | \n",
" | Finally, the following parameters can also be set in the constructor:\n",
" | \n",
" | force_defringe (integer 0 or 1. Default 0. Use this only for example\n",
" | for WFI R filter, which is not automatically\n",
" | defringed)\n",
" | ofieldedverscan (integer 0-13. Default 6)\n",
" | pars (dictionary of configuration parameters, example:\n",
" | {'ReducedScienceFrame.process_params.FRINGE_THRESHOLD_HIGH': 6.0})\n",
" | commit (boolean: False or True. Default False)\n",
" | \n",
" | For all the keywords documented here, the type of the values assigned\n",
" | to it is given between parentheses.\n",
" | \n",
" | --------------------------------------------------------------------------\n",
" | \n",
" | Method resolution order:\n",
" | ReduceTask\n",
" | astro.recipes.mods.Task.Task\n",
" | common.util.Pipeline.Task\n",
" | astro.recipes.mods.Task.TaskMixin\n",
" | astro.recipes.mods.Task.ImagePipelineMixin\n",
" | builtins.object\n",
" | \n",
" | Methods defined here:\n",
" | \n",
" | __init__(self, *args, **kwargs)\n",
" | Recognized options for the astro.recipes.Reduce OptionsParser are:\n",
" | \"i\" or \"instrument\" (default: )\n",
" | \"f\" or \"filter\" (default: )\n",
" | \"c\" or \"chip\" (default: )\n",
" | \"d\" or \"date\" (default: )\n",
" | \"o\" or \"object\" (default: )\n",
" | \"tpl\" or \"template\" (default: None)\n",
" | \"oc\" or \"overscan\" (default: 6)\n",
" | \"rfd\" or \"rfd_filenames\" (default: [])\n",
" | \"raw\" or \"raw_filenames\" (default: [])\n",
" | \"e\" or \"exptime\" (default: -1)\n",
" | \"fd\" or \"force_defringe\" (default: 0)\n",
" | \"p\" or \"pars\" (default: {})\n",
" | \"C\" or \"commit\" (default: False)\n",
"etc.\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Instantiate the Pars class with as argument the task, the pipeline identifier used in the dpu call or the class:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"p = Pars(ReduceTask, instrument='OMEGACAM')\n",
"p = Pars('Reduce', instrument='OMEGACAM')\n",
"p = Pars(ReducedScienceFrame, instrument='OMEGACAM')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By setting the instrument argument, instrument specific default processing parameters are used. Now call the ``show()`` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"p.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A parameter can be set as follows (use Tab completion):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"p.ReducedScienceFrame.process_params.FRINGE_THRESHOLD_HIGH = 6.0"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Call the get() method to obtain the dictionary that you can supply to the task:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"p.get()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.1"
}
},
"nbformat": 4,
"nbformat_minor": 1
}