{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Mining Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Investigating Twilight Behavior from RawTwilightFlatFrames\n",
    "\n",
    "Find all OmegaCAM raw twilight flat frames for the chip with name ESO\\_CCD\\_\\#92 and the filter with name OCAM\\_g\\_SDSS, which have image statistics where the median is greater than 1000 ADU. Only select those that have their quality_flags set to zero. Assign the query to a Python variable.\n",
    "\n",
    "How many raw twilight flat frames satisfy the above criteria?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context.set_project('TUTORIAL')\n",
    "context.set_privileges(1)\n",
    "print(context.get_current_project())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qinstr  = RawTwilightFlatFrame.instrument.name == 'OMEGACAM'\n",
    "qfilter = RawTwilightFlatFrame.filter.name   == 'OCAM_g_SDSS'\n",
    "qchip   = RawTwilightFlatFrame.chip.name     == 'ESO_CCD_#92'\n",
    "qim     = RawTwilightFlatFrame.imstat.median >  1000\n",
    "qqf     = RawTwilightFlatFrame.quality_flags == 0\n",
    "query  = qinstr & qfilter & qchip & qim & qqf\n",
    "print(len(query))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the result from the previous question, assign the MJD\\_OBS (the modified Julian date) of each RawTwilightFlatFrame to variable t and the median of the image statistics divided by the EXPTIME of each RawTwilightFlatFrame to variable y. Make a scatter plot of t versus y."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "t = [rtf.MJD_OBS for rtf in query]\n",
    "y = [rtf.imstat.median/rtf.EXPTIME for rtf in query]\n",
    "pylab.scatter(t, y)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or, quicker:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = [(rtf.MJD_OBS, rtf.imstat.median/rtf.EXPTIME) for rtf in query]\n",
    "t, y = zip(*result)\n",
    "pylab.scatter(t, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since there seems to be a recurring pattern in the previous plot, we'll have a closer look. Take the fractional part of the t variable, which contains the modified Julian date for each RawTwilightFlatFrame and assign it to the variable tfrac. Clear the figure and make a scatter plot of tfrac versus y."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfrac = [k - math.floor(k) for k in t]\n",
    "pylab.clf()\n",
    "pylab.scatter(tfrac, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does the fractional time difference agree with the rough times in UTC when the twilight flats were observed?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bias level for OmegaCAM\n",
    "\n",
    "Display the bias level as a function of time in the last year for chip ESO\\_CCD\\_\\#96 of the OmegaCAM camera (raw biases are represented in Astro-WISE by the class RawBiasFrame)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = (RawBiasFrame.instrument.name == 'OMEGACAM') &\\\n",
    "        (RawBiasFrame.DATE_OBS > datetime.datetime.utcnow() - datetime.timedelta(365)) &\\\n",
    "        (RawBiasFrame.chip.name == 'ESO_CCD_#96') &\\\n",
    "        (RawBiasFrame.quality_flags == 0) &\\\n",
    "        (RawBiasFrame.is_valid > 0)\n",
    "res = [(b.DATE_OBS, b.imstat.median) for b in query]\n",
    "dat, val = zip(*res)\n",
    "pylab.clf()\n",
    "pylab.scatter(dat, val)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}