HOW-TO SourceLists in the Astro-WISE System¶

After having reduced the science data, source lists can be derived. This basically boils down to running SExtractor on this data, but to make sure that the extracted sources are also stored in the database for later scrutiny, some extra steps are performed. The way this is handled in the Astro-WISE system is described in the following subsections.

HOW-TO: Create Simple SourceLists From Science Frames¶

In the Astro-WISE system source lists are represented by SourceList objects. The SourceList class is imported by default when starting AWE, but if you need to import it in a script, this is done as follows:

awe> from astro.main.SourceList import SourceList

In order to derive a source list from a reduced science frame, an instance of the SourceList class must be created and the reduced science frame must be assigned to this object as a dependency. The three types of science frames for which a source list are commonlt made are:

ReducedScienceFrame: requires separate astrometric solution (AstrometricParameters)
RegriddedFrame: a resampled ReducedScienceFrame with incorporated astrometric solution
CoaddedRegriddedFrame: a mosaic of RegriddedFrames

Invoking the ‘make’ method of the SourceList object creates the source list which is then automatically stored into the database. From the awe-promptthis reads for a RegriddedFrame:

awe> query = RegriddedFrame.filename == 'frame.fits'
awe> frame = query[0]
awe> frame.retrieve()
awe> frame.weight.retrieve()
awe> sourcelist = SourceList()
awe> sourcelist.frame = frame
awe> sourcelist.make()
awe> sourcelist.commit()

which will create the source list in the database. The ‘make’ method of the SourceList object first calls the ‘make_sextractor_catalog’ method (which obviously runs SExtractor), followed by a call to the ‘make_sourcelist_from_catalog’ method that ingests the sources into the database. In a similar fashion one can make a SourceList for a CoaddedRegriddedFrame. However, for a ReducedScienceFrame a separate astrometric solution is required. In this case the awe-prompt reads:

awe> query = ReducedScienceFrame.filename == 'frame.fits'
awe> frame = query[0]
awe> astrom = (AstrometricParameters.reduced == frame).max('creation_date')
awe> frame.retrieve()
awe> frame.weight.retrieve()
awe> sourcelist = SourceList()
awe> sourcelist.frame = frame
awe> sourcelist.astrom_params = astrom
awe> sourcelist.make()
awe> sourcelist.commit()

which will create the source list in the database. The ‘make’ method of the SourceList object first calls the ‘make_sextractor_catalog’ method (which obviously runs SExtractor), followed by a call to the ‘make_sourcelist_from_catalog’ method that ingests the sources into the database.

The configuration of SExtractor and its output can be manipulated through the SourceList interface as well. This is done through the sexconf and sexparam dependencies, respectively. For example, if one wants to run SExtractor with a detection threshold of 12, and to have MAG_APER and MAGERR_APER as additional output, the AWE-session above changes into (skipping the database query and retrieve operations):

awe> sourcelist = SourceList()
awe> sourcelist.frame = frame
awe> sourcelist.astrom_params = astrom
awe> sourcelist.sexconf.DETECT_THRESH = 12
awe> sourcelist.sexparam = ['MAG_APER', 'MAGERR_APER']
awe> sourcelist.make()
awe> sourcelist.commit()

with the extra output parameters added to the definition of the sources contained in the list.

A special feature has been implemented to allow for skipping some specified records in the SExtractor catalog. This works as follows:

awe> sourcelist = SourceList()
awe> sourcelist.record_skiplist = [13, 169]

The last statement will make sure that the sources on record 13 and 169 of the SExtractor catalog are NOT ingested into the list.

SegmentationImage¶

When a SourceList is created, by default a SegmentationImage is also made. A segmentation image is a file that SExtractor can generate, that defines which pixels are part of what object as detected by SExtractor.

A SegmentationImage is derived from the class CheckImage. If you have made a SourceList, its associated SegmentationImage can be found with a query such as this one:

awe> sl = (SourceList.SLID == 423431)[0]
awe> segm = (SegmentationImage.sourcelist == sl)[0]

# and display it ...
awe> segm.display()

Using SourceList with SExtractor double-image mode¶

When a SourceList is made for an image one has the option to specify a secondary image. This makes the SExtractor software which SourceList uses, run in the so-called “double-image mode”.

To explain the process, this is how you would normally run SExtractor in double-image mode:

sex -c sex.conf image1.fits image2.fits

Here “image1.fits” is called the detection image and “image2.fits” is the measurement image. The detection image and measurement image must have identical dimensions. The detection image is the image used to determine which pixels are part of sources. Physical properties are derived from the measurement image, using the pixels attributed to sources in the detection image. Changing the measurement image for another image will not modify the number of detected sources, nor will it affect their pixel positions or basic shape parameters. All other parameters (photometric parameters in particular) will use measurement image pixel values, which allows one to easily measure pixel-to-pixel colours.

The images must have identical dimensions because in double-image mode all sources detected in the detection image have their photometry measured at exactly the same pixel locations in the measurement image.

Using SExtractor image mode is only accurately possible when using RegriddedFrames or CoaddedRegriddedFrames that have been regridded to the same grid. A cutout of the overlapping region is automatically made during the creation of the SourceList. For other frames (i.e. ReducedScienceFrames) no attempt it made to determine an overlap; the images are assumed to cover exactly the same area on the sky.

When using SourceListTask the measurement images are specified via the “filenames” variable and all corresponding detection images are specified via the “detection_filenames” variable. Example:

awe> task = SourceListTask(filenames=['image2.fits'],
                           detection_filenames=['image1.fits'])
awe> task.execute()

or with short options:

awe> task = SourceListTask(f=['image2.fits'], df=['image1.fits'])
awe> task.execute()

Note that the primary image used in the SourceList is the measurement image; while this is the second filename that is given to SExtractor.

For your information, here are most (if not all) parameters that are derived from the detection image in double image mode:

X_IMAGE          X2_IMAGE       A_IMAGE
Y_IMAGE          Y2_IMAGE       B_IMAGE
XMIN_IMAGE       XY_IMAGE       ELONGATION
XMAX_IMAGE       CXX_IMAGE      ELLIPTICITY
YMIN_IMAGE       CYY_IMAGE      KRON_RADIUS
YMAX_IMAGE       CXY_IMAGE

THRESHOLD      (same for all sources)
MU_THRESHOLD   (same for all sources)

HOW-TO: Use External SourceLists¶

The description of the use of SourceList objects given in §[simpleframes] deals with deriving source lists directly from a science frame. However, already existing source lists can also be stored into the database through the use of SourceList objects. This is useful if one wants to store a catalog of standard stars into the database. This is done by instantiating a SourceList object with assigning the external source list name to catalog, and calling the ‘make_sourcelist_from_catalog’ method. As seen from the awe-prompt:

awe> sourcelist = SourceList()
awe> sourcelist.catalog = 'external.fits'
awe> sourcelist.make_sourcelist_from_catalog()
awe> sourcelist.commit()

and the ingestion of the external source list is complete.

For this mechanism to work, the external catalog has to meet some conditions. First and foremost, the external catalog has to be of the same type and layout of a Sextrator catalog like the ones produced in the Astro-WISE system. This means that the external catalog should be in LDAC fits format, and must have an OBJECTS and FIELDS table. Secondly, the OBJECTS table should have the following columns : RA or ALPHA_SKY, DEC or DELTA_SKY, A, B, Theta or POSANG, and FLAG. Besides these mandatory columns, any other column may be present in the catalog with no restrictions on the name.

HOW-TO: Use SourceLists¶

Once a SourceList is ingested into the database, the usual method of retrieving database objects should be used to access the stored information, i.e.:

awe> sourcelist = (SourceList.SLID == 0)[0]

Each SourceList has an unique SoureList Identifier (SLID) and/or a name. A SourceList name does not have to be unique, so the following statement might result in many sourcelists:

awe> sourcelists = (SourceList.name == 'MySourcelists')

Each source in a SourceList has a unique Source Identifier (SID), which is assigned during ingestion into the data base and which starts at zero. The number of sources in the sourcelist is obtained with the len function:

awe> number_of_sources = len(sourcelist.sources)

or with the special attribute number_of_sources:

awe> number_of_sources = sourcelist.number_of_sources

The column information is obtained as follows:

awe> column_info = sourcelist.sources.get_attributes()

column_info is a dictionary with column names as keys and column types as contents. Column data is for example retrieved with the following statement:

awe> RA = sourcelist.sources.RA

where RA is a list containing the values of column ‘RA’. Row data is retrieved in a similar way, for example:

awe> first_source = sourcelist.sources[0]

where first_source is a dictionary with column names as keys and column values as contents.

There are some methods defined for sourcelists. At the moment these are:

sourcelist.sources.make_skycat(sid_list=None, filename=None)

make_skycat creates a “dump” of the SourceList object that can be used to overplot a frame in ESO skycat.

If sid_list is given it should be a list of SID’s (Source IDentifiers) which will be output to the skycat output file.

The default filename is the name of the SourceList with extension .scat.
sourcelist.sources.area_search(self_search=True, htm_depth=20, Area=None)

area_search searches the specified area for sources within a given distance from a position {i.e. Area=(RA,DEC,Distance)} or within an area delimited by three or four positions {i.e. Area=[(RA0,DEC0),(RA1,DEC1),(RA2,DEC2)]}.

All positions and distances are in degrees.

For the search, htm trixel ranges are searched at the specified depth (default is 20). If self_search==True, only the current SourceList is examined, if self_search==False, all SourceLists will be examined.

Return value is a dictionary with the SLID’s (SourceList IDentifiers) for keys and lists of SID’s (Source IDentifiers) for values. Example:
```
awe> r = sourcelist.sources.area_search(Area=[(1.0,0.0),(0.0,0.0),(0.0,1.0),(1.0,1.0)])
awe> print('Sources found: ', len(r[sourcelist.SLID]))
```
sourcelist.sources.get_data(dict)

get_data returns the data associated with given attributes. dict is a dictionary where the keys represent the attributes returned and their subsequent values should be on input None or an empty list ([]).

On output the actual values of the attributes are stored as key values.

The function returns a list of Source IDentifiers (SID’s). Example:
```
awe> dict = {'RA':[], 'DEC':[], 'MAG_ISO':[]}
awe> r = sourcelist.sources.get_data(dict)
```
sourcelist.sources.sql_query(dict, query_string)}

sql_query performs an SQL query on source attributes. query_string may contain only valid SQL query syntax and the names of the attributes between double quotes (“).

dict is a dictionary where the keys represent the attributes returned and their subsequent values should be on input None or an empty list ([]).

On output the actual values of the attributes are stored as key values.

The function returns a list of Source IDentifiers (SID’s). Example:
```
awe> dict = {'RA':[], 'DEC':[], 'MAG_ISO':[]}
awe> r = sourcelist.sources.sql_query(dict, '"MAG_ISO"<18.5 AND "A">0.2')
```
sourcelist.sources.make_image_dict(sids, mode='sky')

make_image_dict returns a dictionary which can be used as input for the interface to the image server.

sids may contain one SID or a list of SID’s.

mode is used to specify whether sky (default) or grid coordinates are wanted. Example:
```
awe> imgdict = sourcelist.sources.make_image_dict(0)
awe> from astro.services.imageview.imgclient import imgclient
awe> ic = imgclient(imgdict, wide_high=[150, 150])
awe> ic.getimg()
```

HOW-TO: Associate SourceLists¶

To spatially associate two different sourcelists the make method of the AssociateList class can be used. The association is done in the following way:

First the area of overlap of the two sourcelists is calculated. If there is no overlap no associating will be done.
Second the sources in one sourcelist are paired with sources in the other sourcelist if they are within a certain distance from eachother. Default distance is 5”. The pairs get an unique associate ID (AID) and are stored in the associatelist. A filter is used to select only the closest pairs.
Finally the sources which are not paired with sources in the other list and are inside the overlapping area of the two sourcelists are stored in the associatelist as singles. They too get an unique AID.

NOTE: in default mode all objects in the to-be associated sourcelists, which have SExtractor Flag \(>\) 0, are filtered out. To keep objects which have Flag \(>\) 0, one must type ‘AL.process_params.SEXTRACTOR_FLAG_MASK=255’ in the example below.

In Python this is done as follows:

awe> AL = AssociateList()
awe> sl0 = (SourceList.SLID == 0)[0]
awe> sl1 = (SourceList.SLID == 1)[0]
awe> AL.input_lists.append(sl0)
awe> AL.input_lists.append(sl1)
awe> AL.set_search_distance(5.0)
awe> AL.associatelisttype = 1
awe> AL.make()
awe> AL.commit()

If one does not want to filter for closest pairs, the following statement should be executed before the make:

awe> AL.single_out_closest_pairs(False)

Optionally one can also specify the search area. Only sources from both sourcelists which lie inside this area are matched. This can be done as follows, before you do AL.make():

awe> AL.set_search_area(llra,lldec,lrra,lrdec,urra,urdec,ulra,uldec)

where:

llra  = lower left R.A. of search area
lldec = lower left Dec. of search area
lrra  = lower right R.A. of search area
lrdec = lower right Dec. of search area
urra  = upper right R.A. of search area
urdec = upper right Dec. of search area
ulra  = upper left R.A. of search area
uldec = upper left Dec. of search area

A previously created AssociateList can also be input to a new AssociateList. Simply put the existing AssociateList in the input_lists as shown in the following example:

awe> AL = AssociateList()
awe> ALs = (AssociateList.ALID == 0)
awe> SLs = (SourceList.SLID == 2)
awe> AL.input_lists.append(ALs[0])
awe> AL.input_lists.append(SLs[0])
awe> AL.make()
awe> AL.commit()

The member sourcelists can be derived from the AssociateList attribute sourcelists. The above example creates a so-called ‘chain’ association: The new SourceList is paired with the last SourceList in the sourcelists of the existing AssociateList. Figure 1 gives an example of a chain association.

Figure 1: An example of a chain association with 4 sourcelists. Each sourcelist is always paired with the previous sourcelist.

The pairing can also be done in a different way in that the pairing is always done between the new SourceList and the first one in the sourcelists list of the input AssociateList. This is a so-called ‘master’ association. Figure [master association] gives an example of a master association.

Figure 2: An example of a master association with 4 sourcelists. All sourcelists are paired with the first sourcelist.

Figure 3: An example of a matched association with 4 sourcelists. All sourcelists are paired with the other sourcelists simultaneously. All interconnected pairs make one association.

Another method of pairing sourcelists is by matching sources of a number of sourcelists simultaneously. Here only sources which have at least one companion end up in an association, so singleton sources are out. This is a so called ‘matched’ association. Figure [matched association] gives an example of a matched association. Once a matched associatelist has been created, it can not be used as input to another association.

The AssociateList attribute associatelisttype sets the type of association: 1 for chain, 2 for master and 3 for matched type, chain is the default.

Once an AssociateList is created the number of associates can be obtained as follows:

awe> print(len(AL))

There are two functions which count the associations:

count_from_members(members=0, mode=’EQ’) counts the number of associations which have a specified number of members. With mode you can specify whether you want associations counted with ‘LT’, ‘LE’, ‘EQ’, ‘GE’ or ‘GT’ the specified number of members.

For example, to count the number of pairs in an associatelist derived from only 2 sourcelists one could do the following:
```
awe> print(AL.associates.count_from_members(members=2,mode='EQ'))
```
To count the number of singles one could do:
```
awe> print(AL.associates.count_from_members(members=1,mode='EQ'))
```
count_from_flag(mask=None, mode=’ANY’, count=None, countmode=’EQ’) returns the number of associations which contain sources from specified sourcelists. mask works as a bitmask, each bit set representing a sourcelist which was input to the associatelist. For example bit 2 set means that only the wanted associations which contain a source from sourcelist number 2 will be counted (depends on mode). mode determines how the masking works. mode can be: ‘ALL’ (count only the associations which contain sources from exact the specified sourcelists), ‘INTERSECT’ (count only the associations which contain sources from at least the specified sourcelists) and ‘ANY’ (count only the associations which contain sources from at least one of the specified sourcelists).

It is also possible to specify a bitcount operation with parameter count. count determines the number of different sourcelists which participate in an association, i.e. if count = 3 then only associations are counted which have sources from 3 different sourcelists (depends on countmode). countmode determines whether the number of participation sourcelists should be less than (‘LT’), less than or equal (‘LE’), equal (‘EQ’), greater than or equal (‘GE’) or greater than (‘GT’) count.

For example, to count the number of pairs in an associatelist derived from only 2 sourcelists (i.e. the first and the second sourcelist so mask = 1 + 2) one could do the following:
```
awe> print(AL.associates.count_from_flag(mask=3,mode='ALL'))
```
To count the number of singles one could do:
```
awe> print(AL.associates.count_from_flag(mask=1,mode='ALL'))
awe> print(AL.associates.count_from_flag(mask=2,mode='ALL'))
```
The first command counts the number of singles from the first Sourcelist, the second command those from the second sourcelist.

With the function get_data one can get the SLID, SID or in fact any
attribute of member sources. The function is described as follows:
get_data(attrlist=[], mask=None, mode=’ANY’, count=None,
countmode=’EQ’) returns requested attributes of the associated
sources. The function returns a dictionary with as key the AID
(Association IDentifier) and as value the attribute values according
to the modified list of specified attributes. For each requested
sourcelist a separate list of attributes is returned.
attrlist contains the wanted attributes. Note that SLID and SID will
always be returned and that SLID is the first and SID the second item.
On return attrlist will be modified accordingly.
mask, mode, count and countmode work as in count_from_flag.
The following example shows how to obtain the RA, DEC and MAG_ISO
attributes from an associatelist:

awe> attrlist = ['RA', 'DEC', 'MAG_ISO']
awe> r = AL.associates.get_data(attrlist,mask=3,mode='ALL')
awe> aids = [k for k in r.keys()]
awe> aids.sort()
awe> print('AID', attrlist)
awe> for aid in aids[:10]:
awe>     for row in r[aid]:
awe>         print(aid, row)

The output looks like:

AID ['SLID', 'SID', 'RA', 'DEC', 'MAG_ISO']
[44, 0, 280.86871938032101, 0.36483871219556502, -4.8930621147155797]
[45, 12, 280.868644520494, 0.36475058531692101, -6.9002099037170401]
[44, 1, 280.86753832027802, 0.28356582313986001, -4.3539190292358398]
[45, 24, 280.86745181025799, 0.28365591902279302, -6.2037878036498997]
[44, 2, 280.86763239079198, 0.27250082665266701, -5.1681485176086399]
[45, 11, 280.86766082401601, 0.27245588811475802, -6.6134591102600098]
[44, 3, 280.867557240373, 0.26285186993244197, -5.4812445640564]
[45, 19, 280.867494620603, 0.262849873260255, -7.01251125335693]
[44, 4, 280.86742472466602, 0.25502430125932601, -5.4748520851135298]
[45, 16, 280.86738586918, 0.25500542211914801, -6.6722841262817401]
[44, 7, 280.86738062915998, 0.275450798599644, -4.89949655532837]
[45, 20, 280.86741490360203, 0.27540606762365499, -6.2980375289917001]
[44, 8, 280.86823775792999, 0.32890289165032699, -6.9213681221008301]
[45, 40, 280.86818782868102, 0.328900495041689, -8.8428869247436506]
[44, 9, 280.86780428279701, 0.26950356072341503, -7.1055769920349103]
[45, 32, 280.86776951099699, 0.26946985609771201, -8.1041078567504901]
[44, 13, 280.86790089506002, 0.345712819247041, -6.1025667190551802]
[45, 38, 280.86788627250701, 0.345742041485333, -7.4083743095397896]
[44, 14, 280.86739941046397, 0.333885857133919, -5.7742152214050302]
[45, 41, 280.86730665910602, 0.33388354451305802, -6.7259593009948704]

As for sourcelists, a function called make_image_dict has been
implemented:
al.associates.make_image_dict(aids, mode=’sky’)
make_image_dict returns a dictionary which can be used as input for
the interface to the image server.
aids may contain one AID or a list of AID’s.
mode is used to specify whether sky (default) or grid coordinates are
wanted. Example:

awe> imgdict = al.associates.make_image_dict(0)
awe> from astro.services.imageview.imgclient import imgclient
awe> ic = imgclient(imgdict, wide_high=[150, 150])
awe> ic.getimg()

For inspecting the distances between all associated source pairs one can use the method get_distances:

get_distances calculates the distances between all possible pairs in an association. The output is a dictionary with as key the associate ID (AID) and as value a list containing for each pair a tuple which consists of three items: ((SLID1, SID1), (SLID2, SID2), DISTANCE). SLID1 is always 1.5 one could do the following:

awe> attrlist = ['RA', 'DEC', 'MAG_ISO']
awe> r = al.associates.get_data_on_associates(attrlist, mask=3, mode='ALL')
awe> print('Found %d pairs' % len(r))
awe> i = attrlist.index('MAG_ISO')
awe> newr = {}
awe> for aid in r.keys():
awe>     if (r[aid][0][i] - r[aid][1][i]) > 1.5:
awe>         newr[aid] = r[aid]
awe> print('Found %d pairs with MAG_ISO diff > 1.5' % len(newr))

Note that we use mask = 3 to indicate that we want only those associations which have a source from the first (bit 1 = 1) and second (bit 2 = 2) sourcelist (mask = 1 + 2).

The second example shows how to find ‘drop-outs’ in an AssociateList which has been created associating three sourcelists, each having been obtained for different filters. We are looking for sources which appear only in the second and third sourcelist (i.e. they were not detected with the filter used for obtaining the first sourcelist). This could be done as follows:

awe> attrlist = ['RA', 'DEC', 'MAG_ISO']
awe> r = al.associates.get_data_on_associates(attrlist, mask=6, mode='ALL')
awe> print('Found %d pairs' % len(r))

Note that we use mask = 6 to indicate that we want only those associations which have a source from the second (bit 2 = 2) and third (bit 3 = 4) sourcelist (mask = 2 + 4).

The third example shows how to obtain the data to make a B - V vs. U - V plot. Suppose we have associated three sourcelists, each obtained for different filters U, B, V. Then we could do this as follows:

awe> attrlist = ['RA', 'DEC', 'MAG_ISO']
awe> r = al.associates.get_data_on_associates(attrlist, mask=7, mode='ALL')
awe> print('Found %d triples' % len(r))
awe> i = attrlist.index('MAG_ISO')
awe> x = []
awe> y = []
awe> for aid in r.keys():
awe>     x.append(r[aid][1][i] - r[aid][2][i])
awe>     y.append(r[aid][0][i] - r[aid][2][i])

Note that we use mask = 7 to indicate that we want only those associations which have a source from the first (bit 1 = 1), the second (bit 2 = 2) and third (bit 3 = 4) sourcelist (mask = 1 + 2 + 4).

Visualizing associated sources: creating a skycat catalog¶

It is possible to visualize which sources are associated in an AssociateList by creating a skycat catalog. This catalog can be used in the skycat FITS viewer. Use the following method:

make_skycat_on_associates(slid, mask=None, mode=’ALL’) takes as input a SourceList identifier; the SourceList in the AssociateList for which to produce the skycat catalog. The arguments mask and mode are the same as described in the count_from_flag method. If the optional arguments are not specified the resulting catalog will contain the associates that have members in each input SourceList.

Example:

awe> al.make_skycat_on_associates(slid=123456)

HOW-TO: CombinedList¶

CombinedList is a library which was written to create a SourceList from AssociateList. At the moment the size of the newly created SourceList is limited to 200,000 sources, the bigger SourceLists are possible with the special function provided by the user request.

Main principles¶

The Astro-Wise user has an ability to associate SourceLists cross-identifying sources. The functionality of AssociateList is described in the corresponding how-to.

AssociateList contains links to sources in SourceList only and does not provide any ability to combine attribute values of cross-identified SourceLists. CombinedList fills this gap and allow user to create a permanent SourceList with combined attributes for all sources. This is especially useful for creating of multiband catalogs. At the same moment CombinedList is not a persistent class but a library of methods. The input for the CombinedList methods is an AssociateList, the output is a SourceList.

The main task - creating a stable multiband catalog - is realized by adding a new attributes. The user can set an attribute which he would like to treat as a magnitude (normally this is MAG_ISO or other magnitude attributes from sextractor) and create an average magnitude for associated sources (MAG_1) accompanied by the rms of the magnitude (MAGERR_1), corresponding flags (if user provided information about attribute with flags, MAGFLAG_1), and number of sources which were joint (MAGN_1).

In the case of the same filters for SourceLists (Figure 4) the newly created SourceList will have the only attribute set for magnitude, in the case of different filters (Figure 5) two sets of magnitudes will be created.

Figure 4: CombinedList on the same filter

Figure 5: CombinedList on different filters

The information about filters is saved in the string filters in the following format 'MAG_1:<filter_name>,MAG_2:<filter_name>'. There are up to 20 filters possible.

The user has an ability to set a type of operations which will be provided with input AssociateList. AssociateList is created over SourceLists, sources in SourceLists which corresponds to AID (ID of associations) will be combined according to the specification from user and inserted into a newly created SourceList. The user has an ability to insert into this newly created SourceList not only associated sources but non-associated ones as well.

Let us see the case of an overlapped SourceLists (SL1 and SL2, Figure 6, sources are associated in the area of AssociateList AL). In the case of COMBINE_METHOD=1 all sources, including non-associated from the area out of bounding box of AssociateList will be included into newly created SourceList (Figure 6). In the case of COMBINE_METHOD=2 associated sources from the area covered by AssociateList will be included only (Figure 7), and in the case of COMBINE_METHOD=3 only non-associated sources will be inserted (Figure 8). Please, note, that usually area covered by AssociateList contains non-associated sources as well (sources which do not satisfy to requirements implied by the user for AssociateList). These sources will be included as well if user chooses COMBINE_METHOD=1 and COMBINE_METHOD=3.

Figure 6: Spatial localization in the case of COMBINE_METHOD=1

Figure 7: Spatial localization in the case of COMBINE_METHOD=2

Figure 8: Spatial localization in the case of COMBINE_METHOD=3

The life cycle of SourceList¶

The realization of CombinedList library allows to reuse of a newly created SourceList as an input for a new AssociateList. Figure 9 shows the possible life cycle of the new SourceList. New attributes are:

associatelist - the ID of the parent AssociateList,
filters - the list of magnitudes in the SourceList,
COMBINE_METHOD - the method used by CombinedList to make new coordinates/magnitudes,
for each photometric band 4 attributes - mag_i for magnitudes, magErr_i for errors of magnitudes, magFlag_i for the flag and magN_i for number of stars combined for this value of magnitude.

As result there are three types of SourceLists (in the single SourceList class):

“original” SourceList - a SourceList from sextractor
“external data source” SourceList - a sourcelist ingested from an external to Astro-Wise catalog. This SourceList can contain a number of magnitudes per source
“combined” SourceList produced by CombinedList

Figure 9: New SourceList cycle

Functions and Attributes¶

Initializing¶

CombinedList must be initiated with the AssociateList of type 1 or 2.

awe> cl = CombinedList(input_AssociateList)

The place of CombinedList is in astro.main.CombinedList.

Setting method of combining of sources¶

awe> cl.set_combined_method(COMBINE_METHOD)

COMBINE_METHOD can be 1, 2 or 3. By default COMBINE_METHOD=2

Setting user defined magnitudes¶

awe> cl.set_user_defined_magnitudes(attributes)

attributes is a dictionary which specify magnitude-like attributes defined by the user. The format of the dictionary is

awe> attributes = {<attribute_name>:<filter_name>,}

for example,

awe> attributes = {'MAG_IZO':'#844','MAG_AUTO':'#849'}

Please, note that the definition like this:

awe> attributes = {'MAG_IZO':'#844','MAG_AUTO':'#844'}

will join MAG_IZO and MAG_AUTO in one attribute in newly created SourceList.

Setting user defined attributes¶

Unlike user defined magnitudes user defined attributes will be joint in a new SourceList in an attribute with the same name. For example,

awe> cl.set_user_defined_attributes(attributes)

where attributes is a list

awe> attributes = ['B','MAG_RAD']

will create in a new SourceList two attributes B and MAG_RAD. The aggregation function by default is AVG, but the user can set any Oracle aggregation function as attribute with the same name. For example,

awe> cl.set_aggregate_functions(attributes)

where attributes is a dictionary

attributes = {<attribute_name>:<aggregate_function>}

For example,

awe> attributes = {'B':'MAX','MAG_RAD':'MIN'}

Setting magnitude flags¶

The user can accompany a magnitude with a flag taken from the SourceList using a function

awe> cl.set_magnitude_flags(attributes)

where

attributes={<magnitude_name>:<flag_attribute>}

For example, if the user would like to insert magnitude flags for MAG_ISO from MAG_FLAG

awe> attributes = {'MAG_ISO':'MAG_FLAG'}

By default the maximum flag will be taken but the user can select aggregate function the same way it was described above.

awe> attributes={'MAG_FLAG':'MIN'}

Debugging¶

The user can set a debugging during creating of a SourceList

awe> cl.set_debug()

The debugging information shows user executed SQL statements and time spent for execution.

Making and committing¶

awe> cl.make()

and

awe> cl.commit()

the last function shows the information for created SourceList.

Use in practice¶

AssociateLists is created. How to make a new SourceList from AssociateList? This can be done with CombinedList library.

First, let us select an AssociateList

awe> als = (AssociateList.ALID==7842)
awe> als[0].info()
Name of AssociateList :Assoc-GVERDOES-------------------53923.5717701.0000007842.alist
AssociateList ID      :7842
Associates in list    :26360

Then we initialize a CombinedList with the selected AssociateList

awe> cl = CombinedList(als[0])

We have to select method which we will use to combine data. There are three of them: combined_method=1 will include in the resulting SourceList all sources (associated by AssociateList and non-associated), combined_method=2 will include only sources which are presented in AssociateList, and combined_method=3 will include only sources which are presented in SourceLists (used to create AssociateList) but not in AssociateList itself. By default combined_method=2.

awe> cl.set_combined_method(1)

We have also to specify which attributes of input SourceLists (input to AssociateList) we would like to see in the output SourceList. There are two modes: to treat attributes as a magnitude attribute (4 new attribute will be created - MAG_1 to store average value, MAGFLAG_1 to store flag for the magnitude, MAGERR_1 to store rms of the value, MAGN_1 to store a number of input values, 2 or 1 in the example) or to specify the attribute as user-defined with the user-selected aggregate function.

For example, we want to see in the output SourceList MAG_ISO which is a magnitude in the Astro-WISE filter '#844'.

awe> cl.set_user_defined_magnitudes({'MAG_ISO':'#844'})

At the same time we want to see in the output SourceList FLUX_RADIUS and YM2, and we want a maximum value for the first attribute not an average.

awe> cl.set_user_defined_attributes(['FLUX_RADIUS','YM2'])
awe> cl.set_aggregate_functions({'FLUX_RADIUS':'MAX'})

Next we make and commit a new SourceList

awe> cl.make()
awe> cl.commit()
SourceList: Name of SourceList : SL-YOURNAME-0000423231
SourceList ID      : 423231
Sources in list    : 34645
Parameters in list : 15

 |
 +-COMBINE_METHOD: 1
 +-OBJECT:
 +-SLID: 423231
 +-associatelist: 7842
 +-astrom_params: None
 +-chip: None
 +-creation_date: 2008-08-14 15:46:20.921300
 +-detection_frame: None
 +-filename:
 +-filter: None
 +-filters: MAG_1:#844
 +-frame: None
 +-globalname:
 +-instrument: None
 +-is_valid: 1
 +-llDEC: -43.2780281843
 +-llRA: 201.747154592
 +-lrDEC: -43.2780165148
 +-lrRA: 200.962548787
 +-name: SL-YOURNAME-0000423231
 +-number_of_sources: 34645
 +-object_id: '5220897963995C33E0407D81C5063020'
 +-process_params: <astro.main.SourceList.SourceListParameters object at 0xb42bc5ac>
 +-sexconf: <astro.main.Config.SextractorConfig object at 0xb42bcd4c>
 +-sexparam: <class 'common.database.typed_list.typed_list'>(<type 'str'>, [])
 +-sources: {'MAGFLAG_1': <type 'long'>, 'YM2': <type 'float'>,
    'MAG_1': <class'common.util.types.single_float'>,
    'FLUX_RADIUS': <class 'common.util.types.single_float'>,
    'MAGERR_1': <class 'common.util.types.single_float'>,
    'MAGN_1': <type 'long'>}
 +-ulDEC: -42.7262721851
 +-ulRA: 201.743662417
 +-urDEC: -42.7262607388
 +-urRA: 200.966071491
None

As we can see, new SourceList with SLID=423231 has attributes MAG_1 (contains MAG_ISO from input SourceLists), FLUX_RADIUS (maximum FLUX_RADIUS from input SourceLists) and YM2 (average YM2 from input SourceLists) and contains 34645 sources.