POTSDAM RRA meeting Dec 10-11, 1996 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Participants: ============= (MA) Mike Arida, GSFC, Greenbelt (TB) Thomas Boller, MPE Garching (HB) Hermann Brunner, AIP Potsdam (MC) Mike Corcoran, GSFC, Greenbelt (JE) Jakob Englhauser, MPE Garching (JF) Jens-Uwe Fischer, AIP Potsdam (RG) Rainer Gruber, MPE Garching (DH) Dan Harris, SAO, Boston (GH) Guenther Hasinger, AIP Potsdam (NJ) Norbert Junkes, AIP Potsdam (IL) Ingo Lehmann, AIP Potsdam (JS) Joachim Siebert, MPE Garching (RS) Rodrigo Supper, MPE Garching (WV) Wolfgang Voges, MPE Garching (JW) Joachim Wambsganss, AIP Potsdam (MW) Mike Watson, Leicester University (Dec. 10) Abbreviations: ============== BSC = RASS Bright Source Catalog DPG = Data Products Guide HOPR = HRI Standard Analysis System RASS = ROSAT All-Sky-Survey RUH = ROSAT Users Handbook SASS = Standard Analysis Software System &&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&& Action items (AI) are compiled in the following, and also specifically marked (***) in the context 1) (AIP-MPE) ==> agreement of distribution of flagging sequences between AIP and MPE required (HB<==>JE: has been sorted out) 2) (all) ==> send flagging results to MPE (JE) asap. JE will provide up-to-date list by Jan 15, 1997 internally. 4-6 weeks required to check list internally (another meeting?) 3) (TB) ==> TB will adapt survey GUI to be used for pointed observa- tions. Output has to be compatible to previous version. 4) (MC) ==> MC will add the large area/kill capability to the IDL/HRI VI system, and also deactivate the 'n' button. 5) (TB,MC) ==> MC and TB will change the missing source routines (impro- vement on cursor position, results on intensity, size, S/N and HRs (for PSPC) 6) (all) ==> compile list of PSPC sequences with crashed INTAPE/RDF and and send it to Carlo Izzo, MPE. 7) (all) ==> During the next few months we are all to 'use' the new product so that we can 'quantify' how it is better than other products, and of course to improve on the interface and access. 8) (JE) ==> two lists (PSPC,HRI) will be provided by JE by Jan 15, 1997 9) (WV) ===> memorandum of understanding to be distributed to RRA exploder from MPE (for export of database) 10) (WV) ==> test 100 sequences. Check access to APM/ROE. 11) (GH,NJ) ==> take RIXOS fields as testbeds (optical data already available). Is automatic procedure possible to retrieve APM/ROE data? 12) (MPE) ==> User's Guide for PSPC 13) (DH) ==> User's Guide for HRI 14) (DH) ==> Standard Reference: DH will send draft to WV 15) (TB,MC) ==> adopt TB survey program for missed sources to RRA PSPC; together with MC also for HRI 16) (all to MC) ==> list of HRI sequences where smallest detect cells are completely missing within source list 17) (DH) ==> Is it possible to increase parameter (no. of sources) in the "detect" algorithm within "HOPR" to say 500? 18) (TB) ==> proposal for reduction of flags for the public data catalog 19) (all to JE) ==> determine fields which should be merged. Send list to JE (Konrad Dennerl) 20) (all to WV) ==> determine PIs of already merged fields where RRA cannot improve on available results. WV will contact PIs of these fields/projects &&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&& 0.0 Introductory items ====================== 0.1 Introduction GH: announcement of two X-ray meetings at AIP in 1997: a) XMM Working Group Meeting in April b) "X-ray Surveys" in June (not yet confirmed) 0.2 Chairperson, minutes HB (chair), NJ (minutes) 1.0 Current activities (Phase 1) ================================ 1.1 Status reports (RRA) from groups o SAO/GSFC: working OK. o Leicester: not yet started. o MPE: no RRA flagging, finished BSC. o Potsdam: working OK. MC: most important issue to bring catalog out asap! MC (SAO/GSFC): description of data pipeline at SAO/GSFC; data flagging limited by amount of incoming data (no overhead) not only public data are flagged, but all data directly after standard processing WV (MPE): regular RRA screening not yet started at MPE. But: 1400 sky fields from RASS analyzed. RBS catalog released (paper submitted) RRA work continued with new staff (JS, RS) ***************************************************************************** *AI (AIP-MPE) ==> agreement of distribution of flagging sequences between * * AIP and MPE required (HB<==>JE: has been sorted out) * ***************************************************************************** MW (Leicester): no statistics, since no flagging performed so far. Steve Sembay and Richard West both left RRA group; two new people (Matt Burleigh recently started, Pete Sohl will join Feb. 1). So far major contribution to database area Leicester will perform the screening of the 12% share of UK data 1.2 Statistics o Number of sequences checked at SAO/GSFC, Potsdam, MPE, Leicester; which percentages of total (PSPC/HRI) has been done, already ? 1927 sequences at SAO/GSFC 1200 sequences at AIP o Sequences done per month at each site ~130 sequences/month (SAO/GSFC) ~200 sequences/month (AIP) for detailed reference see WWW pages: http://www.aip.de:8080/~rra/rra_stat.html (AIP) ftp://heasarc.gsfc.nasa.gov/rosat/data/qsrc/www/STATS/ us_screen_results/us_screen_results.html (GSFC) contain statistics on missed source list, missed sources, false detections ==> to be expected: new HRI data, reprocessed data, re-RDF'ed data 1.3 Collection/merging of results from sites, export to remote sites. o Status browse archive last updated July 1996 problems to transfer data AIP-Leicester via ftp (connection too slow) similar between MPE-Leicester o At which intervals do we want to do this in the future ? ==> Discussion on data collection and format suggestion to transform database into ASCII file (easily possible from BROWSE) for distribution WV: MPE proposal to collect ALL flagged data and put them into browse tables. Redistribution to every site in BROWSE, ASCII and FITS format. ==> BROWSE should become primary data exchange format; ASCII and FITS (binary fits table)are provided as secondary formats. *************************************************************************** *AI (all) ==> send flagging results to MPE (JE) asap. JE will provide * * up-to-date list by Jan 15, 1997 internally. * * 4-6 weeks required to check list internally (another meeting?) * *************************************************************************** GH: important to proceed to go public (not "hiding" data) 1.4 Methods to increase the data flow o Completion of reprocessing; reducing the backlog (HRI/PSPC) 130-150 sequences/month/site required at the moment backlog at every site o Expected future data screening rate at each site manpower still limiting factor in data screening rate reprocessed PSPC data (Rev. 2) not finished before February 1997 (plus 6 months proprietary period) o Do we need to upgrade the existing flagging software ? TB: status report on progress with RASS data base (production of BS catalog) 80,000 sources with existence likelihood EL>10, 170,000 sources with EL>7 22394 BS candidates (CR>0.05cts/s, more than 15 photons, EL>15) 3582 spurious sources, 18611 detections most spurious sources close to extended sources (e.g. SNRs like Cygnus loop) faint sources are not checked so far Screening: in RASS started with 7 flags, for catalog reduced to 4-5 ROSAT BSC flags: n nearby sources p position problem e extended source d complex diffuse emission m source missed by SASS extent control is only in broad-band image subdivision in automatic and visual screening process GUI: all file-copying etc. is now done WITHIN GUI. Time estimate: appr. 10-15 min./field ==> Is it worth to change GUI for PSPC flagging within RRA? Has to be kept consistent with old checking policy... ==> How many flags should be kept for the catalog? Are 10 flags too many? Needs to be given further thought Source extraction radius extended for appr. 300 of 18811 sources (extraction radius larger by factor of 3) WV: two points for SASS: a) Is source real? b) Is extraction radius correct? For identification overlay of X-ray contours onto optical image required. **************************************************************************** *AI (TB) ==> TB will adapt survey GUI to be used for pointed PSPC obser-* * vations. Output has to be compatible to previous version. * **************************************************************************** **************************************************************************** *AI (MC) ==> MC will add the large area/kill capability to the IDL/HRI * * VI system, and also deactivate the 'n' button. * **************************************************************************** **************************************************************************** *AI (TB,MC) ==> MC and TB will change the missing source routines (impro- * * vement on cursor position, results on intensity, size, * * S/N and HRs (for PSPC) * **************************************************************************** DH: Problem for sequences with no photons. a) PSPC: Crash (INTAPE/RDF not working properly) b) HRI: Dummy qsrc-file What should be done for house-keeping? (data sets postponed so far) **************************************************************************** *AI (all) ==> compile list of PSPC sequences with crashed INTAPE/RDF and * * and send it to Carlo Izzo, MPE. * **************************************************************************** 1.5 Consistency and Dispersion o Presentation of latest dispersion results (HRI/PSPC) DH: HRI results: two important issues: a) n-flag: should NOT be touched manually in any sequence. b) m-flag: in some cases 50% dispersion - a worry HB: PSPC results: in general OK at 98% level. TB: Should manually changed flags be marked in every sequence? o Can we live with the current level of uniformity ? HB: reliability appr. 98%, but most flagging done automatically. Level between 95% and 99% for individual flags o Is it necessary or desirable to redo any previous screening ? Additional flagging only if simple post-processing possible (e.g. no changes to u=t sources in HRI fields). Automatic procedures to find sequences which require addi- tional correction (should not be more than a few percent). WV: flag Nos. could/should be reduced before final catalog. o Steps to improve dispersion in the future ? ==> Study results of dispersion tests ! Read the flag descriptions ! 1.6 HRI source match table o How can we recover it and get it into the final database ? DH: Is it worth to put it in final data base (matching cell sizes, Nos., S/N)? Should be contained in separate data base. WV: When to be done: before merging in browse tables (JE) or afterwards? MC: Attention: software would have to be changed for further flagging. DH: Could be done automatically (via script) for ASCII table. MC: Should only be done after testing ==> Postponed. 2.0 Dissemination of results ============================ In what respects is the RRA superior to other catalogs (RASS, WGACAT, MPECAT) ? ==> improved positions (Rev. 2, perhaps additional position correction) US tables (status April 1996) contain appr. 10000 PSPC- and 10000 HRI sources no systematic comparison with MPECAT and WGACAT so far, which is definitely required WV: 4-6 weeks test phase (data checking/reviewing with scientific questioning). *************************************************************************** *AI (all) ==> In the next few months we are all to 'use' the new product * * so that we can 'quantify' how it is better than other pro- * * ducts, and of course to improve on the interface and access. * *************************************************************************** ==> Problem for US data sets which are still in proprietary time scale. They cannot be utilized within the archive. Subdivision required. *************************************************************************** *AI (JE) ==> two lists (PSPC,HRI) will be provided by JE by Jan 15, 1997 * *************************************************************************** 2.1 Evaluation of product o Has the database achieved sufficient content to be useful? total No. of sequences: PSPC: appr. 4500, HRI: appr. 3200 present state: PSPC: 1233 sequences, appr. 30% HRI: 1945 sequences, appr. 55% o If not, when will this be ? MW: magic number 70% (in comparison to WGACAT) GH: appr. 50% required, should be available by March 15th. MW: timescale to be defined by: a) is everything OK? b) is everything OK to be released? 2.2 Methods of announcing, and timing. o Talks, poster papers at meetings,items in newsletters. WV: give groups the chance to comment/add to author list o Journal paper ? How can technical stuff (SASS, statistical questions etc.) be published? GH: only valid way: run simulations and compare them to the results. To be included or to be referred to: a) PSPC Products Guide (MPE) b) HRI Products Guide (SAO: Corcoran, Snowden) c) HOPR d) SASS Release of PSPC and HRI catalog should occur simultaneously in two adjacent papers (PSPCCAT, HRICAT) Papers to be discussed within groups, similarly constructed, common introduction. Author list: everyone who has contributed. Same author list for both papers, different order? Timing: aiming for time when catalog goes public (March/April 97??) Journal: Back to back in the same journal/same issue if possible A\&A as first choice MW: both projects require leading author asap in order to have it written! ==> Consensus that HRI journal paper should have US first author and PSPC journal paper should have german first author. o naming convention Problem: How to avoid several names for (tentatively) same source? (e.g. for 3C279, 200 sequences of the same source in RRA!) a) IAU convention: only coordinates, merging of results definitely required beforehand b) Source names contain sequence and source numbers. Not in agreement with IAU convention, same sources with different names ==> Discussion (again) postponed (WV: we have to think about it... DH: we have to drink about it... marks the end of the meeting...) 2.3 Export of database o Memorandum of understanding *************************************************************************** *AI (WV) ===> memorandum of understanding to be distributed to RRA * * exploder from MPE (for export of database) * *************************************************************************** 2.4 Data Base access Access from all participating institutes: AIP, GSFC, Leicester, MPE, SAO Required: a) PSPC: qsrc files, missed source list b) HRI: qsrc files, missed source list + 2" images, 5" images 2.5 Special interfaces o Correlation tools GH: not useful to provide ready correlations/identifications as service for the public Experience from WGACAT: must be done separately; too much work included as service for community WV: interface (e.g. SKYVIEW) has to be provided cf. ROSAT BSC: 18800 sources, mix of candidate list + identi- fications, e.g. from optical follow-ups Not aiming for identifications, but just for correlation tool GH: what do we collect, what kind of information is needed (e.g. optical overlays for 50000 sources???) Comparison to RIXOS (only 80 fields): a) position correction (APM, ROE) applied with ML method pos. accuracy from +/-15" down to +/-3-5" b) SIMBAD: for 20" error radius still big problems with identification ==> Can individual position correction per sequence be applied? GH: correction breaks down for sequences below 5 ksec for statistical reasons JF: positional accuracy probably most important issue (see experience from ROSAT Hard Survey. Possible to correct PSPC with HRI sequences? MC: cross-correlation easy within BROWSE, but difficult the other way round (e.g. in BROWSE: determine all sources which have NO identification candidate...) GH: Best strategy to correlate with primary catalogs (e.g. IRAS, RASS, FIRST) only and not with already derived products (e.g. SIMBAD, NED) TB: suggests overlay with optical images for all sources (could be done with a script in one week for 18000 sources). GH: but can only be done AFTER positional correction for each sequence where APM, ROE information is available GH: works for sequences with t>8ksec and b>30deg (in lower Gal. latitudes confusion problems roughly 20 sources per inner field, 40-50% of the exposures expected at high Gal. latitudes (15% of sky PSPC covered) *************************************************************************** *AI (WV) ==> test 100 sequences. Check access to APM/ROE. * *************************************************************************** *************************************************************************** *AI (GH,NJ) ==> take RIXOS fields as testbeds (optical data already * * available). * * Is automatic procedure possible to retrieve APM/ROE data?* *************************************************************************** 2.6 Documentation DH: Documentation (i.e. User's Guide) should be provided by people/groups directly involved in two separate documents. *************************************************************************** *AI (MPE) ==> User's Guide for PSPC * *AI (DH) ==> User's Guide for HRI * *************************************************************************** JE: DPG (http://ftp.rosat.mpe-garching.mpg.de/rosat_svc/doc/ productguide/ProdGuide.html) RUH (http://ftp.rosat.mpe-garching.mpg.de/rosat_svc/doc/handbook/) both available on the Web! DH: not fully describe, but refer to SASS documents in description o Should we include all SASS parameters, or just those generated by RRA ? If only the latter, how easy is it to get the DPG ? (i.e. for a typical user on the net). o Logistics: already agreed, ultimate will be latex, which can produce both postscript and html. 2.7 Standard Reference DH: standard reference and acknowledgement required! *************************************************************************** *AI (DH) ==> DH will send draft to WV * *************************************************************************** 3.0 Phase 2 (deferred items) =========================== 3.1 Missed sources o ROSAT source detection statistics: Results from Monte Carlo simulations (AIP) GH: No. of spurious sources decreases exponentially with likelihood. Detection threshold needs to be detrmied by Monte Carlo simulations. With Multi-ML-Fit, threshold could be significantly decreased. Problems with flux estimate at low values close to the flux limits of fields o How many weak/bright missed sources are there ? MC: 137 (PSPC) and 59 (HRI) for combined US results HB: PSPC: on average < 1 source per sequence; HRI: less. approx. 10-20 bright missed PSPC sources at AIP. NJ: Plots of examples. *************************************************************************** *AI (TB,MC) ==> adopt TB survey program for missed sources to RRA PSPC; * * together with MC also for HRI * *************************************************************************** *************************************************************************** *AI (all to MC) ==> list of HRI sequences where smallest detect cells * * are completely missing within source list * *************************************************************************** MC: common strategy on missed source detection required. Are these sources contained in either ML or MD detection tables? DH: appr. 1500 missed sources expected now, and 3000 in total What can be done, e.g. adding parameters by hand (e.g. position, intensity, S/N, HR1, HR2) ? o Obtaining reliable (uniformly defined) source parameters WV: only feasible semi-automatically: a) cut out search field around marked position b) look for other sources within that field and cut them out c) determine source parameters automatically (apply INTAPE/RDF resp. INTAPE/USR) o Is it realistic to think that we can run a 'pseudo-SASS' at these positions - both HRI and PSPC ? If so, who is going to do this, and when ? If not, how can we measure parameters ? who will do this when ? o What are our intentions/goals/aims for the missed sources: Get some science out for ourselves ? Get them into the public domain in a usable form ? 3.2 How to obtain reliable parameters for extended sources ? WV: Several method available (e.g. Wavelet analysis; Voronoi Tesselation Percolation). HB: Also needed for XMM SSC (Wavelet analysis). ==> Postponed. Needs further investigation. DH: How should missed sources be characterized in catalog? They cannot be covered by existing flags ==> Separate table (DH) or note within existing table (WV) required. Issue postponed (first catalog version will NOT contain missed sources). 3.3 Fields where source detection aborted (i.e. too many detections) o How may are there ? (PSPC/HRI) WV: Error in HOPR software: for more than 200 entries in one loop/ cell size, no source list is given o How do we deal with them ? For some obvious cases (e.g. M31 fields) HRI sequences could be reprocessed. *************************************************************************** *AI (DH) ==> Is it possible to increase parameter (no. of sources) * * in the "detect" algorithm within "HOPR" to say 500? * *************************************************************************** 3.4 Fields deferred for other reasons For PSPC, source detection aborts in case of strong extended sources (e.g. Cygnus loop). o How many such fields are there ? US: 16 (PSPC) and 44 (HRI) Germany: appr. 10 for PSPC o How do we deal with them ? DH: suggestion to give (rough) position and one source flag for such extended sources (similarly done in RASS BSC) D-flag has to be set Sequences can be identified by D- and E- sequence flags in both, PSPC and HRI. Mask out areas with extended structure (SNRs) and run SASS (PSPC) resp. HOPR (HRI) again on remaining fields 3.5 Source identification ==> see also 2.5: Special Interfaces o Can we improve on the SASS/Simbad automation ? o How can we get optical ID info into the source database ? o What can we do to get better radio data into the database ? GH: FIRST results included in NED, but correlation better done directly. Present status of FIRST: 138,000 sources within 1500 square degrees. 3000 square degrees already observed, 5000 expected for begin of 1997. See at http://sundog.stsci.edu/first/obsstatus.html. MC: Investigate whether FIRST can be accessed via SKYCAT. WV: MPE has collaboration with D. Helfand et al. GH: FIRST shows appr. 100 sources/square degree, corresponding to appr. 400 sources/sequence (radio). Appr. 20,000 sources/sequence are expected optically. Ergo: correlation only useful AFTER position correction! ==> General outcome: correlation and source identification leads too far, only the catalog should be provided for issue 1! o Cross correlation with the RASS GH: first step: correlation of RRA catalog with RASS BSC (look for variable sources). 3.6 Catalogues o Public not yet *************************************************************************** *AI (TB) ==> proposal for reduction of flags for the public data catalog * *************************************************************************** o Internal WV: a) Yes, catalog should be used internally for test purposes in the institutions b) in principle restricted to RRA team; could, however, be checked with people outside RRA in direct collaborations Caveat: only PUBLIC data (US: subdivision required) can be used for that purpose o Formats (ASCII, ...) see above 3.7 Merging of data WV: not planned for phase 1 (publish catalog Vers. 1) Two more catalogs required for merged sources. RG: New catalog required for Rev. 3 sources? o Which data should be merged ? In which fraction of the data does merging result in a desirable increased sensitivity ? o Merging technique a) find overlapping fields b) two positions within 30' (PSPC) resp. 20' (HRI) Example: M31 raster scans with 20' distance (RS) c) Position correction (see above) required before merging o By which system (SASS/PSPC/HRI, EXSAS ?) Survey Software could be applied for merging PSPC fields Problem: How to deal with position correction in (HRI) fields without enough sources to apply astrometry? Suggestion: Should perfectly matched fields like M31 (RS), Lockman hole (GH) directly be included in data base? First, number of areas where merging is applicable have to be determined, rest of overlapping areas is to be deferred. *************************************************************************** *AI (all to JE) ==> determine fields which could be merged. Send list * * to JE (Konrad Dennerl) * *************************************************************************** *************************************************************************** *AI (all to WV) ==> determine PIs of already merged fields where RRA * * cannot improve on available results. * * WV will contact PIs of these fields/projects * *************************************************************************** 4) Talks ======== In connection to the discussed RRA items, the following talks/contributions were presented: GH: Monte Carlo Simulations on ROSAT Source Detection Statistics comparison of LDETECT, MDETECT and Multi-ML missed sources (visual inspection breaks down at certain level) RS: On Differences SASS-EXSAS poor correlation between SASS- and EXSAS-detected sources (50% coincidence!) is significantly improved if "distorted" sequences (e.g. extended emission from SNRs) are not considered (resulting in 95% coincidence) RG: HRI Background Determination report on error in background determination model in aspect histogram which could lead to erroneous source determinnation for sequences with more than 100 entries in aspect histogram Different results on DEC/AXP and DEC/VAX. Differences only for sources with low likelihoods.