The SMEI Real-Time Data Pipeline: From Raw CCD Frames to Photometrically Accurate Full-Sky Maps

Proc. SPIE 5901, 59011B, 1−7, 2005

Solar Physics and Space Weather Instrumentation
S. Fineschi and R.A. Viereck (eds.)
© SPIE − The International Society for Optical Engineering

The SMEI Real-Time Data Pipeline: From Raw CCD Frames to Photometrically Accurate Full-Sky Maps

P.P. Hick, A. Buffington and B.V. Jackson
Center for Astrophysics and Space Sciences, University of California San Diego

Abstract

The Solar Mass Ejection Imager (SMEI) records a photometric white-light response of the interplanetary medium from Earth over most of the sky in near real time. We present the techniques required to process the SMEI data from the time the raw CCD images become available to their final assembly into photometrically accurate maps of the sky brightness of Thomson scattered sun light. Steps in the SMEI data processing include: integration of new data into the SMEI data base; conditioning to remove from the raw CCD images an electronic offset (pedestal) and a temperature-dependent dark current pattern; placement ("indexing") of the CCD images onto a high-resolution sidereal grid using known spacecraft pointing information. During the indexing the bulk of high-energy-particle hits (cosmic rays), space debris inside the field of view, and pixels with a sudden state change ("flipper pixels") are identified. Once the high-resolution grid is produced, it is reformatted to a lower-resolution set of sidereal maps of sky brightness. From these we remove bright stars, background stars, and a zodiacal cloud model (their brightnesses are retained as additional data products). The final maps can be represented in any convenient sky coordinate system, e.g., Sun-centered Hammer-Aitoff or "fisheye" projections. Time series at selected sidereal locations are extracted and processed further to remove aurorae, variable stars and other unwanted signals. These time series of the heliospheric Thomson scattering brightness (with a long-term base removed) are used in 3D tomographic reconstructions. The data processing is distributed over multiple PCs running Linux, and, runs as much as possible automatically using recurring batch jobs ("cronjobs") mostly written in Python. The core data processing routines are written in Fortran, C++ and IDL.