The ALMA Interferometric Pipeline Heuristics

Abstract

Abstract We describe the calibration and imaging heuristics developed and deployed in the Atacama Large Millimeter/submillimeter Array (ALMA) interferometric data processing pipeline, as of ALMA Cycle 9 operations. The pipeline software framework is written in Python, with each data reduction stage layered on top of tasks and toolkit functions provided by the Common Astronomy Software Applications package. This framework supports a variety of tasks for observatory operations, including science data quality assurance, observing mode commissioning, and user reprocessing. It supports ALMA and Very Large Array interferometric data along with ALMA and NRO 45 m single dish data, via different stages and heuristics. In addition to producing calibration tables, calibrated measurement sets, and cleaned images, the pipeline creates a WebLog which serves as the primary interface for verifying the quality assurance of the data by the observatory and for examining the contents of the data by the user. Following the adoption of the pipeline by ALMA Operations in 2014, the heuristics have been refined through annual prioritized development cycles, culminating in a new pipeline release aligned with the start of each ALMA Cycle of observations. Initial development focused on basic calibration and flagging heuristics (Cycles 2–3), followed by imaging heuristics (Cycles 4–5). Further refinement of the flagging and imaging heuristics, including the introduction of parallel processing, proceeded for Cycles 6–7. In the 2020 release, the algorithm to identify channels to use for continuum subtraction and imaging was substantially improved by the addition of a moment difference analysis. A spectral renormalization stage was added for the 2021 release (Cycle 8) to correct high spectral resolution visibility data acquired on targets exhibiting strong celestial line emission in their autocorrelation spectra. The calibration heuristics used in the low signal-to-noise regime were improved for the 2022 release (Cycle 9). In the two most recent Cycles, 97% of ALMA data sets were calibrated and imaged with the pipeline, ensuring long-term automated reproducibility of results. We conclude with a brief description of plans for future additions, including a self-calibration stage, support for multi-configuration imaging, and complete calibration and imaging of full polarization data.