Parallel Astronomical Data Processing with Python: Recipes for multicore machines

Most astronomers (myself included) have a high performance compute engine on their desktops. Modern computers now contain multicore processors, whose development was prompted by the need to reduce heat dissipation and power consumption but which give users a powerful processing machine at their fingertips. Singh, Browne and Butler have recently posted a preprint on astro-ph, submitted to Astronomy and Computing, that offers recipes in Python for running data parallel processing on multicore machines. Such machines offer an alternative to grids, clouds and clusters for many tasks, and the authors give examples based on commonly used astronomy toolkits.

The paper restricts itself to the use of CPython’s native  multiprocessing module, for two reasons: much astronomical software is written in it, and it places sufficiently strong restrictions on managing threads launched by the OS that it can make parallel jobs run slower than serial jobs (not so for other flavors of Python, though, such as PyPy and Jython). The authors also chose to study data parallel applications, which are common in astronomy, rather than task parallel applications. The heart of the paper is a comparison of three approaches to multiprocessing in Python, with sample code snippets for each:

• Pool/Map, which spawns a pool of worker processes and returns a list of results;
• Process/Queue, which supports multiple arguments as input to a function through the process class; and
• Parallel Python, an open source cross-platform module (distinct from the multiprocessing module) that offers dynamic computation resource allocation as well as dynamic load balancing at runtime.

The study ran benchmarks on three quad-core machines -  a homebuilt one with an AMD processor, a Dell Studio XPS, and and an iMac for three use cases, all embarrassingly parallel, as described below. These examples all use modules from widely used astronomy toolkits such as IRAF and DAOPHOT.

Coordinate Transformation of CCD Pixel Data to Sky Coordinates for HST Images.

The performance – speed-up vs. number of processes, is shown below:

The principal conclusions are: