Astronomical data analysis with Apache Spark

The coming decade promises to be extremely exciting for astronomers and data/computer scientists alike with the coming of Large Synoptic Survey Telescope (LSST), James Webb Space Telescope, and others. These projects will produce a huge amounts of data that need to be searched, corellated, analyzed and learned from in order to find answers to the questions, such as “What are Dark Energy and Dark Matter?”, “How did our Universe form?”, “How many Earth-threatening asteroids are out there?” LSST with its unique architecture will go both “wide” and “deep”, meaning that it will acquire images of large parts of the sky capturing the most distant galaxies. It will continually scan the visible sky during the period of 10 years and will produce the first video of the Universe in history.

These new and exciting times require new tools that will help astronomers perform these analytical tasks more efficiently. In collaboration with astronomers from the University of Washington I built AXS, Astronomy Extensions for Spark, a tool based on Apache Spark, designed for fast cross-matching of astronomical catalogs and easy astronomical data processing. In this talk I will go through details of AXS’ architecture and explain why it is so fast.