Apache Spark is a general-purpose engine for processing and analyzing large data sets. In addition to its core features, Spark includes a scalable machine learning library called MLlib that implements many of the iterative algorithms used in data science.
In some settings Spark is beginning to replace MapReduce for certain data processing tasks due to its performance improvements, but it is a less mature technology than MapReduce and hence may not be appropriate for all production workloads.
Mortar supports running Spark scripts from the Mortar Web application or the command line via the Mortar Development Framework. At present Spark access is open to a limited number of Mortar customers—if you are interested in running Spark on Mortar please drop us a note for more information.