Mortar has joined Datadog, the leading SaaS-based monitoring service for cloud applications. Read more about what this means here.

Cluster Management Tasks

Cluster management Tasks manage Hadoop clusters using Mortar's API.


These Tasks are part of the mortar-luigi project, an open-source collection of extensions for using Mortar from within Luigi. It is installed automatically when you use the mortar local:luigi or mortar luigi commands.


This task shuts down any idle clusters. This just saves a little on AWS costs if it runs at the end of every Luigi script.

Example Usage

class ShutdownClusters(mortartask.MortarClusterShutdownTask):

    def requires(self):
        return [MyBasicPigscriptTask()]

    def output(self):
        return ['path-to-token-output']

This task just needs a requires method indicating its dependencies, and an output method decribing where to write a token indicating that it has completed.

Example in Context

The Luigi scripts in the mortar-recsys project each use ShutdownClusters as the final task in the pipeline.