Mortar has joined Datadog, the leading SaaS-based monitoring service for cloud applications. Read more about what this means here.

Cluster Management Tasks

Cluster management Tasks manage Hadoop clusters using Mortar's API.

mortar-luigi

These Tasks are part of the mortar-luigi project, an open-source collection of extensions for using Mortar from within Luigi. It is installed automatically when you use the mortar local:luigi or mortar luigi commands.

MortarClusterShutdownTask

This task shuts down any idle clusters. This just saves a little on AWS costs if it runs at the end of every Luigi script.

Example Usage

class ShutdownClusters(mortartask.MortarClusterShutdownTask):

    def requires(self):
        return [MyBasicPigscriptTask()]

    def output(self):
        return ['path-to-token-output']

This task just needs a requires method indicating its dependencies, and an output method decribing where to write a token indicating that it has completed.

Example in Context

The Luigi scripts in the mortar-recsys project each use ShutdownClusters as the final task in the pipeline.