Luigi is a powerful, easy-to-use framework for building data pipelines. It was developed at Spotify, where it runs thousands of jobs per day.
Luigi manages all of the plumbing typically associated with long-running batch processes. You want to chain many tasks together, automate them, and gracefully handle any failures. You need to run Hadoop jobs, but you also need to do many other things for a production pipeline—setting up database tables, waiting for files to arrive on S3 or FTP, calling external APIs. Luigi makes it easy to stitch these together into a pipeline and places no restrictions on what you can run.
Luigi pipelines automatically handle dependency resolution, checkpointing and failure recovery, parallel execution, command line integration, and much more. Pipelines are expressed as simple, easy-to-read Python that can be reused anywhere. The open source community has contributed a library of common tasks, with frequent updates and contributions.
The Mortar platform offers full support for Luigi. You can build pipelines on your computer with the Mortar Development Framework, and then deploy and run them in the cloud with one command.