This quick tutorial will walk you through the basics of building a data pipeline with a Luigi script that runs multiple tasks in succession.
Luigi is great for linking complex pipelines of Mortar jobs together, but it can also interact with AWS services, run shell scripts, and more. Absolutely anything you can do in Python can be run as part of Luigi.
In this example, we will craft a simple Luigi script that checks if input data is available, runs a Mortar job that processes that data and stores the results to S3, and then checks that the output file contains the expected output.
What You'll Need:
A Mortar account. If you don't have a Mortar login, you can request a free trial account here.
The Mortar Development Framework. You can download the Mortar development framework here. It's the fastest way to develop and run data pipelines.
Ready to build your first data pipeline? Let's begin!