Mortar has joined Datadog, the leading SaaS-based monitoring service for cloud applications. Read more about what this means here.

Automate Your Pipeline

Now that you have a working Luigi recommendation engine pipeline, you will want it to run every night (or week) to keep your recommendations fresh. Mortar provides built-in job scheduling for Luigi pipelines to make this easy.

Scheduling via Mortar

It's easy to schedule your Luigi pipeline to run via Mortar. To do so, follow the instructions on the Job Scheduling page to schedule the Luigi Pipeline you created. Be sure to add any parameters that your script takes to the scheduled job.

Your pipeline will run automatically as often as you need and will provide email notification upon completion. If you need the pipeline to wait on arriving data, you can set the "Retry Failures" option, which will automatically retry your job every 15 minutes until data arrives.

Other Options

Scheduling via CRON

CRON is a lightweight scheduler included on most UNIX style systems. It can be a bit cranky to work with, but is widely-available. Here are a few gotchas worth mentioning if you use CRON to schedule your job:

  • The time field for schedules uses the 24 hour format so 7pm should be:
0 19 * * * /opt/run_luigi.sh
  • You need to provide the full path to commands to avoid execution issues unless you explicitly set PATH= as part of your CRON entry
  • The scripts and commands you want to run need the execute bit set
chmod +x /opt/run_luigi.sh
  • You need to escape special characters in order to avoid execution issues. For example
0 19 * * * /opt/run_luigi.sh && date +"\%d" >> /tmp/run.history

Scheduling via Jenkins

While CRON is readily available it can take a bit of effort to setup just right. Depending on your environment CRON can also be quite challenging to debug on an ongoing basis. If you happen to have Jenkins or a similar build system available, you can use it to run your Luigi pipelines. This will provide better visibility into to runtime issues and make changing pre-run, runtime and post-runtime scripts easier.

Example Pipeline trigger script run_luigi.sh

Here is an example script that you can customize if you decide to schedule with CRON or jenkins:

#!/bin/bash
# This is an example script to run a Mortar pipeline via Luigi.


# we want the pipeline to fail if any part of this script has errors
set -e

# prerequisites 
# make sure the mortar cli is installed
# curl -sSL install.mortardata.com | bash
# make sure you are logged into mortar
# mortar login

# run the pipeline
# note: the return value of the python script will be 0 even if
# tasks fail. if you want alerting you must rely on sending alerts
# inside of luigi via email notifications

cd YOUR_PROJECT_ROOT_DIRECTORY
mortar luigi luigiscripts/$LUIGI_SCRIPT --my-param my-value