Now that you have a working Redshift ETL pipeline, you will want it to run frequently to keep your data fresh. Mortar provides built-in job scheduling for Luigi pipelines to make this easy.
It's easy to schedule your Luigi pipeline to run via Mortar. To do so, follow the instructions on the Job Scheduling page to schedule the Luigi Pipeline you created. Be sure to add any parameters that your script takes to the scheduled job.
Your pipeline will run automatically as often as you need and will provide email notification upon completion. If you need the pipeline to wait on arriving data, you can set the "Retry Failures" option, which will automatically retry your job every 15 minutes until data arrives.
CRON is a lightweight scheduler included on most UNIX style systems. It can be a bit cranky to work with, but is widely-available. Here are a few gotchas worth mentioning if you use CRON to schedule your job:
0 19 * * * /opt/run_luigi.sh
chmod +x /opt/run_luigi.sh
0 19 * * * /opt/run_luigi.sh && date +"\%d" >> /tmp/run.history
While CRON is readily available it can take a bit of effort to setup just right. Depending on your environment CRON can also be quite challenging to debug on an ongoing basis. If you happen to have Jenkins or a similar build system available, you can use it to run your Luigi pipelines. This will provide better visibility into to runtime issues and make changing pre-run, runtime and post-runtime scripts easier.
Here is an example script that you can customize if you decide to schedule with CRON or jenkins:
#!/bin/bash # This is an example script to run a Mortar pipeline via Luigi. # we want the pipeline to fail if any part of this script has errors set -e # prerequisites # make sure the mortar cli is installed # curl -sSL install.mortardata.com | bash # make sure you are logged into mortar # mortar login # run the pipeline # note: the return value of the python script will be 0 even if # tasks fail. if you want alerting you must rely on sending alerts # inside of luigi via email notifications cd YOUR_PROJECT_ROOT_DIRECTORY mortar luigi luigiscripts/$LUIGI_SCRIPT --my-param my-value