Mortar has joined Datadog, the leading SaaS-based monitoring service for cloud applications. Read more about what this means here.

Set up a Mortar ETL Redshift Project

All code that runs using the Mortar framework is stored in git-backed Mortar projects. The first step in running an example is to set up a project.

Public vs Private

Mortar Projects come in two flavors: Public and Private. Public Mortar Projects can be viewed and forked by anyone. Private Mortar Projects are only accessible to users in your Mortar account. When creating a new project you will need to decide if this is code that you would like to share with the world or if this is code that you want to keep within your organization.


Create your project

To allow you to get started quickly we have created a base Redshift project that you can fork to create your own project. This project contains a number of runnable examples and templates. The base Mortar project is open source and hosted in Github at https://github.com/mortardata/mortar-etl-redshift.

Mortar Projects share one global namespace, and so each project must have a unique name.

mortar projects:fork git@github.com:mortardata/mortar-etl-redshift.git <your_project_name>

To read more about Mortar projects see How Mortar Projects Work.


Psycopg Library

In order to connect to Redshift this project uses psycopg2 a popular PostgresSQL adaptor for Python. This library requires your system to have python-dev and libpq-dev packages. Mortar will attempt to use this library and will generate an error if you do not have the required packages installed.

To test your project, run the following command from the root directory of your newly forked project:

mortar local:configure

This is the output if your system is all set up and ready to go:

=> mortar local:configure

Installing pig-0.12 to .mortar-local... done
Installing pig dependencies to .mortar-local... done
Installing python to .mortar-local... done
Installing user defined python dependencies... done
Installing python dependencies to .mortar-local... done
Installing jython to .mortar-local... done

However, if you see output like:

=> mortar local:configure

Installing pig-0.12 to .mortar-local... done
Installing pig dependencies to .mortar-local... done
Installing python to .mortar-local... done
Installing user defined python dependencies... failed
 !
 !    Unable to setup a python environment with your dependencies, see dependency_install.log for more details

you will need to install the psycopg2 dependencies.

Mac OS X

On Mac OS X you can use Homebrew to install PostgresSQL which will add all the dependencies necessary for psycopg2.

brew install postgresql

For alternative installation options see Installing on Mac OS X.

Ubuntu/Debian

To install the required packages you can do:

sudo apt-get install libpq-dev python-dev

For more information see Psycopg Install Instructions.


Getting Code Updates

This project is now in your Mortar account. If in the future you want to get updates made to Mortar's version of mortar-etl-redshift, use:

git fetch base
git merge base/master

And that's it! You now have all of the code forked into your own Mortar project all ready to go.


Setting up your Development Environment

Before getting into the details of our project, take a minute to set up your development environment. You should pick an editor you want to use for viewing and working with code. If you don't already have a favorite editor you can review some options here.