Mortar has joined Datadog, the leading SaaS-based monitoring service for cloud applications. Read more about what this means here.

Secure Configuration Parameters

Secure Configuration Parameters allow you to store sensitive values for use in Pig and Luigi scripts.

About Secure Configuration Parameters

Often when developing a Luigi pipeline or a Pigscript you need to use sensitive values like passwords, credentials, or connection strings. To keep these values out of your code, Mortar offers Secure Configuration Parameters.

Each Mortar Project gets its own set of Secure Configuration Parameters. These are stored encrypted at Mortar and are made available to your Pig and Luigi scripts whenever they are run.

Setting a Configuration Parameter

To set a configuration parameter, use the mortar config:set command from the root of your Mortar project.

mortar config:set "PASSWORD=my_password"

View Configuration Parameters

To view all the configuration parameters for the current project:

mortar config

This will list all parameters and their values for the project. You can also look at a single value using config:get

mortar config:get PASSWORD

Remove a Configuration Parameter

To remove a parameter, use the config:unset command.

config:unset PASSWORD

Automatic Configuration Parameters

In addition to the secure configuration parameters you define, Mortar provides a handful of automatic parameters for use in your code:

  • MORTAR_EMAIL: Your Mortar login email (e.g. myemail@me.org)
  • MORTAR_EMAIL_S3_ESCAPED: An escaped version of your email, suitable for use in file paths (e.g. myemail-me-org)
  • MORTAR_API_KEY: Your Mortar API key
  • MORTAR_PROJECT_ROOT: The absolute path where your project is being run from. Useful for forming paths to register other code or JARs.
  • MORTAR_PROJECT_NAME: The name of your project.
  • AWS_ACCESS_KEY_ID: Your configured AWS access key ID (also available as aws_access_key_id or AWS_ACCESS_KEY)
  • AWS_SECRET_ACCESS_KEY: Your configured AWS secret access key (also available as aws_secret_access_key or AWS_SECRET_KEY)

Using Configuration Parameters in a Pigscript

Whenever you validate, illustrate, or run a Pigscript, your configuration parameters will be passed to the script as Pig parameters. This happens when you're running locally or on the Mortar platform.

So, to use a configuration parameter in your Pigscript, just reference it as you would any other Pig parameter. For example, if you've set a configuration parameter named CONN_STRING for your project, you might reference it in a Pigscript like this:

    data = LOAD 'mongodb://$CONN_STRING'
         USING com.mongodb.hadoop.pig.MongoLoader();

Using Configuration Parameters in a Luigiscript

To provide configuration to a Luigi script, put it into the client.cfg.template file, which is stored in your Mortar project's luigiscripts directory.

An example client.cfg.template might look like:

[mortar]
email: ${MORTAR_EMAIL}
api_key: ${MORTAR_API_KEY}
host: api.mortardata.com
project_name: ${MORTAR_PROJECT_NAME}

[myconfigsection]
db_password: ${DB_PASSWORD}

Each time you run Luigi, either via mortar local:luigi or mortar luigi, Mortar will use your project's Secure Configuration Parameters to fill in the referenced parameters (like DB_PASSWORD and MORTAR_EMAIL above).

The results of this expansion will be stored in luigiscripts/client.cfg. You should not check luigiscripts/client.cfg into source control, as it will be generated on each new execution of Luigi.

If you reference any undefined configuration parameters in your client.cfg.template, the expansion will fail with an error message, allowing you to fix the issue.

Inside of your Luigiscript, you can reference your configuration values in this way:

from luigi import configuration

config = configuration.get_config()
value = config.get('myconfigsection', 'db_password')