Mortar has joined Datadog, the leading SaaS-based monitoring service for cloud applications. Read more about what this means here.

Using Your Own AWS Account

Mortar supports running your Pig jobs on Elastic MapReduce clusters running in your own AWS account.

Using Your Own AWS Account

By default your Mortar jobs are run on private Elastic MapReduce clusters running in the US-East region of Mortar's AWS account. Any associated EC2 and Elastic MapReduce costs are then passed on to you on your Mortar bill. This works well for most people as it allows Mortar to handle all of the AWS configuration required to run your jobs.

However, there are a few reasons you might want to have Mortar run your Pig jobs in your own AWS account:

  1. Multi-Region Support: If your S3 data is in a region other than US Standard, you will want to run your Elastic MapReduce clusters in the same region to minimize latency, costs, and potential regulatory requirements.
  2. Credits and Reserved Instances: By running the Elastic MapReduce jobs in your own account you gain flexibility on how you pay for your cluster costs. You can use AWS credits or purchase reserved instances.

Getting Started

Here's how to get started running Mortar jobs in your own AWS account.

Create a Mortar Account

In order to get started you need a Mortar account. You can request a free trial of Mortar here.

Contact Support

Send an email to support@mortardata.com from the email address you signed up with asking us to switch you over to running in your own AWS account. In the email please include the AWS region you would like us to use to launch your clusters. Once you have received a response from Mortar Support you can continue with the following steps.

Create a Mortar IAM User

In order to run Mortar jobs in your AWS account, Mortar requires permissions to launch Elastic MapReduce clusters in your account, to access Mortar's S3 buckets, and to write logs to a bucket in S3. It's easy to grant these permissions with the AWS Identity and Access Management (IAM) console.

Log in to your AWS IAM Console:

IAM Console

Choose the Users option from the lefthand sidebar, and click the "Create New Users" button:

Create New Users

Provide a descriptive user name, such as "mortar-iam-emr", leave "Generate an access key for each User" checked, and click "Create":

Enter User Names

Open the "Show User Security Credentials" section, and write down the "Access Key Id" and "Secret Access Key ID" for your newly created user:

Show Security Credentials

While still on the Users tab in the IAM Console, click the checkbox next to the name of the user you created:

Click User

Choose the "Permissions" tab at the bottom of the screen, and click "Attach User Policy":

Attach User Policy

Choose "Custom Policy" from the list of options, and hit "Select" to open the policy editor.

Manage User Permissions

Name your policy 'Mortar-IAM-EMR-Policy'

Next, you need to write the Policy Document. It is written in Amazon's JSON IAM policy format.

Here's the standard policy granting the Mortar user all of the necessary permissions. Be sure to change "[YOUR LOG BUCKET]" to one of your own S3 buckets. This bucket will contain the Elastic MapReduce logs from your jobs.

  {
  "Statement": [
    {
      "Action": [
        "s3:*"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::*elasticmapreduce",
        "arn:aws:s3:::*elasticmapreduce/*",
        "arn:aws:s3:::mortar-emr-bootstrap",
        "arn:aws:s3:::mortar-emr-bootstrap/*",
        "arn:aws:s3:::[YOUR LOG BUCKET]",
        "arn:aws:s3:::[YOUR LOG BUCKET]/*"
      ]
    },
    {
      "Action": [
        "elasticmapreduce:*",
        "cloudwatch:*"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Action": [
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CancelSpotInstanceRequests",
        "ec2:CreateTags",
        "ec2:RequestSpotInstances",
        "ec2:RunInstances",
        "ec2:ModifyImageAttribute",
        "ec2:DescribeAccountAttributes",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstances",
        "ec2:DescribeKeyPairs",
        "ec2:DescribeRouteTables",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSpotInstanceRequests",
        "ec2:DescribeSpotPriceHistory",
        "ec2:DescribeSubnets",
        "ec2:RevokeSecurityGroupIngress"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
 }

Click "Apply Policy":

Set the Mortar IAM user permissions

Add the keys you copied down to your Mortar account on the Mortar EMR Cluster Settings page. If you can not view this page contact support@mortardata.com to ensure your Mortar account has been configured to run jobs in your own AWS account.

Mortar EMR Cluster Settings

On this same page you can set the S3 bucket where your Elastic MapReduce jobs will write their logs.

Mortar EMR Log Bucket