Mortar has joined Datadog, the leading SaaS-based monitoring service for cloud applications. Read more about what this means here.

S3 Tasks

S3 Transfer Tasks move files from a local directory to S3 (Amazon's Simple Storage Service), and vice versa.

mortar-luigi

These Tasks are part of the mortar-luigi project, an open-source collection of extensions for using Mortar from within Luigi. It is installed automatically when you use the mortar local:luigi or mortar luigi commands.

LocalToS3Task

Transfers file from a local directory to S3.

Example Usage

class UploadToS3(s3transfer.LocalToS3Task):

    # Parameters needed by LocalToS3Task 
    s3_path = luigi.Parameter()
    local_path = luigi.Parameter()

def requires(self):
        return [OtherTask()]

This task requires a specific pair of parameters: s3_path and local_path, which define its input and output paths. Note that both paths should include the file name, not just the directory.

In this Task the run and output methods are implemented by the base class S3TransferTask, so you do not need to include them here.

Example in Context

This example pipeline uses a LocalToS3Task to copy a graph (the output of an R script in the pipeline) from local storage to S3.

S3ToLocalTask

Transfers file from S3 to a local directory.

Example Usage

class DownloadFromS3(s3transfer.S3ToLocalTask):

    # Parameters needed by S3ToLocalTask 
    s3_path = luigi.Parameter()
    local_path = luigi.Parameter()

def requires(self):
        return [OtherTask()]

Just like LocalToS3Task, this task requires a specific pair of parameters: s3_path and local_path, both of which should include a file name. The Task will use these parameters as its input and output paths.

In this Task the run and output methods are implemented by the base class S3TransferTask, so you do not need to include them here.

Example in Context

Mortar's example pipeline that transforms and graphs data from the Olympics uses an S3ToLocalTask to copy the Olympics data set from S3 to the local filesystem.