Matillion ETL Shared Job

Author: Matillion
Date Posted: Nov 10, 2023
Last Modified: Nov 14, 2023

Run Serverless via AWS Lambda

Run your code via AWS Lambda serverless cloud infrastructure.

This shared job enables you to run scripting code using AWS Lambda, rather than locally on your Matillion ETL instance with a Python Script or Bash Script component.

Using serverless cloud infrastructure in this way has two DevOps advantages:

  • Instantly gives you unlimited scalability
  • You don’t have to install and manage Python libraries on your own Matillion ETL instance

To use this shared job, create your AWS Lambda source code as a .zip file and save it into an S3 bucket. You must also create an IAM role with the privileges that the Lambda function will need to run successfully.

Run Serverless via AWS Lambda

Parameters

ParameterDescription
AWS RegionAWS region e.g. eu-west-1
Handler NameName of the Lambda handler function
IAM Role ARNThe ARN of the IAM role for the function execution. Pattern: arn:(aws[a-zA-Z-]*)?:iam::\d{12}:role/?[a-zA-Z_0-9+=,.@-_/]+
Lambda function nameThe name of the Lambda function, e.g. tmp_myfunction
Memory MbAmount of memory to allocate, in Mb
Lambda RuntimeThe Lambda Runtime, e.g. python3.6
Source bucket nameName of S3 bucket containing the source code
Source file nameFilename of the source code, e.g. mysource.zip
Timeout secsTimeout for the function, in seconds
S3 log outputFull S3 path for the output logfile E.g. s3://mybucket/mypath/mylogfile.log

Prerequisites

The aws command line utility must be installed on your Matillion ETL instance. If the shared job fails with an error line X: aws: command not found then please follow this guide to installing the aws command.

This shared job deletes and recreates the Lambda function every time before attempting to run it. Ensure that the EC2 instance credentials attached to your Matillion ETL instance include the privilege to manage Lambda functions, and to read from S3. For more information, refer to the “IAM in AWS” section in this article on RBAC in the Cloud.


Downloads

Licensed under: Matillion Free Subscription License

Installation Instructions

How to Install a Matillion ETL Shared Job