Author: Matillion
Date Posted: Nov 10, 2023
Last Modified: Nov 14, 2023
Run Serverless via AWS Lambda
Run your code via AWS Lambda serverless cloud infrastructure.
This shared job enables you to run scripting code using AWS Lambda, rather than locally on your Matillion ETL instance with a Python Script or Bash Script component.
Using serverless cloud infrastructure in this way has two DevOps advantages:
- Instantly gives you unlimited scalability
- You don’t have to install and manage Python libraries on your own Matillion ETL instance
To use this shared job, create your AWS Lambda source code as a .zip file and save it into an S3 bucket. You must also create an IAM role with the privileges that the Lambda function will need to run successfully.
Parameters
| Parameter | Description |
|---|---|
| AWS Region | AWS region e.g. eu-west-1 |
| Handler Name | Name of the Lambda handler function |
| IAM Role ARN | The ARN of the IAM role for the function execution. Pattern: arn:(aws[a-zA-Z-]*)?:iam::\d{12}:role/?[a-zA-Z_0-9+=,.@-_/]+ |
| Lambda function name | The name of the Lambda function, e.g. tmp_myfunction |
| Memory Mb | Amount of memory to allocate, in Mb |
| Lambda Runtime | The Lambda Runtime, e.g. python3.6 |
| Source bucket name | Name of S3 bucket containing the source code |
| Source file name | Filename of the source code, e.g. mysource.zip |
| Timeout secs | Timeout for the function, in seconds |
| S3 log output | Full S3 path for the output logfile E.g. s3://mybucket/mypath/mylogfile.log |
Prerequisites
The aws command line utility must be installed on your Matillion ETL instance. If the shared job fails with an error line X: aws: command not found then please follow this guide to installing the aws command.
This shared job deletes and recreates the Lambda function every time before attempting to run it. Ensure that the EC2 instance credentials attached to your Matillion ETL instance include the privilege to manage Lambda functions, and to read from S3. For more information, refer to the “IAM in AWS” section in this article on RBAC in the Cloud.
Downloads
Licensed under: Matillion Free Subscription License
- Download METL-aws-1.61.6-run-serverless-aws-lambda.melt
- Platform: AWS
- Target: Any target cloud data platform
- Version: 1.61.6 or higher