Matillion ETL Job
Author: Matillion
Date Posted: Oct 28, 2023
Last Modified: Nov 30, 2023
Author: Matillion
Date Posted: Oct 28, 2023
Last Modified: Nov 30, 2023
Metadata API SQL Logging
A pair of Matillion ETL jobs that maintain a table of the SQL executed by Matillion ETL.
The SQL information can be used to construct data lineage.
The orchestration job uses the Metadata API to acquire the data. Afterwards it calls the transformation job to copy the newly loaded data into a permanent table.
Features of these jobs include:
- High water mark loading with Matillion ETL
- Demonstration of bronze-to-silver layer data transformation in a medallion data architecture
- How to un-nest and relationalize semi-structured data
- Using an API Extract
Prerequisites
You must first install the Metadata API Extract connector.
To run the jobs in your own environment you must supply values for the four Job Variables
| Job Variable | Description |
|---|---|
| prvt_api_username | A Matillion ETL user that has permission to use the API |
| prvt_api_password | That user’s password |
| prvt_hwm_timestamp | You can leave this blank and the job will still run OK. Set it to 1000000 if you wish to validate the orchestration job |
| prvt_staging_bucket_name | The name of an S3 bucket to be used for staging. Do not include the s3:// prefix nor any path |
Downloads
Licensed under: Matillion Free Subscription License
- Download METL-aws-sf-1.68.3-metadata-api-sql-logging.json
- Platform: AWS
- Target: Snowflake
- Version: 1.68.3 or higher