Matillion ETL Shared Job
CDC Sync to Target
An updated version of the Shared Job used in Matillion ETL for CDC Processing.
This Shared Job is designed to be called by the Matillion CDC process. It will set all of the Public job variables required for the job to run.
This wrapper job:
- Sets up the Private variables
- Manages the logging table
- Runs the sub job with the required variable values
- Writes the result to the log
This updated version of CDC Sync To Target will convert date time with zero values to null which addresses a specific use case.
Parameters
| Parameter | Description |
|---|---|
| Source Engine | The source database type (E.g MySQL). Must match the name expected by the Matillion Database Query Component |
| Server Name | The IP or DNS address of the source database server |
| Port | The port number for the source database server |
| Database | The name of database on the source host to connect to |
| User Name | The username on the source database server |
| Password | The Matillion “Password Manager” password associated with the source database |
| S3 Bucket | The S3 bucket used to store the CDC files produced by AWS DMS |
| S3 Key | The location within the S3 bucket of the CDC DMS file to be processed |
| Table | The table name |
| Staging Database | The Snowflake database where the staging tables will be created during a CDC update load |
| Staging Schema | The Snowflake schema where the staging tables will be created during a CDC update load |
| Staging Table Prefix | The Prefix that will be applied to the staging tables created for a CDC update load |
| Target Database | The target database |
| Target Schema | The Snowflake schema to be used for the target tables |
| Target Table Prefix | The prefix added to the Source Table Name (from the source database) when it is created |
| Task Name | The task name |
| JDBC Connection Options | A grid variable containing any required JDBC connection options used for connecting to the Source database (only used during Initial Loads to get the table metadata) |
| Debug Mode | Set to “TRUE” to print debug information to the Task History during a run |
| Last Updated | Set to either YES or NO, to add an optional “MAT_LAST_UPDATED” timestamp to the target table. If YES it will add a new column to the table called mat_last_update_stamp. For initial loads will be NULL, but any rows updated will get an update timestamp |
Downloads
Licensed under: Matillion Free Subscription License
Download METL-aws-sf-1.50.9-CDC-sync-to-target.melt
- Platform: AWS
- Target: Snowflake
- Version: 1.50.9 or higher
Installation Instructions
How to Install a Matillion ETL Shared JobAuthor: Matillion
Date Posted: Oct 29, 2023
Last Modified: Oct 29, 2023