Maia Incremental Load

Incrementally load data from Salesforce. Migrated from the Matillion ETL incremental load shared job.

On Matillion ETL, a shared job was available for Salesforce Incremental Load. To simplify migration to Maia Foundation, this download provides a drop-in replacement, to be published as a shared pipeline.

Once set up, this shared pipeline can be consumed from any project in your account — just like the original Matillion ETL shared job.

Setup Instructions

1. Create a new project

We recommend a dedicated project to keep the shared pipeline source separate from your working pipelines. Use the name:

Maia Incremental Load - Snowflake

This name forms part of the shared pipeline identifier that consumers will see (see Usage below), so choosing a consistent name avoids confusion.

2. Import from the Matillion Exchange

1. Open a branch on your project.
2. Click “Add > Browse Exchange”

3. Search for “Maia Incremental Load” and select the pipeline to import it into your project.
4. You should now have a folder named “Imported from Exchange > Incremental load” containing the latest version of this pipeline.

3. Create the .matillion folder

If your project doesn’t already have a .matillion folder in the root, create one.

4. Add the shared pipelines configuration

Copy the file:

Imported from Exchange/Incremental load/shared-pipelines.yaml

into the .matillion folder so it exists at:

.matillion/shared-pipelines.yaml

Already have a .matillion/shared-pipelines.yaml?

Instead of overwriting it, add the following pipeline entry to your existing file’s pipelines: list.

5. Commit your changes

Commit all changes to the project branch.

6. Push with shared pipelines enabled

Push the branch and ensure both of the following options are ticked:

• ✅ Publish
• ✅ Include shared pipelines

This publishes the shared pipeline to your account.

Migrating METL Pipelines That Use This Shared Job

When importing METL pipelines that contain a reference to the Salesforce Incremental Load shared job, the import process needs a JSON mapping file to translate the shared job parameters into the shared pipeline format.

Use the mapping file at:

Imported from Exchange/Incremental load/Salesforce/shared-job-mapping.json

Provide this file when prompted during the METL pipeline import.

Note: The mapping file contains the project name (Maia Incremental Load - Snowflake) in the pipelineName field. If you used a different project name in step 1, update the pipelineName value in the JSON to match:

"pipelineName": "Your Project Name#Salesforce-Incremental-Load"

Known Limitations

⚠️ Encryption parameters — The Encryption and KMS Key ID parameters exist for backwards compatibility with migrating pipelines, but are no longer implemented. Encryption types of None and SSE S3 will continue to work automatically, but SSE KMS is not currently supported on Maia Foundation.

⚠️ MATILLION_BATCH_ID — The MATILLION_BATCH_ID column on target tables will be hard-coded to zero on all rows updated by the shared pipelines.

Usage

Once published, the shared pipeline is available from any project in the same account. Add a Run Shared Pipeline component to your orchestration pipelines and select:

Maia Incremental Load - Snowflake#Salesforce-Incremental-Load

The identifier is formed as <project name>#<id>, where the id is defined in shared-pipelines.yaml. If you used a different project name in step 1, the prefix will reflect that instead.

Configure the pipeline parameters (OAuth credential, target database/schema, staging settings, source list, etc.) as documented in the shared pipeline’s parameter descriptions.

Downloads

Licensed under: Matillion Free Subscription License

• Download maia_incremental_load_snowflake_20260528.zip
  • Target: Snowflake

Installation Instructions