Author: Matillion
Date Posted: Dec 3, 2024
Last Modified: Dec 3, 2024
DPC API Pipeline Execution Monitoring
Monitor and track Data Productivity Cloud pipelines for auditing, logging and performance analysis.
This is a framework that enables you to record the timing and status of your Data Productivity Cloud pipelines. Please refer to our online Pipeline Execution and Status Retrieval Framework series for background information.
Two entry points are available:
1 - Execute Pipeline- Run this to launch a published pipeline2 - Refresh Pipeline Execution Status- Run this periodically to check all launched pipelines for their eventual status
Status will be recorded in a metadata table named PIPELINE_EXECUTION_LOG in a database and schema defined by parameters.
Prerequisites
This framework is for pipelines that have been published.
You will need to set up an OAuth for the Data Productivity Cloud provider.
Launch a pipeline
Use 1 - Execute Pipeline to launch a published pipeline. The parameters are:
| Parameter | Value |
|---|---|
| prv_metaDB | Name of the database to store metadata |
| prv_metaSchema | Name of the schema to store metadata |
| piv_projectId | Your DPC Project ID - you can find in the URL of your browser |
| piv_execidtablename | Your choice of name for a temporary staging table |
| piv_pipelineName | Full path of the pipeline to run |
| piv_environmentName | Name of the Environment to use |
| piv_server | Set to eu1 or us1 according to your account location. More information here |
| piv_credentials_name | Name of your OAuth for the Data Productivity Cloud provider |
Refresh Pipeline Execution Status
Run 2 - Refresh Pipeline Execution Status periodically. It will check all launched pipelines and update their eventual status into the metadata table PIPELINE_EXECUTION_LOG.
| Parameter | Value |
|---|---|
| prv_metaDB | Name of the database to store metadata |
| prv_metaSchema | Name of the schema to store metadata |
| piv_projectId | Your DPC Project ID - you can find in the URL of your browser |
| piv_execidtablename | Your choice of name for a temporary staging table |
| piv_server | Set to eu1 or us1 according to your account location. More information here |
| piv_credentials_name | Name of your OAuth for the Data Productivity Cloud provider |
Downloads
Licensed under: Matillion Free Subscription License
- Download DPC-API-pipeline-execution-status-Snowflake.zip
- Target: Snowflake