{"objects":[{"metadata":{"identifier":{"packageName":"Matillion Exchange","name":"Bing Search Incremental Load","revision":1,"type":"DYNAMIC"},"rootJobReference":{"name":"Bing Search - 1 - Orchestrator","type":"ORCHESTRATION","parameterMetadata":[{"slot":2,"variableName":"api_key","variableType":"SCALAR","displayName":"API Key","description":"The name in the Matillion Password Manager for the api key.","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":true},{"slot":3,"variableName":"search_terms","variableType":"SCALAR","displayName":"Search Terms","description":"The search expression.","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":true},{"slot":4,"variableName":"safe_search","variableType":"SCALAR","displayName":"Safe Search","description":"The search safety level.\nThe allowed values are Strict, Moderate, Off.","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":false},{"slot":5,"variableName":"market","variableType":"SCALAR","displayName":"Market","description":"The market where the results come from.","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":false},{"slot":6,"variableName":"truncate_tables","variableType":"SCALAR","displayName":"Truncate Tables","description":"Truncate the target tables before every job execution.  The allowed values are Yes or No.","defaultValue":[{"values":{"1":"Yes"}}],"defaultValueType":"TEXT","required":true},{"slot":7,"variableName":"advanced_connection_options","variableType":"GRID","displayName":"Connection Options","description":"A JDBC parameter supported by the Database Driver. The available parameters are determined automatically from the driver, and may change from version to version.\nThey are usually not required as sensible defaults are assumed.","defaultValue":[{"values":{"1":""}}],"defaultValueType":null,"required":false},{"slot":8,"variableName":"source_list","variableType":"GRID","displayName":"Tables and Columns","description":"Contains the list of tables and columns to be processed. The incremental_column should be set to 1 to indicate which column the incremental load is to be based on, 0 otherwise. There should be only 1 incremental column per table","defaultValue":[{"values":{"1":""}}],"defaultValueType":null,"required":true},{"slot":9,"variableName":"load_concurrent","variableType":"SCALAR","displayName":"Load Type","description":"Sequential - Iterations are done in sequence, waiting for each to complete before starting the next. \nThis is the default.\n<br>\nConcurrent - Iterations are run concurrently. This requires all \"Variables to Iterate\" to be defined as\n<a href=\"https://redshiftsupport.matillion.com/customer/portal/articles/2037630?b_id=8915#header1\" target=\"_blank\"><i>Copied</i> variables</a>, so that each iteration gets its own copy of the variable isolated from the same\nvariable being used by other concurrent executions.\n<span class=\"env rs sf\"><br><b>Note:</b> The maximum concurrency is limited by the number of available threads (2x the number of processors on your cloud instance).</span>","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":true},{"slot":10,"variableName":"stage_warehouse","variableType":"SCALAR","displayName":"Stage Warehouse","description":"The warehouse name where the staging data will be stored.","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":true},{"slot":11,"variableName":"stage_database","variableType":"SCALAR","displayName":"Stage Database","description":"The database name where the staging data will be stored.","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":true},{"slot":12,"variableName":"stage_schema","variableType":"SCALAR","displayName":"Stage Schema","description":"The schema name where the staging data will be stored.","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":true},{"slot":13,"variableName":"stage_prefix","variableType":"SCALAR","displayName":"Stage Prefix","description":"A prefix value that will be added to the start of the stage table names.\n<br>\ne.g. If a Stage Prefix of 'stage_' is specified and the table being processed is named 'test_data' then the target table will be named 'stage_test_data'.","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":true},{"slot":14,"variableName":"target_warehouse","variableType":"SCALAR","displayName":"Target Warehouse","description":"The warehouse name where the target data will be stored.","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":true},{"slot":15,"variableName":"target_database","variableType":"SCALAR","displayName":"Target Database","description":"The database name where the target data will be stored.","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":true},{"slot":16,"variableName":"target_schema","variableType":"SCALAR","displayName":"Target Schema","description":"The schema name where the target data will be stored.","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":true},{"slot":17,"variableName":"target_prefix","variableType":"SCALAR","displayName":"Target Prefix","description":"A prefix value that will be added to the start of the target table names.\n<br>\ne.g. If a Target Prefix of 'target_' is specified and the table being processed is named 'test_data' then the target table will be named 'target_test_data'.","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":false},{"slot":18,"variableName":"staging_type","variableType":"SCALAR","displayName":"Staging","description":"<b>Snowflake Managed:</b> Allow Matillion ETL to create and use a temporary internal stage on Snowflake for staging the data. This stage, along with the staged data, will cease to exist after loading is complete.<br>\n                <b>Existing Amazon S3 Location:</b> Selecting this will avail the user of properties to specify a custom staging area on S3.\n                <b>Existing Azure Blob Storage Location:</b> Selecting this will avail the user of properties to specify a custom staging area on Azure.","defaultValue":[{"values":{"1":"Snowflake Managed"}}],"defaultValueType":"TEXT","required":false},{"slot":19,"variableName":"s3_bucket_name","variableType":"SCALAR","displayName":"S3 Bucket Name","description":"<span class=\"env sf\">(AWS Only) </span>The name of an S3 bucket for temporary storage. Ensure your access credentials have S3 access and permission to write to the bucket. See <a href=\"https://redshiftsupport.matillion.com/customer/portal/articles/2054760\" target=\"_blank\">this document</a> for details on setting up access. The temporary objects created in this bucket will be removed again after the load completes, they are not kept.<br>\n<span class=\"env sf\">This property is available when using an Existing Amazon S3 Location for Staging.</span>","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":false},{"slot":20,"variableName":"encryption_method","variableType":"SCALAR","displayName":"Encryption","description":"<span class=\"env sf\">(AWS Only) </span>Decide on how the files are encrypted inside the S3 Bucket.<span class=\"env sf\">This property is available when using an Existing Amazon S3 Location for Staging.</span><br>\n                <b>None:</b> No encryption.<br>\n                <b>SSE KMS:</b> Encrypt the data according to a key stored on <a href=\"https://aws.amazon.com/kms/\" target=\"_blank\">KMS</a>.<br>\n                <b>SSE S3:</b> Encrypt the data according to a key stored on an <a href=\"http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html\" target=\"_blank\">S3 bucket</a>","defaultValue":[{"values":{"1":"None"}}],"defaultValueType":"TEXT","required":false},{"slot":21,"variableName":"kms_key_id","variableType":"SCALAR","displayName":"KMS Key ID","description":"The ID of the KMS encryption key you have chosen to use in the 'Encryption' property.","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":false},{"slot":22,"variableName":"storage_account","variableType":"SCALAR","displayName":"Azure Storage Account","description":"(Azure Only) Azure Storage Account to allow Matillion ETL to specify a blob container object on Snowflake for staging data. ","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":false},{"slot":23,"variableName":"blob_container","variableType":"SCALAR","displayName":"Azure Blob Container","description":"(Azure Only) Azure Blob Container to allow Matillion ETL to use the blob storage container object on Snowflake for staging data. ","defaultValue":[{"values":{"1":""}}],"defaultValueType":"TEXT","required":false},{"slot":24,"variableName":"log_metrics","variableType":"SCALAR","displayName":"Log Metrics","description":"Option to switch off Metrics logging in the full product - default is TRUE which records the metrics for the batch run. ","defaultValue":[{"values":{"1":"TRUE"}}],"defaultValueType":"TEXT","required":false}]},"orchestrationJobs":["Bing Search - 1 - Orchestrator","Bing Search - 4 - Query Source","Bing Search - 2 - Wrapper","Bing Search - 3 - Stage Object"],"transformationJobs":[],"imageMetadata":{"componentIcon":"iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAARGVYSWZNTQAqAAAACAABh2kABAAAAAEAAAAaAAAAAAADoAEAAwAAAAEAAQAAoAIABAAAAAEAAAmSoAMABAAAAAEAAAmSAAAAAE7PgAoAAASnSURBVHic7ZZdbBRVFMf/597ZWXaXUrchlEathg+R1ieNiQmaEE00NoEHZFcRDBG1jSFCNillZ6nenWjd1ZRoDRBLVRJ5kMyKiUSIEsWPRPHBB0PcFQSDidrQ0A9Yttt0Pu71YXdLaYuVpkYf+nuazL3nnP+ce865A8wxxxxz/K9QguFLof3XMm4cBZqpaclQKQIAyvS8rrjWB5+zD2tbRsbcXzewYCBTzjT4tQKIFFn7s6gJN6jBS8uB6vPIZgmm6U5pd+yYjqamUQDA4e469KMfvb0eGhtLPqNRDwBgWRwAkM0qmJPFThDQ/R18/nuVY9+BaPP5sbVkkiOZ9ECkcOCNmygY3INgaJBGhvslwmmGyxulph/Bus0DN5qBawtOkQYlNUgWgtX9BMB+ANE5AC5Ms7QnFAoDGFJrNm2Dtf8QtKFbpUtfwQ0U8VHPSrhyCUABRJs/BABu7V8PqchjdBbZP09NzAK7Nh9KQeMAoz4CPU7hBWfJ6vkGVncbLCsAAHCpCGAJHT2YhsIlrGv5lUBxuPkwuWo3AA7gQWTeeYh90P2Ux2iZ5+NFAvZMijfVi1KQok8NqSiuDB9FVegB0vXX4BUWlkTaOpTqU9DeU5CXkelZraB+R4DrIDqDaMsRSJXRpHunnKctjZzpPakee/kLBpxcVlMTnE6AhCclGK9BGM8BaAAjwLZ/gT1cLJtwEIKwMUKgGrhSB9R8OK4GqPlQgnFgoTtPLzT2XRq+3VWfrura+ogX0GvPbd+eF0KwvxFAC8CIEWnHaXHtXkAtQr5gKgersPmFUoHxYh+BfiLdfUaBTmBDy3HO+PdwnAGmcMzKNBJnMnvfHwOBjb/1P+8fdfz3DF45EP259+Fdu3bdb5qmVOW2BypdIASDaSpY3ZtI17vheTrAdqtR2YUnn+2d8pim4c3mbfV9tQtO+IgtPT1Pu1I/Yg+FNb2+aI++1dHRsT0SifBMJiOvTrByK+LQvrugcRfrW06XxWljLVih0ttAqd+FYEgmFZJJEsCiHDCYMU27tbV1sT8Y+PzwLTVdp1eveL/93a8jwerqg/l8/pV0Ov2iNd7PJMeWxTEuVdNRTisZhpEzDCNWSXNra+titXZLVcVRW1tbazKZVIlE4mZg4hyofE3l+R8ihGBEJNvb25dzzlc6jnOOiFRzc7Ovs7PzQmd5Ty6XI8dx9jqO0yGlXAPg7ck33xTjcjpyuRwBgOd59QAkEf0IAHV1dV6l6s2rfkd27tx5kYiWABMzMEMaGhoq9XGRMcZc162NRCK9jY2NiJYzWS46LxaLBTjnYSnlhVkTYJqmAkC6ruds2x4EsDWTyTwdDod9Qgg/AGaaZkEIwWzbvtvv9weHh4c/m43YY1QqOh6Pb06lUiqRSGyprMXj8U927NixAgBisVggHo83oTwCZvwjMRVCCGaapjQM49WqqiqjUCh863nex0T0kqZpA57nrU2lUqfG20x9F8wQ0zSlEIKlUqlEPp9/lHPucM7biYiHQqHbADSVheqzGXcS4+e9EEIzDKPfMIyuiWvALB/BeCKRCC93hz46OrohnU4fKMe7/i/ev8G4i2fKj/0LEEEATXYNHLYAAAAASUVORK5CYII=","smallIcon":"iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAARGVYSWZNTQAqAAAACAABh2kABAAAAAEAAAAaAAAAAAADoAEAAwAAAAEAAQAAoAIABAAAAAEAAAmSoAMABAAAAAEAAAmSAAAAAE7PgAoAAAG+SURBVHic3ZK9a1NhFMZ/581N7DWlase6BMUKtrhJxujiKEVIrXPF4JShRW6I+vK69D8oeHEUlXQRdBNsxQ8EURAc1KKDQsSqWAQ/knvzHoe2JgEli5NnPDznOYffeeA/KWsN1pp/b6rWADJYfC0ep7G4H7WG5eXgLyrZvFJQlW4ToBGfE5jU6dMzv+VL8QQwimZX+dL+TKWSbE1pnytA49KcwKSoPPSB+c7x2assxSdQ3YOaO6DDZDK7CMwLoBD8SNbqL5uPHbAJTkIQr0aKkstdIY5DkBYiXzGMGyNjgfHvdq9/m5568+FQuiPMO+d86TBmw8DIU/LhMSCrSesAzeZPkvQZ3t9je3pzqKW303Ll0cnV9+G+tj9/5snrs3Z+/uDdIy7tEr5xeYypU80/0XteLueuFwpH97bb9x+MjpRGEn27LU0dIrP9LLQnD6pSsjZAkbkoKkZRFPeaRlFUr9VqM1svU1QFEd/FK7qi2hFxOlTPfPLeD/csUxEJgY8DQ2KtNc45X6vVLgL5IAiSTqfzKpvN3gLWB8bXOedVVRYWFi6IyIqIFFV1wjm3BqSD5vsuAahWqzsBdCON8guaKq8LCLeRyAAAAABJRU5ErkJggg=="},"description":"","helpHtml":"<div class=\"mtln\"><img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAARGVYSWZNTQAqAAAACAABh2kABAAAAAEAAAAaAAAAAAADoAEAAwAAAAEAAQAAoAIABAAAAAEAAAmSoAMABAAAAAEAAAmSAAAAAE7PgAoAAASnSURBVHic7ZZdbBRVFMf/597ZWXaXUrchlEathg+R1ieNiQmaEE00NoEHZFcRDBG1jSFCNillZ6nenWjd1ZRoDRBLVRJ5kMyKiUSIEsWPRPHBB0PcFQSDidrQ0A9Yttt0Pu71YXdLaYuVpkYf+nuazL3nnP+ce865A8wxxxxz/K9QguFLof3XMm4cBZqpaclQKQIAyvS8rrjWB5+zD2tbRsbcXzewYCBTzjT4tQKIFFn7s6gJN6jBS8uB6vPIZgmm6U5pd+yYjqamUQDA4e469KMfvb0eGhtLPqNRDwBgWRwAkM0qmJPFThDQ/R18/nuVY9+BaPP5sbVkkiOZ9ECkcOCNmygY3INgaJBGhvslwmmGyxulph/Bus0DN5qBawtOkQYlNUgWgtX9BMB+ANE5AC5Ms7QnFAoDGFJrNm2Dtf8QtKFbpUtfwQ0U8VHPSrhyCUABRJs/BABu7V8PqchjdBbZP09NzAK7Nh9KQeMAoz4CPU7hBWfJ6vkGVncbLCsAAHCpCGAJHT2YhsIlrGv5lUBxuPkwuWo3AA7gQWTeeYh90P2Ux2iZ5+NFAvZMijfVi1KQok8NqSiuDB9FVegB0vXX4BUWlkTaOpTqU9DeU5CXkelZraB+R4DrIDqDaMsRSJXRpHunnKctjZzpPakee/kLBpxcVlMTnE6AhCclGK9BGM8BaAAjwLZ/gT1cLJtwEIKwMUKgGrhSB9R8OK4GqPlQgnFgoTtPLzT2XRq+3VWfrura+ogX0GvPbd+eF0KwvxFAC8CIEWnHaXHtXkAtQr5gKgersPmFUoHxYh+BfiLdfUaBTmBDy3HO+PdwnAGmcMzKNBJnMnvfHwOBjb/1P+8fdfz3DF45EP259+Fdu3bdb5qmVOW2BypdIASDaSpY3ZtI17vheTrAdqtR2YUnn+2d8pim4c3mbfV9tQtO+IgtPT1Pu1I/Yg+FNb2+aI++1dHRsT0SifBMJiOvTrByK+LQvrugcRfrW06XxWljLVih0ttAqd+FYEgmFZJJEsCiHDCYMU27tbV1sT8Y+PzwLTVdp1eveL/93a8jwerqg/l8/pV0Ov2iNd7PJMeWxTEuVdNRTisZhpEzDCNWSXNra+titXZLVcVRW1tbazKZVIlE4mZg4hyofE3l+R8ihGBEJNvb25dzzlc6jnOOiFRzc7Ovs7PzQmd5Ty6XI8dx9jqO0yGlXAPg7ck33xTjcjpyuRwBgOd59QAkEf0IAHV1dV6l6s2rfkd27tx5kYiWABMzMEMaGhoq9XGRMcZc162NRCK9jY2NiJYzWS46LxaLBTjnYSnlhVkTYJqmAkC6ruds2x4EsDWTyTwdDod9Qgg/AGaaZkEIwWzbvtvv9weHh4c/m43YY1QqOh6Pb06lUiqRSGyprMXj8U927NixAgBisVggHo83oTwCZvwjMRVCCGaapjQM49WqqiqjUCh863nex0T0kqZpA57nrU2lUqfG20x9F8wQ0zSlEIKlUqlEPp9/lHPucM7biYiHQqHbADSVheqzGXcS4+e9EEIzDKPfMIyuiWvALB/BeCKRCC93hz46OrohnU4fKMe7/i/ev8G4i2fKj/0LEEEATXYNHLYAAAAASUVORK5CYII=\"><h3>Bing Search Incremental Load</h3><p style='white-space: pre-line; word-wrap: break-word;'></p><h3>Properties</h3><table><tbody><tr><th>Property</th><th>Type</th><th>Description</th></tr><tr><td>API Key</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>The name in the Matillion Password Manager for the api key.</td></tr><tr><td>Search Terms</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>The search expression.</td></tr><tr><td>Safe Search</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>The search safety level.\nThe allowed values are Strict, Moderate, Off.</td></tr><tr><td>Market</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>The market where the results come from.</td></tr><tr><td>Truncate Tables</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>Truncate the target tables before every job execution.  The allowed values are Yes or No.</td></tr><tr><td>Connection Options</td><td>Grid</td><td style='white-space: pre-line; word-wrap: break-word;'>A JDBC parameter supported by the Database Driver. The available parameters are determined automatically from the driver, and may change from version to version.\nThey are usually not required as sensible defaults are assumed.</td></tr><tr><td>Tables and Columns</td><td>Grid</td><td style='white-space: pre-line; word-wrap: break-word;'>Contains the list of tables and columns to be processed. The incremental_column should be set to 1 to indicate which column the incremental load is to be based on, 0 otherwise. There should be only 1 incremental column per table</td></tr><tr><td>Load Type</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>Sequential - Iterations are done in sequence, waiting for each to complete before starting the next. \nThis is the default.\n<br>\nConcurrent - Iterations are run concurrently. This requires all &quot;Variables to Iterate&quot; to be defined as\n&lt;a href=&quot;https://redshiftsupport.matillion.com/customer/portal/articles/2037630?b_id=8915#header1&quot; target=&quot;_blank&quot;&gt;<i>Copied</i> variables&lt;/a&gt;, so that each iteration gets its own copy of the variable isolated from the same\nvariable being used by other concurrent executions.\n&lt;span class=&quot;env rs sf&quot;&gt;<br><b>Note:</b> The maximum concurrency is limited by the number of available threads (2x the number of processors on your cloud instance).</span></td></tr><tr><td>Stage Warehouse</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>The warehouse name where the staging data will be stored.</td></tr><tr><td>Stage Database</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>The database name where the staging data will be stored.</td></tr><tr><td>Stage Schema</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>The schema name where the staging data will be stored.</td></tr><tr><td>Stage Prefix</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>A prefix value that will be added to the start of the stage table names.\n<br>\ne.g. If a Stage Prefix of &#39;stage_&#39; is specified and the table being processed is named &#39;test_data&#39; then the target table will be named &#39;stage_test_data&#39;.</td></tr><tr><td>Target Warehouse</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>The warehouse name where the target data will be stored.</td></tr><tr><td>Target Database</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>The database name where the target data will be stored.</td></tr><tr><td>Target Schema</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>The schema name where the target data will be stored.</td></tr><tr><td>Target Prefix</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>A prefix value that will be added to the start of the target table names.\n<br>\ne.g. If a Target Prefix of &#39;target_&#39; is specified and the table being processed is named &#39;test_data&#39; then the target table will be named &#39;target_test_data&#39;.</td></tr><tr><td>Staging</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'><b>Snowflake Managed:</b> Allow Matillion ETL to create and use a temporary internal stage on Snowflake for staging the data. This stage, along with the staged data, will cease to exist after loading is complete.<br>\n                <b>Existing Amazon S3 Location:</b> Selecting this will avail the user of properties to specify a custom staging area on S3.\n                <b>Existing Azure Blob Storage Location:</b> Selecting this will avail the user of properties to specify a custom staging area on Azure.</td></tr><tr><td>S3 Bucket Name</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>&lt;span class=&quot;env sf&quot;&gt;(AWS Only) </span>The name of an S3 bucket for temporary storage. Ensure your access credentials have S3 access and permission to write to the bucket. See &lt;a href=&quot;https://redshiftsupport.matillion.com/customer/portal/articles/2054760&quot; target=&quot;_blank&quot;&gt;this document&lt;/a&gt; for details on setting up access. The temporary objects created in this bucket will be removed again after the load completes, they are not kept.<br>\n&lt;span class=&quot;env sf&quot;&gt;This property is available when using an Existing Amazon S3 Location for Staging.</span></td></tr><tr><td>Encryption</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>&lt;span class=&quot;env sf&quot;&gt;(AWS Only) </span>Decide on how the files are encrypted inside the S3 Bucket.&lt;span class=&quot;env sf&quot;&gt;This property is available when using an Existing Amazon S3 Location for Staging.</span><br>\n                <b>None:</b> No encryption.<br>\n                <b>SSE KMS:</b> Encrypt the data according to a key stored on &lt;a href=&quot;https://aws.amazon.com/kms/&quot; target=&quot;_blank&quot;&gt;KMS&lt;/a&gt;.<br>\n                <b>SSE S3:</b> Encrypt the data according to a key stored on an &lt;a href=&quot;http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html&quot; target=&quot;_blank&quot;&gt;S3 bucket&lt;/a&gt;</td></tr><tr><td>KMS Key ID</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>The ID of the KMS encryption key you have chosen to use in the &#39;Encryption&#39; property.</td></tr><tr><td>Azure Storage Account</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>(Azure Only) Azure Storage Account to allow Matillion ETL to specify a blob container object on Snowflake for staging data. </td></tr><tr><td>Azure Blob Container</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>(Azure Only) Azure Blob Container to allow Matillion ETL to use the blob storage container object on Snowflake for staging data. </td></tr><tr><td>Log Metrics</td><td>Variable</td><td style='white-space: pre-line; word-wrap: break-word;'>Option to switch off Metrics logging in the full product - default is TRUE which records the metrics for the batch run. </td></tr></tbody></table></div>","created":1620657545929},"orchestrationJobs":{"Bing Search - 4 - Query Source":{"id":-2,"revision":2,"created":1620657489293,"timestamp":1620657489293,"components":{"25552":{"id":25552,"inputCardinality":"ONE","outputCardinality":"ZERO","connectorHint":"UNCONDITIONAL","executionHint":"FLOW","implementationID":-1946388514,"x":320,"y":0,"width":32,"height":32,"inputConnectorIDs":[25558],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"End"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25553":{"id":25553,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":-1334993426,"x":160,"y":320,"width":32,"height":32,"inputConnectorIDs":[25560],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[25559],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Log Error"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Level","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"error"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"Namespace","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_table_name}"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Messages","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Unable to retrieve data from source"},"2":{"slot":2,"type":"STRING","value":"${error_message}"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25554":{"id":25554,"inputCardinality":"ONE","outputCardinality":"ZERO","connectorHint":"UNCONDITIONAL","executionHint":"FLOW","implementationID":515156205,"x":320,"y":320,"width":32,"height":32,"inputConnectorIDs":[25559],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"End Failure"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25555":{"id":25555,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":-1773186829,"x":166,"y":160,"width":32,"height":32,"inputConnectorIDs":[25562],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[25560],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Print Error"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Script","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"msg = \\\n\"\"\"There was an issue whilst retrieving data from the source. \nThe attempted query was: \\n{q}\nError: {e}\n\"\"\".format(q = sql, e = error_message) \nprint(msg)\n"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"Interpreter","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Python 3"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Timeout","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"INTEGER","value":"360"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"User","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Restricted"}}}},"visible":false,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25556":{"id":25556,"inputCardinality":"ZERO","outputCardinality":"MANY","connectorHint":"UNCONDITIONAL","executionHint":"FLOW","implementationID":444132438,"x":0,"y":0,"width":32,"height":32,"inputConnectorIDs":[],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[25561],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Start","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Start"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25557":{"id":25557,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":944477051,"x":152,"y":-3,"width":32,"height":32,"inputConnectorIDs":[25561],"outputSuccessConnectorIDs":[25558],"outputFailureConnectorIDs":[25562],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{"1":{"slot":1,"fromId":null,"fromName":"Row Count","mapTo":"static_row_count","validationStatus":null,"validationMessage":null},"2":{"slot":2,"fromId":null,"fromName":"Message","mapTo":"error_message","validationStatus":null,"validationMessage":null}},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Query Source"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Basic/Advanced Mode","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Advanced"}}}},"visible":true,"status":{},"columnNames":[]},"6":{"slot":6,"name":"Data Source","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":false,"status":{},"columnNames":[]},"7":{"slot":7,"name":"Data Selection","elements":{},"visible":false,"status":{},"columnNames":[]},"8":{"slot":8,"name":"Data Source Filter","elements":{},"visible":false,"status":{},"columnNames":[]},"9":{"slot":9,"name":"Combine Filters","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"And"}}}},"visible":false,"status":{},"columnNames":[]},"10":{"slot":10,"name":"SQL Query","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${sql}"}}}},"visible":true,"status":{},"columnNames":[]},"11":{"slot":11,"name":"Limit","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"INTEGER","value":"100"}}}},"visible":false,"status":{},"columnNames":[]},"12":{"slot":12,"name":"Connection Options","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"GRID","value":"advanced_connection_options"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"GRID","value":"parameter"},"2":{"slot":2,"type":"GRID","value":"value"}}}},"visible":true,"status":{},"columnNames":[]},"13":{"slot":13,"name":"Database","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${database}"}}}},"visible":true,"status":{},"columnNames":[]},"14":{"slot":14,"name":"Schema","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${schema}"}}}},"visible":true,"status":{},"columnNames":[]},"15":{"slot":15,"name":"Target Table","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${table}"}}}},"visible":true,"status":{},"columnNames":[]},"16":{"slot":16,"name":"Warehouse","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${warehouse}"}}}},"visible":true,"status":{},"columnNames":[]},"17":{"slot":17,"name":"S3 Staging Area","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${s3_bucket_name}"}}}},"visible":false,"status":{},"columnNames":[]},"18":{"slot":18,"name":"Primary Keys","elements":{},"visible":true,"status":{},"columnNames":[]},"20":{"slot":20,"name":"Data Schema","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Bing"}}}},"visible":false,"status":{},"columnNames":[]},"101":{"slot":101,"name":"API Key","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"PASSWORD","value":"${api_key}"}}}},"visible":true,"status":{},"columnNames":[]},"1001":{"slot":1001,"name":"","elements":{},"visible":false,"status":{},"columnNames":[]},"1013":{"slot":1013,"name":"Auto Debug","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${auto_debug}"}}}},"visible":true,"status":{},"columnNames":[]},"1014":{"slot":1014,"name":"Debug Level","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${auto_debug_level}"}}}},"visible":false,"status":{},"columnNames":[]},"40000":{"slot":40000,"name":"Encryption","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${encryption_method}"}}}},"visible":true,"status":{},"columnNames":[]},"40001":{"slot":40001,"name":"KMS Key ID","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${kms_key_id}"}}}},"visible":true,"status":{},"columnNames":[]},"40501":{"slot":40501,"name":"","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":false,"status":{},"columnNames":[]},"40502":{"slot":40502,"name":"Load Options","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"On"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"STRING","value":"Off"}}},"3":{"slot":3,"values":{"1":{"slot":1,"type":"STRING","value":"On"}}},"5":{"slot":5,"values":{"1":{"slot":1,"type":"STRING","value":"On"}}}},"visible":true,"status":{},"columnNames":[]},"63320":{"slot":63320,"name":"Storage Account","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${storage_account}"}}}},"visible":false,"status":{},"columnNames":[]},"63321":{"slot":63321,"name":"Blob Container","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${blob_container}"}}}},"visible":false,"status":{},"columnNames":[]},"88340":{"slot":88340,"name":"","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":false,"status":{},"columnNames":[]},"88341":{"slot":88341,"name":"Staging","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${staging_type}"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]}},"successConnectors":{"25558":{"id":25558,"sourceID":25557,"targetID":25552}},"failureConnectors":{"25562":{"id":25562,"sourceID":25557,"targetID":25555}},"unconditionalConnectors":{"25559":{"id":25559,"sourceID":25553,"targetID":25554},"25560":{"id":25560,"sourceID":25555,"targetID":25553},"25561":{"id":25561,"sourceID":25556,"targetID":25557}},"trueConnectors":{},"falseConnectors":{},"iterationConnectors":{},"noteConnectors":{},"canUndo":false,"undoCommand":"","undoCreated":-1,"canRedo":false,"redoCommand":"","redoCreated":-1,"notes":{"25551":{"id":25551,"x":53,"y":-171,"width":212,"height":549,"text":"**Query Source**\n\n+ Execute supplied SQL against source\n+ If an error occurs,\n- Print the error (so it appears in the Task History)\n- Log the error (so it appears in the Metrics store)\n- End the job with failure","colour":"e6e63c"}},"variables":{"sql":{"definition":{"name":"sql","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"SELECT 1"},"table":{"definition":{"name":"table","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"schema":{"definition":{"name":"schema","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"[Environment Default]"},"api_key":{"definition":{"name":"api_key","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"dataset":{"definition":{"name":"dataset","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"[Environment Default]"},"project":{"definition":{"name":"project","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"[Environment Default]"},"database":{"definition":{"name":"database","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"[Environment Default]"},"warehouse":{"definition":{"name":"warehouse","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"[Environment Default]"},"auto_debug":{"definition":{"name":"auto_debug","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"Off"},"dist_style":{"definition":{"name":"dist_style","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"Even"},"kms_key_id":{"definition":{"name":"kms_key_id","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"staging_type":{"definition":{"name":"staging_type","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"Snowflake Managed"},"error_message":{"definition":{"name":"error_message","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PUBLIC"},"value":""},"blob_container":{"definition":{"name":"blob_container","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"s3_bucket_name":{"definition":{"name":"s3_bucket_name","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"storage_account":{"definition":{"name":"storage_account","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"auto_debug_level":{"definition":{"name":"auto_debug_level","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"3"},"static_row_count":{"definition":{"name":"static_row_count","type":"DECIMAL","scope":"TASKBATCH","description":"","visibility":"PUBLIC"},"value":""},"encryption_method":{"definition":{"name":"encryption_method","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"None"},"target_table_name":{"definition":{"name":"target_table_name","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"cloud_storage_area":{"definition":{"name":"cloud_storage_area","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""}},"grids":{"advanced_connection_options":{"definition":{"name":"advanced_connection_options","scope":"BRANCH","definitions":[{"name":"parameter","type":"TEXT"},{"name":"value","type":"TEXT"}],"description":"","visibility":"PUBLIC"},"values":[]}}},"Bing Search - 1 - Orchestrator":{"id":-2,"revision":2,"created":1620657489333,"timestamp":1620657489333,"components":{"25612":{"id":25612,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":-1773186960,"x":-240,"y":16,"width":32,"height":32,"inputConnectorIDs":[25627],"outputSuccessConnectorIDs":[25624],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Get unique table names"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Script","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"source_list = context.getGridVariable('internal_source_list')\n\n# get distinct list of table names from source_list\ndistinct_table_names = list(set([table for table, col, inc_col, key_col in source_list]))\n\n# convert to list of lists to store in grid variable\ndistinct_table_names = [[name] for name in sorted(distinct_table_names)]\ncontext.updateGridVariable(\"distinct_table_names\", distinct_table_names)\n\n# convert truncate_tables to lower case for consistency\ntruncate_tables = truncate_tables.lower()\ncontext.updateVariable('truncate_tables', truncate_tables)\n\nprint(f\"source_list: {source_list}\")\nprint(f\"distinct_table_names: {distinct_table_names}\")\nprint(f'truncate_tables: {truncate_tables}')\n"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"Interpreter","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Python 3"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Timeout","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"INTEGER","value":"360"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25613":{"id":25613,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":-1773186960,"x":-32,"y":16,"width":32,"height":32,"inputConnectorIDs":[25624],"outputSuccessConnectorIDs":[25623],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Update environment default"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Script","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"# Script to replace any [Environment Default] entries with the schema name\n\nif target_platform == 'Redshift':\n  \n    if stage_schema == '[Environment Default]':\n        context.updateVariable('stage_schema', environment_default_schema)\n  \n    if target_schema == '[Environment Default]':\n        context.updateVariable('target_schema', environment_default_schema)\n\nelif target_platform == 'Snowflake':\n    \n    if stage_schema == '[Environment Default]':\n        context.updateVariable('stage_schema', environment_default_schema)\n  \n    if target_schema == '[Environment Default]':\n        context.updateVariable('target_schema', environment_default_schema)\n\n    if stage_database == '[Environment Default]':\n        context.updateVariable('stage_database', environment_database)\n\n    if target_database == '[Environment Default]':\n        context.updateVariable('target_database', environment_database)\n  \nelif target_platform == 'BigQuery':\n  \n    if stage_dataset == '[Environment Default]':\n        context.updateVariable('stage_dataset', environment_default_schema)\n\n    if target_dataset == '[Environment Default]':\n        context.updateVariable('target_dataset', environment_default_schema)\n\n    if stage_project == '[Environment Default]':\n        context.updateVariable('stage_project', environment_database)\n\n    if target_project == '[Environment Default]':\n        context.updateVariable('target_project', environment_database)\n"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"Interpreter","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Python 3"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Timeout","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"INTEGER","value":"360"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25614":{"id":25614,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"TRANSFORM","implementationID":-1180878087,"x":176,"y":48,"width":32,"height":32,"inputConnectorIDs":[],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"2 - Wrapper"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Job Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Bing Search - 2 - Wrapper"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"","elements":{},"visible":false,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Set Scalar Variables","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"s3_bucket_name"},"2":{"slot":2,"type":"STRING","value":"${s3_bucket_name}"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"STRING","value":"stage_prefix"},"2":{"slot":2,"type":"STRING","value":"${stage_prefix}"}}},"3":{"slot":3,"values":{"1":{"slot":1,"type":"STRING","value":"stage_schema"},"2":{"slot":2,"type":"STRING","value":"${stage_schema}"}}},"4":{"slot":4,"values":{"1":{"slot":1,"type":"STRING","value":"target_prefix"},"2":{"slot":2,"type":"STRING","value":"${target_prefix}"}}},"5":{"slot":5,"values":{"1":{"slot":1,"type":"STRING","value":"target_schema"},"2":{"slot":2,"type":"STRING","value":"${target_schema}"}}},"6":{"slot":6,"values":{"1":{"slot":1,"type":"STRING","value":"table_name"},"2":{"slot":2,"type":"STRING","value":"${table_name}"}}},"7":{"slot":7,"values":{"1":{"slot":1,"type":"STRING","value":"dist_style"},"2":{"slot":2,"type":"STRING","value":"${dist_style}"}}},"8":{"slot":8,"values":{"1":{"slot":1,"type":"STRING","value":"target_platform"},"2":{"slot":2,"type":"STRING","value":"${target_platform}"}}},"9":{"slot":9,"values":{"1":{"slot":1,"type":"STRING","value":"target_database"},"2":{"slot":2,"type":"STRING","value":"${target_database}"}}},"10":{"slot":10,"values":{"1":{"slot":1,"type":"STRING","value":"target_warehouse"},"2":{"slot":2,"type":"STRING","value":"${target_warehouse}"}}},"11":{"slot":11,"values":{"1":{"slot":1,"type":"STRING","value":"stage_database"},"2":{"slot":2,"type":"STRING","value":"${stage_database}"}}},"12":{"slot":12,"values":{"1":{"slot":1,"type":"STRING","value":"stage_warehouse"},"2":{"slot":2,"type":"STRING","value":"${stage_warehouse}"}}},"13":{"slot":13,"values":{"1":{"slot":1,"type":"STRING","value":"target_dataset"},"2":{"slot":2,"type":"STRING","value":"${target_dataset}"}}},"14":{"slot":14,"values":{"1":{"slot":1,"type":"STRING","value":"target_project"},"2":{"slot":2,"type":"STRING","value":"${target_project}"}}},"15":{"slot":15,"values":{"1":{"slot":1,"type":"STRING","value":"stage_dataset"},"2":{"slot":2,"type":"STRING","value":"${stage_dataset}"}}},"16":{"slot":16,"values":{"1":{"slot":1,"type":"STRING","value":"stage_project"},"2":{"slot":2,"type":"STRING","value":"${stage_project}"}}},"17":{"slot":17,"values":{"1":{"slot":1,"type":"STRING","value":"cloud_storage_area"},"2":{"slot":2,"type":"STRING","value":"${cloud_storage_area}"}}},"18":{"slot":18,"values":{"1":{"slot":1,"type":"STRING","value":"storage_account"},"2":{"slot":2,"type":"STRING","value":"${storage_account}"}}},"19":{"slot":19,"values":{"1":{"slot":1,"type":"STRING","value":"blob_container"},"2":{"slot":2,"type":"STRING","value":"${blob_container}"}}},"20":{"slot":20,"values":{"1":{"slot":1,"type":"STRING","value":"encryption_method"},"2":{"slot":2,"type":"STRING","value":"${encryption_method}"}}},"21":{"slot":21,"values":{"1":{"slot":1,"type":"STRING","value":"kms_key_id"},"2":{"slot":2,"type":"STRING","value":"${kms_key_id}"}}},"22":{"slot":22,"values":{"1":{"slot":1,"type":"STRING","value":"staging_type"},"2":{"slot":2,"type":"STRING","value":"${staging_type}"}}},"23":{"slot":23,"values":{"1":{"slot":1,"type":"STRING","value":"log_metrics"},"2":{"slot":2,"type":"STRING","value":"${log_metrics}"}}},"24":{"slot":24,"values":{"1":{"slot":1,"type":"STRING","value":"api_key"},"2":{"slot":2,"type":"STRING","value":"${api_key}"}}},"25":{"slot":25,"values":{"1":{"slot":1,"type":"STRING","value":"search_terms"},"2":{"slot":2,"type":"STRING","value":"${search_terms}"}}},"26":{"slot":26,"values":{"1":{"slot":1,"type":"STRING","value":"safe_search"},"2":{"slot":2,"type":"STRING","value":"${safe_search}"}}},"27":{"slot":27,"values":{"1":{"slot":1,"type":"STRING","value":"market"},"2":{"slot":2,"type":"STRING","value":"${market}"}}},"28":{"slot":28,"values":{"1":{"slot":1,"type":"STRING","value":"truncate_tables"},"2":{"slot":2,"type":"STRING","value":"${truncate_tables}"}}},"29":{"slot":29,"values":{"1":{"slot":1,"type":"STRING","value":"auto_debug"},"2":{"slot":2,"type":"STRING","value":"${auto_debug}"}}},"30":{"slot":30,"values":{"1":{"slot":1,"type":"STRING","value":"auto_debug_level"},"2":{"slot":2,"type":"STRING","value":"${auto_debug_level}"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"Set Grid Variables","elements":{"1":{"slot":1,"elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"GRID","value":"advanced_connection_options"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"GRID","value":"parameter"},"2":{"slot":2,"type":"GRID","value":"value"}}}},"values":{"1":{"slot":1,"type":"STRING","value":"advanced_connection_options"},"2":{"slot":2,"type":"STRING","value":"grid"}}},"2":{"slot":2,"elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"GRID","value":"internal_source_list"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"GRID","value":"table"},"2":{"slot":2,"type":"GRID","value":"column"},"3":{"slot":3,"type":"GRID","value":"incremental_column"},"4":{"slot":4,"type":"GRID","value":"key_column"}}}},"values":{"1":{"slot":1,"type":"STRING","value":"source_list"},"2":{"slot":2,"type":"STRING","value":"grid"}}}},"visible":true,"status":{},"columnNames":[]},"6":{"slot":6,"name":"Melt Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Bing Search Incremental Load"}}}},"visible":true,"status":{},"columnNames":[]},"7":{"slot":7,"name":"Melt Package","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Matillion.Incremental"}}}},"visible":true,"status":{},"columnNames":[]},"8":{"slot":8,"name":"Melt Revision","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"INTEGER","value":"1"}}}},"visible":true,"status":{},"columnNames":[]},"9":{"slot":9,"name":"Melt Load Type","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"DYNAMIC"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[25632]},"25615":{"id":25615,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"ITERATE","implementationID":-424773870,"x":176,"y":16,"width":32,"height":16,"inputConnectorIDs":[25623],"outputSuccessConnectorIDs":[25625,25626],"outputFailureConnectorIDs":[25631],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{"1":{"slot":1,"fromId":null,"fromName":"Duration","mapTo":"logged_duration","validationStatus":null,"validationMessage":null}},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Iterate over distinct table names"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"Grid Variable","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"distinct_table_names"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Grid Variable Mapping","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"table_name"},"2":{"slot":2,"type":"STRING","value":"table_name"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"Break on Failure","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"No"}}}},"visible":false,"status":{},"columnNames":[]},"6":{"slot":6,"name":"Concurrency","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${load_concurrent}"}}}},"visible":true,"status":{},"columnNames":[]},"999":{"slot":999,"name":"Record Values In Task History","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Yes"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[25632],"inputIterationConnectorIDs":[]},"25616":{"id":25616,"inputCardinality":"ZERO","outputCardinality":"MANY","connectorHint":"UNCONDITIONAL","executionHint":"FLOW","implementationID":444132438,"x":-704,"y":16,"width":32,"height":32,"inputConnectorIDs":[],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[25630],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Start"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25617":{"id":25617,"inputCardinality":"ONE","outputCardinality":"ZERO","connectorHint":"UNCONDITIONAL","executionHint":"FLOW","implementationID":-1946388514,"x":416,"y":16,"width":32,"height":32,"inputConnectorIDs":[25625],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"End Success"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25618":{"id":25618,"inputCardinality":"ONE","outputCardinality":"ZERO","connectorHint":"UNCONDITIONAL","executionHint":"FLOW","implementationID":515156205,"x":416,"y":96,"width":32,"height":32,"inputConnectorIDs":[25631],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"End Failure"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25619":{"id":25619,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":-2112089365,"x":592,"y":-64,"width":32,"height":32,"inputConnectorIDs":[25629],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Log duration"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Metrics","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"duration"},"2":{"slot":2,"type":"STRING","value":"Numeric"},"3":{"slot":3,"type":"STRING","value":"${logged_duration}"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"Namespace","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":" "}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25620":{"id":25620,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"CONDITIONAL","executionHint":"FLOW","implementationID":-1357378929,"x":416,"y":-64,"width":32,"height":32,"inputConnectorIDs":[25626],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[25629],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Log Metrics?"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Mode","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Simple"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"Condition","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"log_metrics"},"2":{"slot":2,"type":"STRING","value":"Is"},"3":{"slot":3,"type":"STRING","value":"Equal to"},"4":{"slot":4,"type":"STRING","value":"TRUE"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Combine Conditions","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"And"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"Condition","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":false,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25621":{"id":25621,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":-1773186960,"x":-416,"y":16,"width":32,"height":32,"inputConnectorIDs":[25628],"outputSuccessConnectorIDs":[25627],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Copy source list"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Script","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"# map external source list with two columns to internal source list with four columns\n# setting inc_columns and key_columns to zero - actual values set in stage 2\nsource_list = context.getGridVariable('source_list')\ninternal_source_list = [[x[0], x[1], '0', '0'] for x in source_list]\ncontext.updateGridVariable('internal_source_list', internal_source_list)\n"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"Interpreter","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Python 3"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Timeout","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"INTEGER","value":"360"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25622":{"id":25622,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":-1773186829,"x":-576,"y":16,"width":32,"height":32,"inputConnectorIDs":[25630],"outputSuccessConnectorIDs":[25628],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Magic Variables"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Script","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"# this may or may not contain variables which are not valid connection options\n# and are designed to be used elsewhere within the shared job, they need to be removed from the grid \n# to prevent downstream errors \nopts = context.getGridVariable('advanced_connection_options') or [] \n\n# create a new list to store the real connection options \nfiltered = [] \n\n# magic vars is a map: lower case param name => job variable name \nmagic_vars = {\n\t'autodebug': 'auto_debug', # passed into XX Query Components for CData Debugging \n    'autodebuglevel': 'auto_debug_level', # passed into XX Query Components (default = 3)\n}\n\nif opts: \n  for r in opts:\n    param = r[0]\n    val = r[1]\n    \n    p = param.lower()\n    # check if this is a magic variable \n    # update the underlying job variable if it is \n    if p in magic_vars:\n      context.updateVariable(magic_vars[p], val)\n    # otherwise, its a normal connection option \n    else:\n      filtered.append( [param, val] )\n    \n  context.updateGridVariable('advanced_connection_options', filtered)\n  \n  \n  "}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"Interpreter","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Python 3"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Timeout","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"INTEGER","value":"360"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"User","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Restricted"}}}},"visible":false,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]}},"successConnectors":{"25623":{"id":25623,"sourceID":25613,"targetID":25615},"25624":{"id":25624,"sourceID":25612,"targetID":25613},"25625":{"id":25625,"sourceID":25615,"targetID":25617},"25626":{"id":25626,"sourceID":25615,"targetID":25620},"25627":{"id":25627,"sourceID":25621,"targetID":25612},"25628":{"id":25628,"sourceID":25622,"targetID":25621}},"failureConnectors":{"25631":{"id":25631,"sourceID":25615,"targetID":25618}},"unconditionalConnectors":{"25630":{"id":25630,"sourceID":25616,"targetID":25622}},"trueConnectors":{"25629":{"id":25629,"sourceID":25620,"targetID":25619}},"falseConnectors":{},"iterationConnectors":{"25632":{"id":25632,"sourceID":25615,"targetID":25614}},"noteConnectors":{},"canUndo":false,"undoCommand":"","undoCreated":-1,"canRedo":false,"redoCommand":"","redoCreated":-1,"notes":{"25610":{"id":25610,"x":-1384,"y":-373,"width":626,"height":460,"text":"**Bing Search Incremental Load** \n\nRoot (Stage 1) Orchestration Job of the Matillion Incremental Load pattern. \nThis job is the entry point of the shared job and is responsible for validating and reformatting some of the supplied data.\nThe pattern aims to synchronize INSERTS and UPDATES in a particular source, to a table within the relevant Cloud Data Warehouse.\n\n**Pattern Overview**\n\nThere are 4 distinct stages, each stage is represented by an orchestration job using the naming convention:\n__**<Source Name> - <Stage Number> - <Stage Name>**__\n\n+ **Stage 1** -  performs input validation and prepares the variables for **Stage 3**\n\n+ **Stage 2** - wrapper for **Stage 3** to ensure that in the case of any errors, relevant staging tables are nicely cleaned up.\n\n+ **Stage 3** - Orchestration job containing the ETL logic for a single source table.\n\n+ **Stage 4** - wrapper orchestration job to Query the underlying source. This is abstracted away from the rest of the logic to make the pattern more reusable.\n\nMore detailed information and specifics relating to each connector can be found within each job.\n\n\n======================================================================================================================================================================================================","colour":"00ce4f"},"25611":{"id":25611,"x":-629,"y":-250,"width":937,"height":346,"text":"** Prepare User Input **\n\n- (__Magic Variables__) Script that extracts some special variables from connection options. These are used internally to allow us to use 'hidden' variables that will not affect the slots of the shared job. Currently, this is used to switch on CData logging.\n\n- (__Copy Source List__) Script that reformats the __source_list__ grid variable so make sure it has 4 columns.\n\n- (__Get Unique Table Names__) Script to extract a list of table names from the source. This list is then iterated over, and the same ETL logic is applied to each source. This information is stored in the __distinct_table_names__ grid variable.\n\n- (__Update [Environment Default]__) Script to replace any [Environment Default] values with their actual value.\n\n- (__2 - Wrapper__) Iterate over each of the __distinct_table_names__ and call the lower level ETL process for each table in the source.","colour":"e6e63c"}},"variables":{"market":{"definition":{"name":"market","type":"TEXT","scope":"BRANCH","description":"The market where the results come from","visibility":"PUBLIC"},"value":""},"api_key":{"definition":{"name":"api_key","type":"TEXT","scope":"BRANCH","description":"The name in the Matillion Password Manager for the api key.","visibility":"PUBLIC"},"value":""},"auto_debug":{"definition":{"name":"auto_debug","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":"Off"},"dist_style":{"definition":{"name":"dist_style","type":"TEXT","scope":"BRANCH","description":"<b>Even:</b> Distributes rows around the Redshift cluster evenly.<br>\n<b>All:</b> Copy rows to all nodes in the Redshift cluster.<br>\nTable distribution is critical to good performance. See the <a href=\"http://docs.aws.amazon.com/redshift/latest/dg/tutorial-tuning-tables-distribution.html\" target=\"_blank\">Amazon Redshift</a> documentation for more information.","visibility":"PRIVATE"},"value":""},"kms_key_id":{"definition":{"name":"kms_key_id","type":"TEXT","scope":"BRANCH","description":"The ID of the KMS encryption key you have chosen to use in the 'Encryption' property.","visibility":"PUBLIC"},"value":""},"table_name":{"definition":{"name":"table_name","type":"TEXT","scope":"BRANCH","description":"","visibility":"PRIVATE"},"value":""},"log_metrics":{"definition":{"name":"log_metrics","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"TRUE"},"safe_search":{"definition":{"name":"safe_search","type":"TEXT","scope":"BRANCH","description":"The search safety level.\nThe allowed values are Strict, Moderate, Off.","visibility":"PUBLIC"},"value":""},"search_terms":{"definition":{"name":"search_terms","type":"TEXT","scope":"BRANCH","description":"The search expression","visibility":"PUBLIC"},"value":""},"stage_prefix":{"definition":{"name":"stage_prefix","type":"TEXT","scope":"BRANCH","description":"A prefix value that will be added to the start of the stage table names.\n<br>\ne.g. If a Stage Prefix of 'stage_' is specified and the table being processed is named 'test_data' then the target table will be named 'stage_test_data'.","visibility":"PUBLIC"},"value":""},"stage_schema":{"definition":{"name":"stage_schema","type":"TEXT","scope":"BRANCH","description":"The schema name where the staging data will be stored.","visibility":"PUBLIC"},"value":""},"staging_type":{"definition":{"name":"staging_type","type":"TEXT","scope":"BRANCH","description":"(AWS Only) <b>Snowflake Managed:</b> Allow Matillion ETL to create and use a temporary internal stage on Snowflake for staging the data. This stage, along with the staged data, will cease to exist after loading is complete.<br>\n                <b>Existing Amazon S3 Location:</b> Selecting this will avail the user of properties to specify a custom staging area on S3.","visibility":"PUBLIC"},"value":"Snowflake Managed"},"stage_dataset":{"definition":{"name":"stage_dataset","type":"TEXT","scope":"BRANCH","description":"The dataset name where the staging data will be stored.","visibility":"PRIVATE"},"value":""},"stage_project":{"definition":{"name":"stage_project","type":"TEXT","scope":"BRANCH","description":"The project name where the staging data will be stored.","visibility":"PRIVATE"},"value":""},"target_prefix":{"definition":{"name":"target_prefix","type":"TEXT","scope":"BRANCH","description":"A prefix value that will be added to the start of the target table names.\n<br>\ne.g. If a Target Prefix of 'target_' is specified and the table being processed is named 'test_data' then the target table will be named 'target_test_data'.","visibility":"PUBLIC"},"value":""},"target_schema":{"definition":{"name":"target_schema","type":"TEXT","scope":"BRANCH","description":"The schema name where the target data will be stored.","visibility":"PUBLIC"},"value":""},"blob_container":{"definition":{"name":"blob_container","type":"TEXT","scope":"BRANCH","description":"(Azure Only) Azure Blob Container to allow Matillion ETL to use the blob storage container object on Snowflake for staging data. ","visibility":"PUBLIC"},"value":""},"s3_bucket_name":{"definition":{"name":"s3_bucket_name","type":"TEXT","scope":"BRANCH","description":"<span class=\"env sf\">(AWS Only) </span>The name of an S3 bucket for temporary storage. Ensure your access credentials have S3 access and permission to write to the bucket. See <a href=\"https://redshiftsupport.matillion.com/customer/portal/articles/2054760\" target=\"_blank\">this document</a> for details on setting up access. The temporary objects created in this bucket will be removed again after the load completes, they are not kept.<br>\n<span class=\"env sf\">This property is available when using an Existing Amazon S3 Location for Staging.</span>","visibility":"PUBLIC"},"value":""},"stage_database":{"definition":{"name":"stage_database","type":"TEXT","scope":"BRANCH","description":"The database name where the staging data will be stored.","visibility":"PUBLIC"},"value":""},"target_dataset":{"definition":{"name":"target_dataset","type":"TEXT","scope":"BRANCH","description":"The dataset name where the target data will be stored.","visibility":"PRIVATE"},"value":""},"target_project":{"definition":{"name":"target_project","type":"TEXT","scope":"BRANCH","description":"The project name where the target data will be stored.","visibility":"PRIVATE"},"value":""},"load_concurrent":{"definition":{"name":"load_concurrent","type":"TEXT","scope":"BRANCH","description":"Sequential - Iterations are done in sequence, waiting for each to complete before starting the next. \nThis is the default.\n<br>\nConcurrent - Iterations are run concurrently. This requires all \"Variables to Iterate\" to be defined as\n<a href=\"https://redshiftsupport.matillion.com/customer/portal/articles/2037630?b_id=8915#header1\" target=\"_blank\"><i>Copied</i> variables</a>, so that each iteration gets its own copy of the variable isolated from the same\nvariable being used by other concurrent executions.\n<span class=\"env rs sf\"><br><b>Note:</b> The maximum concurrency is limited by the number of available threads (2x the number of virtual cpus on your cloud instance).</span>","visibility":"PUBLIC"},"value":""},"logged_duration":{"definition":{"name":"logged_duration","type":"DECIMAL","scope":"BRANCH","description":"","visibility":"PRIVATE"},"value":""},"stage_warehouse":{"definition":{"name":"stage_warehouse","type":"TEXT","scope":"BRANCH","description":"The warehouse name where the staging data will be stored.","visibility":"PUBLIC"},"value":""},"storage_account":{"definition":{"name":"storage_account","type":"TEXT","scope":"BRANCH","description":"(Azure Only) Azure Storage Account to allow Matillion ETL to specify a blob container object on Snowflake for staging data. ","visibility":"PUBLIC"},"value":""},"target_database":{"definition":{"name":"target_database","type":"TEXT","scope":"BRANCH","description":"The database name where the target data will be stored.","visibility":"PUBLIC"},"value":""},"target_platform":{"definition":{"name":"target_platform","type":"TEXT","scope":"BRANCH","description":"","visibility":"PRIVATE"},"value":"Snowflake"},"truncate_tables":{"definition":{"name":"truncate_tables","type":"TEXT","scope":"BRANCH","description":"Truncate the target tables before every job execution.  The allowed values are \"Yes\" and \"No\".","visibility":"PUBLIC"},"value":"Yes"},"auto_debug_level":{"definition":{"name":"auto_debug_level","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":"3"},"target_warehouse":{"definition":{"name":"target_warehouse","type":"TEXT","scope":"BRANCH","description":"The warehouse name where the target data will be stored.","visibility":"PUBLIC"},"value":""},"encryption_method":{"definition":{"name":"encryption_method","type":"TEXT","scope":"BRANCH","description":"<span class=\"env sf\">(AWS Only) </span>Decide on how the files are encrypted inside the S3 Bucket.<span class=\"env sf\">This property is available when using an Existing Amazon S3 Location for Staging.</span><br>\n                <b>None:</b> No encryption.<br>\n                <b>SSE KMS:</b> Encrypt the data according to a key stored on <a href=\"https://aws.amazon.com/kms/\" target=\"_blank\">KMS</a>.<br>\n                <b>SSE S3:</b> Encrypt the data according to a key stored on an <a href=\"http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html\" target=\"_blank\">S3 bucket</a>","visibility":"PUBLIC"},"value":"None"},"cloud_storage_area":{"definition":{"name":"cloud_storage_area","type":"TEXT","scope":"BRANCH","description":"The URL and path of the target Google Storage bucket to be used for staging the queried data throughout the job.","visibility":"PRIVATE"},"value":""}},"grids":{"source_list":{"definition":{"name":"source_list","scope":"BRANCH","definitions":[{"name":"table","type":"TEXT"},{"name":"column","type":"TEXT"}],"description":"Contains the list of tables and columns to be processed. The incremental_column should be set to 1 to indicate which column the incremental load is to be based on, 0 otherwise. There should be only 1 incremental column per table","visibility":"PUBLIC"},"values":[]},"distinct_table_names":{"definition":{"name":"distinct_table_names","scope":"BRANCH","definitions":[{"name":"table_name","type":"TEXT"}],"description":"","visibility":"PRIVATE"},"values":[]},"internal_source_list":{"definition":{"name":"internal_source_list","scope":"BRANCH","definitions":[{"name":"table","type":"TEXT"},{"name":"column","type":"TEXT"},{"name":"incremental_column","type":"DECIMAL"},{"name":"key_column","type":"DECIMAL"}],"description":"","visibility":"PRIVATE"},"values":[]},"advanced_connection_options":{"definition":{"name":"advanced_connection_options","scope":"BRANCH","definitions":[{"name":"parameter","type":"TEXT"},{"name":"value","type":"TEXT"}],"description":"A list of values and parameters. Parameters and their allowed values are database/driver specific. Referring to the data model will provide insight of what you could provide here.\n<br>\nThey are usually not required as sensible defaults are assumed.","visibility":"PUBLIC"},"values":[]}}},"Bing Search - 3 - Stage Object":{"id":-2,"revision":2,"created":1620657489316,"timestamp":1620657489316,"components":{"25579":{"id":25579,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":332733136,"x":1680,"y":0,"width":32,"height":32,"inputConnectorIDs":[25604],"outputSuccessConnectorIDs":[25594],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{"1":{"slot":1,"fromId":null,"fromName":"load_strategy","mapTo":"load_strategy","validationStatus":null,"validationMessage":null},"2":{"slot":2,"fromId":null,"fromName":"inc_col_type","mapTo":"inc_col_type","validationStatus":null,"validationMessage":null},"3":{"slot":3,"fromId":null,"fromName":"stage_sql","mapTo":"stage_sql","validationStatus":null,"validationMessage":null}},"gridExportMappings":{"1":{"slot":1,"fromGrid":"stage_table_metadata","toGrid":"stage_table_metadata","columnMapToFrom":{"column_name":"column_name","database_type":"database_type","size":"size","type":"type","precision":"precision"},"validationStatus":null,"validationMessage":null}},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Decide Load Strategy"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${table_name}"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"source_column_list","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${source_column_list}"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"key_column_count","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${key_column_count}"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"incremental_column_count","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"1"}}}},"visible":true,"status":{},"columnNames":[]},"6":{"slot":6,"name":"incremental_column","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${incremental_column}"}}}},"visible":true,"status":{},"columnNames":[]},"7":{"slot":7,"name":"skip_inc_col_check","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"N"}}}},"visible":true,"status":{},"columnNames":[]},"8":{"slot":8,"name":"timestamp_precision","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${timestamp_precision}"}}}},"visible":true,"status":{},"columnNames":[]},"9":{"slot":9,"name":"safely_cast_datetime_to_date","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"N"}}}},"visible":true,"status":{},"columnNames":[]},"10":{"slot":10,"name":"encloseTableString","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"TRUE"}}}},"visible":true,"status":{},"columnNames":[]},"11":{"slot":11,"name":"source_enclosing_char_s","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"\""}}}},"visible":true,"status":{},"columnNames":[]},"12":{"slot":12,"name":"source_enclosing_char_e","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"\""}}}},"visible":true,"status":{},"columnNames":[]},"13":{"slot":13,"name":"target_platform","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_platform}"}}}},"visible":true,"status":{},"columnNames":[]},"14":{"slot":14,"name":"stage_database","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_database}"}}}},"visible":true,"status":{},"columnNames":[]},"15":{"slot":15,"name":"stage_schema","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_schema}"}}}},"visible":true,"status":{},"columnNames":[]},"16":{"slot":16,"name":"stage_project","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_project}"}}}},"visible":true,"status":{},"columnNames":[]},"17":{"slot":17,"name":"stage_dataset","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_dataset}"}}}},"visible":true,"status":{},"columnNames":[]},"18":{"slot":18,"name":"stage_table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_table_name}"}}}},"visible":true,"status":{},"columnNames":[]},"19":{"slot":19,"name":"target_database","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_database}"}}}},"visible":true,"status":{},"columnNames":[]},"20":{"slot":20,"name":"target_schema","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_schema}"}}}},"visible":true,"status":{},"columnNames":[]},"21":{"slot":21,"name":"target_project","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_project}"}}}},"visible":true,"status":{},"columnNames":[]},"22":{"slot":22,"name":"target_dataset","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_dataset}"}}}},"visible":true,"status":{},"columnNames":[]},"23":{"slot":23,"name":"target_table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_table_name}"}}}},"visible":true,"status":{},"columnNames":[]},"24":{"slot":24,"name":"fix_target_data_types","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Yes"}}}},"visible":true,"status":{},"columnNames":[]},"25":{"slot":25,"name":"load_strategy","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]},"26":{"slot":26,"name":"max_value","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]},"27":{"slot":27,"name":"stage_sql","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]},"28":{"slot":28,"name":"inc_col_type","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]},"29":{"slot":29,"name":"stage_table_metadata","elements":{},"visible":true,"status":{},"columnNames":[]},"30":{"slot":30,"name":"messages","elements":{},"visible":true,"status":{},"columnNames":[]},"31":{"slot":31,"name":"use_incremental_column","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${use_incremental_column}"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25580":{"id":25580,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":-1773186829,"x":1392,"y":0,"width":32,"height":32,"inputConnectorIDs":[25603],"outputSuccessConnectorIDs":[25604],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Link use_incremental_column"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Script","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"# link variable use_incremental_column to truncate_tables \n\n\nif truncate_tables.lower() == 'yes':\n  use_incremental_column = 'Y'\nelse:\n  use_incremental_column = 'N'\n  \ncontext.updateVariable('use_incremental_column', use_incremental_column)\nprint(f\"Updated Variable 'use_incremental_column' with {use_incremental_column}\")\n"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"Interpreter","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Python 3"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Timeout","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"INTEGER","value":"360"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"User","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Restricted"}}}},"visible":false,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25581":{"id":25581,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":1010676649,"x":-160,"y":0,"width":32,"height":32,"inputConnectorIDs":[25608],"outputSuccessConnectorIDs":[25595],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{"1":{"slot":1,"fromId":null,"fromName":"metadata_stage_table_name","mapTo":"metadata_stage_table_name","validationStatus":null,"validationMessage":null},"2":{"slot":2,"fromId":null,"fromName":"source_metadata_sql","mapTo":"source_metadata_sql","validationStatus":null,"validationMessage":null},"3":{"slot":3,"fromId":null,"fromName":"get_metadata_sql","mapTo":"get_metadata_sql","validationStatus":null,"validationMessage":null},"4":{"slot":4,"fromId":null,"fromName":"stage_table_name","mapTo":"stage_table_name","validationStatus":null,"validationMessage":null},"5":{"slot":5,"fromId":null,"fromName":"target_table_name","mapTo":"target_table_name","validationStatus":null,"validationMessage":null}},"gridExportMappings":{"1":{"slot":1,"fromGrid":"source_list","toGrid":"source_list","columnMapToFrom":{"column_name":"column","key_column":"key_column","incremental_column":"incremental_column","table_name":"table"},"validationStatus":null,"validationMessage":null}},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Build Strings"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${table_name}"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"stage_database","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_database}"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"stage_schema","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_schema}"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"stage_project","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_project}"}}}},"visible":true,"status":{},"columnNames":[]},"6":{"slot":6,"name":"stage_dataset","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_dataset}"}}}},"visible":true,"status":{},"columnNames":[]},"7":{"slot":7,"name":"stage_prefix","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_prefix}"}}}},"visible":true,"status":{},"columnNames":[]},"8":{"slot":8,"name":"target_prefix","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_prefix}"}}}},"visible":true,"status":{},"columnNames":[]},"9":{"slot":9,"name":"metadata_stage_table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]},"10":{"slot":10,"name":"stage_table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]},"11":{"slot":11,"name":"target_table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]},"12":{"slot":12,"name":"source_metadata_sql","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]},"13":{"slot":13,"name":"get_metadata_sql","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]},"14":{"slot":14,"name":"source_list","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"GRID","value":"source_list"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"GRID","value":"table_name"},"2":{"slot":2,"type":"GRID","value":"column_name"},"3":{"slot":3,"type":"GRID","value":"incremental_column"},"4":{"slot":4,"type":"GRID","value":"key_column"}}}},"visible":true,"status":{},"columnNames":[]},"15":{"slot":15,"name":"random_suffix","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25582":{"id":25582,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":-1096987946,"x":160,"y":0,"width":32,"height":32,"inputConnectorIDs":[25598],"outputSuccessConnectorIDs":[25599],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"gridExportMappings":{"1":{"slot":1,"fromGrid":"source_table_metadata","toGrid":"source_table_metadata","columnMapToFrom":{"is_key":"is_key","nullable":"is_nullable","length":"length","column_name":"column_name","data_type_name":"data_type_name","numeric_scale":"numeric_scale","ordinal":"ordinal"},"validationStatus":null,"validationMessage":null}},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Check Source Metadata"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"metadata_stage_table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${metadata_stage_table_name}"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"target_table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_table_name}"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"get_metadata_sql","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${get_metadata_sql}"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"stage_database","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_database}"}}}},"visible":true,"status":{},"columnNames":[]},"6":{"slot":6,"name":"stage_schema","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_schema}"}}}},"visible":true,"status":{},"columnNames":[]},"7":{"slot":7,"name":"stage_project","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_project}"}}}},"visible":true,"status":{},"columnNames":[]},"8":{"slot":8,"name":"stage_dataset","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_dataset}"}}}},"visible":true,"status":{},"columnNames":[]},"9":{"slot":9,"name":"source_table_metadata","elements":{},"visible":true,"status":{},"columnNames":[]},"10":{"slot":10,"name":"error_message","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25583":{"id":25583,"inputCardinality":"ZERO","outputCardinality":"MANY","connectorHint":"UNCONDITIONAL","executionHint":"FLOW","implementationID":444132438,"x":-320,"y":0,"width":32,"height":32,"inputConnectorIDs":[],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[25608],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Start","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Start"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25584":{"id":25584,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"TRANSFORM","implementationID":-1180878087,"x":0,"y":0,"width":32,"height":32,"inputConnectorIDs":[25595],"outputSuccessConnectorIDs":[25598],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Get Source Metadata"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Job Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Bing Search - 4 - Query Source"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"","elements":{},"visible":false,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Set Scalar Variables","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"sql"},"2":{"slot":2,"type":"STRING","value":"${source_metadata_sql}"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"STRING","value":"warehouse"},"2":{"slot":2,"type":"STRING","value":"${stage_warehouse}"}}},"3":{"slot":3,"values":{"1":{"slot":1,"type":"STRING","value":"database"},"2":{"slot":2,"type":"STRING","value":"${stage_database}"}}},"4":{"slot":4,"values":{"1":{"slot":1,"type":"STRING","value":"schema"},"2":{"slot":2,"type":"STRING","value":"${stage_schema}"}}},"5":{"slot":5,"values":{"1":{"slot":1,"type":"STRING","value":"project"},"2":{"slot":2,"type":"STRING","value":"${stage_project}"}}},"6":{"slot":6,"values":{"1":{"slot":1,"type":"STRING","value":"dataset"},"2":{"slot":2,"type":"STRING","value":"${stage_dataset}"}}},"7":{"slot":7,"values":{"1":{"slot":1,"type":"STRING","value":"table"},"2":{"slot":2,"type":"STRING","value":"${metadata_stage_table_name}"}}},"8":{"slot":8,"values":{"1":{"slot":1,"type":"STRING","value":"s3_bucket_name"},"2":{"slot":2,"type":"STRING","value":"${s3_bucket_name}"}}},"9":{"slot":9,"values":{"1":{"slot":1,"type":"STRING","value":"dist_style"},"2":{"slot":2,"type":"STRING","value":"${dist_style}"}}},"10":{"slot":10,"values":{"1":{"slot":1,"type":"STRING","value":"staging_type"},"2":{"slot":2,"type":"STRING","value":"${staging_type}"}}},"11":{"slot":11,"values":{"1":{"slot":1,"type":"STRING","value":"blob_container"},"2":{"slot":2,"type":"STRING","value":"${blob_container}"}}},"12":{"slot":12,"values":{"1":{"slot":1,"type":"STRING","value":"storage_account"},"2":{"slot":2,"type":"STRING","value":"${storage_account}"}}},"13":{"slot":13,"values":{"1":{"slot":1,"type":"STRING","value":"cloud_storage_area"},"2":{"slot":2,"type":"STRING","value":"${cloud_storage_area}"}}},"14":{"slot":14,"values":{"1":{"slot":1,"type":"STRING","value":"encryption_method"},"2":{"slot":2,"type":"STRING","value":"${encryption_method}"}}},"15":{"slot":15,"values":{"1":{"slot":1,"type":"STRING","value":"kms_key_id"},"2":{"slot":2,"type":"STRING","value":"${kms_key_id}"}}},"16":{"slot":16,"values":{"1":{"slot":1,"type":"STRING","value":"auto_debug"},"2":{"slot":2,"type":"STRING","value":"${auto_debug}"}}},"17":{"slot":17,"values":{"1":{"slot":1,"type":"STRING","value":"auto_debug_level"},"2":{"slot":2,"type":"STRING","value":"${auto_debug_level}"}}},"18":{"slot":18,"values":{"1":{"slot":1,"type":"STRING","value":"target_table_name"},"2":{"slot":2,"type":"STRING","value":"${target_table_name}"}}},"19":{"slot":19,"values":{"1":{"slot":1,"type":"STRING","value":"api_key"},"2":{"slot":2,"type":"STRING","value":"${api_key}"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"Set Grid Variables","elements":{"1":{"slot":1,"elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"GRID","value":"advanced_connection_options"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"GRID","value":"parameter"},"2":{"slot":2,"type":"GRID","value":"value"}}}},"values":{"1":{"slot":1,"type":"STRING","value":"advanced_connection_options"},"2":{"slot":2,"type":"STRING","value":"grid"}}}},"visible":true,"status":{},"columnNames":[]},"6":{"slot":6,"name":"Melt Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Bing Search Incremental Load"}}}},"visible":true,"status":{},"columnNames":[]},"7":{"slot":7,"name":"Melt Package","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Matillion.Incremental"}}}},"visible":true,"status":{},"columnNames":[]},"8":{"slot":8,"name":"Melt Revision","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"INTEGER","value":"1"}}}},"visible":true,"status":{},"columnNames":[]},"9":{"slot":9,"name":"Melt Load Type","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"DYNAMIC"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25585":{"id":25585,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":-1773186829,"x":928,"y":0,"width":32,"height":32,"inputConnectorIDs":[25606],"outputSuccessConnectorIDs":[25596],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Build stage_single_sql"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Script","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"# set base stage_single_sql\nstage_single_sql = (f'SELECT {source_column_list} \\n' + \n             f'FROM \"{table_name}\"'\n)\n\npredicates = []\nfilter_columns = context.getGridVariable('filter_columns')\n\n# if filter columns have been identified, add them to the query\nif filter_columns:\n    for name, value in filter_columns:\n        predicates.append(f'\"{name}\" = \\'{value}\\'')\n\n# add WHERE clause, if necessary\nif predicates:\n    stage_single_sql += ' \\nWHERE ' + ' \\nAND '.join(predicates)\n\nstage_single_sql += ' \\nLIMIT 1'\n\ncontext.updateVariable('stage_single_sql', stage_single_sql)\nprint(f\"Updated Variable 'stage_single_sql' with {stage_single_sql}\")"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"Interpreter","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Python 3"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Timeout","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"INTEGER","value":"360"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"User","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Restricted"}}}},"visible":false,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25586":{"id":25586,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"TRANSFORM","implementationID":-1180878087,"x":1104,"y":0,"width":32,"height":32,"inputConnectorIDs":[25596],"outputSuccessConnectorIDs":[25603],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Stage Single Row"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Job Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Bing Search - 4 - Query Source"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"","elements":{},"visible":false,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Set Scalar Variables","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"sql"},"2":{"slot":2,"type":"STRING","value":"${stage_single_sql}"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"STRING","value":"warehouse"},"2":{"slot":2,"type":"STRING","value":"${stage_warehouse}"}}},"3":{"slot":3,"values":{"1":{"slot":1,"type":"STRING","value":"database"},"2":{"slot":2,"type":"STRING","value":"${stage_database}"}}},"4":{"slot":4,"values":{"1":{"slot":1,"type":"STRING","value":"schema"},"2":{"slot":2,"type":"STRING","value":"${stage_schema}"}}},"5":{"slot":5,"values":{"1":{"slot":1,"type":"STRING","value":"project"},"2":{"slot":2,"type":"STRING","value":"${stage_project}"}}},"6":{"slot":6,"values":{"1":{"slot":1,"type":"STRING","value":"dataset"},"2":{"slot":2,"type":"STRING","value":"${stage_dataset}"}}},"7":{"slot":7,"values":{"1":{"slot":1,"type":"STRING","value":"table"},"2":{"slot":2,"type":"STRING","value":"${stage_table_name}"}}},"8":{"slot":8,"values":{"1":{"slot":1,"type":"STRING","value":"s3_bucket_name"},"2":{"slot":2,"type":"STRING","value":"${s3_bucket_name}"}}},"9":{"slot":9,"values":{"1":{"slot":1,"type":"STRING","value":"dist_style"},"2":{"slot":2,"type":"STRING","value":"${dist_style}"}}},"10":{"slot":10,"values":{"1":{"slot":1,"type":"STRING","value":"staging_type"},"2":{"slot":2,"type":"STRING","value":"${staging_type}"}}},"11":{"slot":11,"values":{"1":{"slot":1,"type":"STRING","value":"blob_container"},"2":{"slot":2,"type":"STRING","value":"${blob_container}"}}},"12":{"slot":12,"values":{"1":{"slot":1,"type":"STRING","value":"storage_account"},"2":{"slot":2,"type":"STRING","value":"${storage_account}"}}},"13":{"slot":13,"values":{"1":{"slot":1,"type":"STRING","value":"cloud_storage_area"},"2":{"slot":2,"type":"STRING","value":"${cloud_storage_area}"}}},"14":{"slot":14,"values":{"1":{"slot":1,"type":"STRING","value":"encryption_method"},"2":{"slot":2,"type":"STRING","value":"${encryption_method}"}}},"15":{"slot":15,"values":{"1":{"slot":1,"type":"STRING","value":"kms_key_id"},"2":{"slot":2,"type":"STRING","value":"${kms_key_id}"}}},"16":{"slot":16,"values":{"1":{"slot":1,"type":"STRING","value":"auto_debug"},"2":{"slot":2,"type":"STRING","value":"${auto_debug}"}}},"17":{"slot":17,"values":{"1":{"slot":1,"type":"STRING","value":"auto_debug_level"},"2":{"slot":2,"type":"STRING","value":"${auto_debug_level}"}}},"18":{"slot":18,"values":{"1":{"slot":1,"type":"STRING","value":"target_table_name"},"2":{"slot":2,"type":"STRING","value":"${target_table_name}"}}},"19":{"slot":19,"values":{"1":{"slot":1,"type":"STRING","value":"api_key"},"2":{"slot":2,"type":"STRING","value":"${api_key}"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"Set Grid Variables","elements":{"1":{"slot":1,"elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"GRID","value":"advanced_connection_options"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"GRID","value":"parameter"},"2":{"slot":2,"type":"GRID","value":"value"}}}},"values":{"1":{"slot":1,"type":"STRING","value":"advanced_connection_options"},"2":{"slot":2,"type":"STRING","value":"grid"}}}},"visible":true,"status":{},"columnNames":[]},"6":{"slot":6,"name":"Melt Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Bing Search Incremental Load"}}}},"visible":true,"status":{},"columnNames":[]},"7":{"slot":7,"name":"Melt Package","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Matillion.Incremental"}}}},"visible":true,"status":{},"columnNames":[]},"8":{"slot":8,"name":"Melt Revision","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"INTEGER","value":"1"}}}},"visible":true,"status":{},"columnNames":[]},"9":{"slot":9,"name":"Melt Load Type","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"DYNAMIC"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25587":{"id":25587,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":-231597779,"x":448,"y":0,"width":32,"height":32,"inputConnectorIDs":[25599],"outputSuccessConnectorIDs":[25602],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"gridExportMappings":{"1":{"slot":1,"fromGrid":"required_columns","toGrid":"source_list","columnMapToFrom":{"column_name":"column_name","key_column":"key_column","incremental_column":"incremental_column","table_name":"table"},"validationStatus":null,"validationMessage":null}},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Check Required Columns"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"target_table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_table_name}"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"actual_columns","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"GRID","value":"source_table_metadata"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"GRID","value":"column_name"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"required_columns","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"GRID","value":"source_list"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"GRID","value":"table_name"},"2":{"slot":2,"type":"GRID","value":"column_name"},"3":{"slot":3,"type":"GRID","value":"incremental_column"},"4":{"slot":4,"type":"GRID","value":"key_column"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"missing_columns","elements":{},"visible":true,"status":{},"columnNames":[]},"6":{"slot":6,"name":"error_message","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25588":{"id":25588,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":947479920,"x":608,"y":0,"width":32,"height":32,"inputConnectorIDs":[25602],"outputSuccessConnectorIDs":[25605],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{"1":{"slot":1,"fromId":null,"fromName":"incremental_column","mapTo":"incremental_column","validationStatus":null,"validationMessage":null},"2":{"slot":2,"fromId":null,"fromName":"incremental_column_count","mapTo":"incremental_column_count","validationStatus":null,"validationMessage":null},"3":{"slot":3,"fromId":null,"fromName":"SELECT_STRING","mapTo":"source_column_list","validationStatus":null,"validationMessage":null},"4":{"slot":4,"fromId":null,"fromName":"key_column_count","mapTo":"key_column_count","validationStatus":null,"validationMessage":null}},"gridExportMappings":{"1":{"slot":1,"fromGrid":"key_columns","toGrid":"key_columns","columnMapToFrom":{"column_name":"column_name"},"validationStatus":null,"validationMessage":null}},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Parameter Parser"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"platform","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_platform}"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${table_name}"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"incremental_source_list","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"GRID","value":"source_list"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"GRID","value":"table_name"},"2":{"slot":2,"type":"GRID","value":"column_name"},"3":{"slot":3,"type":"GRID","value":"incremental_column"},"4":{"slot":4,"type":"GRID","value":"key_column"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"source_list","elements":{},"visible":true,"status":{},"columnNames":[]},"6":{"slot":6,"name":"stage_warehouse","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_warehouse}"}}}},"visible":true,"status":{},"columnNames":[]},"7":{"slot":7,"name":"stage_database","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_database}"}}}},"visible":true,"status":{},"columnNames":[]},"8":{"slot":8,"name":"stage_schema","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_schema}"}}}},"visible":true,"status":{},"columnNames":[]},"9":{"slot":9,"name":"stage_project","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_project}"}}}},"visible":true,"status":{},"columnNames":[]},"10":{"slot":10,"name":"stage_dataset","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_dataset}"}}}},"visible":true,"status":{},"columnNames":[]},"11":{"slot":11,"name":"stage_prefix","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_prefix}"}}}},"visible":true,"status":{},"columnNames":[]},"12":{"slot":12,"name":"target_warehouse","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_warehouse}"}}}},"visible":true,"status":{},"columnNames":[]},"13":{"slot":13,"name":"target_database","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_database}"}}}},"visible":true,"status":{},"columnNames":[]},"14":{"slot":14,"name":"target_schema","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_schema}"}}}},"visible":true,"status":{},"columnNames":[]},"15":{"slot":15,"name":"target_project","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_project}"}}}},"visible":true,"status":{},"columnNames":[]},"16":{"slot":16,"name":"target_dataset","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_dataset}"}}}},"visible":true,"status":{},"columnNames":[]},"17":{"slot":17,"name":"target_prefix","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_prefix}"}}}},"visible":true,"status":{},"columnNames":[]},"18":{"slot":18,"name":"maximum_incremental_columns","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"DECIMAL","value":"1"}}}},"visible":true,"status":{},"columnNames":[]},"19":{"slot":19,"name":"source_enclosing_char_e","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"\""}}}},"visible":true,"status":{},"columnNames":[]},"20":{"slot":20,"name":"source_enclosing_char_s","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"\""}}}},"visible":true,"status":{},"columnNames":[]},"21":{"slot":21,"name":"metadata_stage_table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${metadata_stage_table_name}"}}}},"visible":true,"status":{},"columnNames":[]},"22":{"slot":22,"name":"stage_table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_table_name}"}}}},"visible":true,"status":{},"columnNames":[]},"23":{"slot":23,"name":"target_table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_table_name}"}}}},"visible":true,"status":{},"columnNames":[]},"24":{"slot":24,"name":"column_count","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"DECIMAL","value":"0"}}}},"visible":true,"status":{},"columnNames":[]},"25":{"slot":25,"name":"key_column_count","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"DECIMAL","value":"0"}}}},"visible":true,"status":{},"columnNames":[]},"26":{"slot":26,"name":"key_columns","elements":{},"visible":true,"status":{},"columnNames":[]},"27":{"slot":27,"name":"incremental_column_count","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"DECIMAL","value":"0"}}}},"visible":true,"status":{},"columnNames":[]},"28":{"slot":28,"name":"incremental_column","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]},"29":{"slot":29,"name":"SELECT_STRING","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]},"30":{"slot":30,"name":"stage_table_metadata_sql","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25589":{"id":25589,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"TRANSFORM","implementationID":-1180878087,"x":2064,"y":0,"width":32,"height":32,"inputConnectorIDs":[25607],"outputSuccessConnectorIDs":[25597],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{"1":{"slot":1,"fromId":null,"fromName":"static_row_count","mapTo":"static_row_count","validationStatus":null,"validationMessage":null}},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Stage Data"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Job Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Bing Search - 4 - Query Source"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"","elements":{},"visible":false,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Set Scalar Variables","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"sql"},"2":{"slot":2,"type":"STRING","value":"${stage_sql}"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"STRING","value":"warehouse"},"2":{"slot":2,"type":"STRING","value":"${stage_warehouse}"}}},"3":{"slot":3,"values":{"1":{"slot":1,"type":"STRING","value":"database"},"2":{"slot":2,"type":"STRING","value":"${stage_database}"}}},"4":{"slot":4,"values":{"1":{"slot":1,"type":"STRING","value":"schema"},"2":{"slot":2,"type":"STRING","value":"${stage_schema}"}}},"5":{"slot":5,"values":{"1":{"slot":1,"type":"STRING","value":"project"},"2":{"slot":2,"type":"STRING","value":"${stage_project}"}}},"6":{"slot":6,"values":{"1":{"slot":1,"type":"STRING","value":"dataset"},"2":{"slot":2,"type":"STRING","value":"${stage_dataset}"}}},"7":{"slot":7,"values":{"1":{"slot":1,"type":"STRING","value":"table"},"2":{"slot":2,"type":"STRING","value":"${stage_table_name}"}}},"8":{"slot":8,"values":{"1":{"slot":1,"type":"STRING","value":"s3_bucket_name"},"2":{"slot":2,"type":"STRING","value":"${s3_bucket_name}"}}},"9":{"slot":9,"values":{"1":{"slot":1,"type":"STRING","value":"dist_style"},"2":{"slot":2,"type":"STRING","value":"${dist_style}"}}},"10":{"slot":10,"values":{"1":{"slot":1,"type":"STRING","value":"staging_type"},"2":{"slot":2,"type":"STRING","value":"${staging_type}"}}},"11":{"slot":11,"values":{"1":{"slot":1,"type":"STRING","value":"blob_container"},"2":{"slot":2,"type":"STRING","value":"${blob_container}"}}},"12":{"slot":12,"values":{"1":{"slot":1,"type":"STRING","value":"storage_account"},"2":{"slot":2,"type":"STRING","value":"${storage_account}"}}},"13":{"slot":13,"values":{"1":{"slot":1,"type":"STRING","value":"cloud_storage_area"},"2":{"slot":2,"type":"STRING","value":"${cloud_storage_area}"}}},"14":{"slot":14,"values":{"1":{"slot":1,"type":"STRING","value":"encryption_method"},"2":{"slot":2,"type":"STRING","value":"${encryption_method}"}}},"15":{"slot":15,"values":{"1":{"slot":1,"type":"STRING","value":"kms_key_id"},"2":{"slot":2,"type":"STRING","value":"${kms_key_id}"}}},"16":{"slot":16,"values":{"1":{"slot":1,"type":"STRING","value":"auto_debug"},"2":{"slot":2,"type":"STRING","value":"${auto_debug}"}}},"17":{"slot":17,"values":{"1":{"slot":1,"type":"STRING","value":"auto_debug_level"},"2":{"slot":2,"type":"STRING","value":"${auto_debug_level}"}}},"18":{"slot":18,"values":{"1":{"slot":1,"type":"STRING","value":"target_table_name"},"2":{"slot":2,"type":"STRING","value":"${target_table_name}"}}},"19":{"slot":19,"values":{"1":{"slot":1,"type":"STRING","value":"api_key"},"2":{"slot":2,"type":"STRING","value":"${api_key}"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"Set Grid Variables","elements":{"1":{"slot":1,"elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"GRID","value":"advanced_connection_options"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"GRID","value":"parameter"},"2":{"slot":2,"type":"GRID","value":"value"}}}},"values":{"1":{"slot":1,"type":"STRING","value":"advanced_connection_options"},"2":{"slot":2,"type":"STRING","value":"grid"}}}},"visible":true,"status":{},"columnNames":[]},"6":{"slot":6,"name":"Melt Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Bing Search Incremental Load"}}}},"visible":true,"status":{},"columnNames":[]},"7":{"slot":7,"name":"Melt Package","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Matillion.Incremental"}}}},"visible":true,"status":{},"columnNames":[]},"8":{"slot":8,"name":"Melt Revision","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"INTEGER","value":"1"}}}},"visible":true,"status":{},"columnNames":[]},"9":{"slot":9,"name":"Melt Load Type","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"DYNAMIC"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25590":{"id":25590,"inputCardinality":"ONE","outputCardinality":"ZERO","connectorHint":"UNCONDITIONAL","executionHint":"FLOW","implementationID":-1946388514,"x":2304,"y":0,"width":32,"height":32,"inputConnectorIDs":[25601],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"End Success"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25591":{"id":25591,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":-313802845,"x":2192,"y":0,"width":32,"height":32,"inputConnectorIDs":[25597],"outputSuccessConnectorIDs":[25601],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Load Target"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"platform","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_platform}"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"stage_warehouse","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_warehouse}"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"stage_database","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_database}"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"stage_schema","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_schema}"}}}},"visible":true,"status":{},"columnNames":[]},"6":{"slot":6,"name":"stage_project","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_project}"}}}},"visible":true,"status":{},"columnNames":[]},"7":{"slot":7,"name":"stage_dataset","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_dataset}"}}}},"visible":true,"status":{},"columnNames":[]},"8":{"slot":8,"name":"stage_table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_table_name}"}}}},"visible":true,"status":{},"columnNames":[]},"9":{"slot":9,"name":"target_warehouse","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_warehouse}"}}}},"visible":true,"status":{},"columnNames":[]},"10":{"slot":10,"name":"target_database","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_database}"}}}},"visible":true,"status":{},"columnNames":[]},"11":{"slot":11,"name":"target_schema","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_schema}"}}}},"visible":true,"status":{},"columnNames":[]},"12":{"slot":12,"name":"target_project","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_project}"}}}},"visible":true,"status":{},"columnNames":[]},"13":{"slot":13,"name":"target_dataset","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_dataset}"}}}},"visible":true,"status":{},"columnNames":[]},"14":{"slot":14,"name":"target_table_name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${target_table_name}"}}}},"visible":true,"status":{},"columnNames":[]},"15":{"slot":15,"name":"load_strategy","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${load_strategy}"}}}},"visible":true,"status":{},"columnNames":[]},"16":{"slot":16,"name":"dist_style","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${dist_style}"}}}},"visible":true,"status":{},"columnNames":[]},"17":{"slot":17,"name":"columns","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"GRID","value":"stage_table_metadata"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"GRID","value":"column_name"}}}},"visible":true,"status":{},"columnNames":[]},"18":{"slot":18,"name":"key_columns","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"GRID","value":"key_columns"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"GRID","value":"column_name"}}}},"visible":true,"status":{},"columnNames":[]},"19":{"slot":19,"name":"incremental_column","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${incremental_column}"}}}},"visible":true,"status":{},"columnNames":[]},"20":{"slot":20,"name":"join_statement","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]},"21":{"slot":21,"name":"log_metrics","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${log_metrics}"}}}},"visible":true,"status":{},"columnNames":[]},"22":{"slot":22,"name":"static_row_count","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${static_row_count}"}}}},"visible":true,"status":{},"columnNames":[]},"23":{"slot":23,"name":"logged_rows_inserted","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"DECIMAL","value":"0"}}}},"visible":true,"status":{},"columnNames":[]},"24":{"slot":24,"name":"logged_rows_update","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"DECIMAL","value":"0"}}}},"visible":true,"status":{},"columnNames":[]},"25":{"slot":25,"name":"logged_rows_deleted","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"DECIMAL","value":"0"}}}},"visible":true,"status":{},"columnNames":[]},"26":{"slot":26,"name":"sub_RangeStartDate","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":""}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25592":{"id":25592,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":-1773186960,"x":768,"y":0,"width":32,"height":32,"inputConnectorIDs":[25605],"outputSuccessConnectorIDs":[25606],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Identify filter column"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Script","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"def truncate(word, length = 12):\n  \"\"\"\n  `truncate()` will convert items to a string and ensure they are the length specified by user \n  Used within stringer function to ensure all columns are the same width \n  \"\"\"\n  if len(word.strip()) <= length:\n    return word\n  return \"{}...\".format( word[:(length-3)] )\n  \ndef stringer(lst, colwidth = 30):\n  \"\"\"\n  `stringer()` is used for printing Grid Variables in a more readable manner.\n  \n  It will handle formatting the grid variable to look more like how it is presented in the product \n  (as a table) \n  \"\"\"\n  msgs = [] \n  for row in lst:\n    msg = \" | \".join(\n    \t[\n        \"{:<{width}}\".format( truncate(str(x), length = colwidth), width = colwidth)\n        for x in row \n        ]\n    )\n    msgs.append(msg)\n    \n  return \"\\n\" + \"\\n\".join(msgs)\n\nfilter_vars = {\n  'SearchTerms': 'search_terms',\n  'SafeSearch': 'safe_search',\n  'Market': 'market'\n}\nfilter_columns = []\n\n# If any of the filter parameters are supplied, add them to the filter_columns grid var\nfor key, val in filter_vars.items():\n  \n  filter_val = locals().copy()[val]\n  \n  if filter_val:\n    # remove single quotes to prevent SQL injection / errors\n    filter_val = filter_val.replace(\"'\", \"\")\n    \n    filter_columns.append([key, filter_val])\n  else:\n    print(f'\\nNot applying a filter on \"{key}\" as {val} has not been set')\n\nif filter_columns:\n  context.updateGridVariable('filter_columns', filter_columns)\n  print(\"\\nGrid variable 'filter_columns': {}\".format(stringer(filter_columns)))\nelse:\n  print('\\nNo filter columns applied')\n"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"Interpreter","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Python 3"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Timeout","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"INTEGER","value":"360"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25593":{"id":25593,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":-1773186960,"x":1952,"y":0,"width":32,"height":32,"inputConnectorIDs":[25594],"outputSuccessConnectorIDs":[25607],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Build stage_sql"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Script","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"# set base stage_sql\nstage_sql = (f'SELECT {source_column_list} \\n' + \n             f'FROM \"{table_name}\"'\n)\n\npredicates = []\nfilter_columns = context.getGridVariable('filter_columns')\n\n# if filter columns have been identified, add them to the query\nif filter_columns:\n    for name, value in filter_columns:\n        predicates.append(f'\"{name}\" = \\'{value}\\'')\n    \n# add WHERE clause, if necessary\nif predicates:\n    stage_sql += ' \\nWHERE ' + ' \\nAND '.join(predicates)\n\ncontext.updateVariable('stage_sql', stage_sql)\nprint(f\"Updated Variable 'stage_sql' with {stage_sql}\")\n"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"Interpreter","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Python 3"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Timeout","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"INTEGER","value":"360"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]}},"successConnectors":{"25594":{"id":25594,"sourceID":25579,"targetID":25593},"25595":{"id":25595,"sourceID":25581,"targetID":25584},"25596":{"id":25596,"sourceID":25585,"targetID":25586},"25597":{"id":25597,"sourceID":25589,"targetID":25591},"25598":{"id":25598,"sourceID":25584,"targetID":25582},"25599":{"id":25599,"sourceID":25582,"targetID":25587},"25601":{"id":25601,"sourceID":25591,"targetID":25590},"25602":{"id":25602,"sourceID":25587,"targetID":25588},"25603":{"id":25603,"sourceID":25586,"targetID":25580},"25604":{"id":25604,"sourceID":25580,"targetID":25579},"25605":{"id":25605,"sourceID":25588,"targetID":25592},"25606":{"id":25606,"sourceID":25592,"targetID":25585},"25607":{"id":25607,"sourceID":25593,"targetID":25589}},"failureConnectors":{},"unconditionalConnectors":{"25608":{"id":25608,"sourceID":25583,"targetID":25581}},"trueConnectors":{},"falseConnectors":{},"iterationConnectors":{},"noteConnectors":{},"canUndo":false,"undoCommand":"","undoCreated":-1,"canRedo":false,"redoCommand":"","redoCreated":-1,"notes":{"25575":{"id":25575,"x":1896,"y":-252,"width":446,"height":313,"text":"** Load Data **\n- (__Build stage_sql__) Generates and sets stage_single_sql variable which is used to query all rows from the source. This logic is in its own Python script so that we can log the query. This will include  'SearchTerms', 'SafeSearch' and  'Market' as part of the predicate.\n\n- (__Stage Data__) Executes __stage_sql__ query and stores the data in the staging table \n\n- (__Load Target__) Executes logic to move the Stage Table Data into the Target Table using the desired __Load Strategy__","colour":"e6e63c"},"25576":{"id":25576,"x":-234,"y":-239,"width":472,"height":314,"text":"** Source Validation **\n\n- (__Build Strings__) Generate generic job variables (stage/target table names etc. - see Export tab) \n\n- (__Get Source Metadata__) Query CData __sys_tablecolumns__ metadata table to get information about the source and store in CDW \n\n- (__Check Source Metadata__) Query table created above, ensure it is valid, export the metadata to a grid (see Grid Export tab)","colour":"e6e63c"},"25577":{"id":25577,"x":369,"y":-244,"width":802,"height":316,"text":"**Input/Target Validation**\n\n- (__Check Required Columns__) Contains logic to validate that the columns which have been requested still exist in the source\n\n- (__Parameter Parser__) Contains logic to generate some useful Grid/Job variables (e.g key_columns Grid, SELECT list. See Export/Grid Export tabs)\n\n- (__Identify filter column__) Query source metadata to find optional filter columns\n+ If the table contains 'SearchTerms', 'SafeSearch' or 'Market', add it to the __filter_columns__ grid variable\n\n- (__Build stage_single_sql__) Generates and sets stage_single_sql variable which is used to query 1 row from the source. This logic is in its own Python script so that we can log the query. This will include  'SearchTerms', 'SafeSearch' and  'Market' as part of the predicate.\n\n- (__Stage Single Row__) Executes the __stage_single_sql__ query. This creates a table which will be identical to the stage table for the ETL process, but with only 1 row - the metadata of this table is used within Decide Load Strategy.","colour":"e6e63c"},"25578":{"id":25578,"x":1208,"y":-255,"width":660,"height":328,"text":"The tables have no key or incremental columns, so the user sets __truncate_tables__ to decide between a full load and append load.\n\n- (__Link use_incremental_column__) A python script is added to link the variable __use_incremental_column__ used in __Decided Load Strategy__, to the variable __truncate_tables__. \nThis will connect which option the user sets in __truncate_tables__ to be the same option in __use_incremental_column__ to ensure __Decided Load Strategy__ can work effectively. \n\nIf __truncate_tables__ is \"yes\", we can call __Decide Load Strategy__ as standard, setting the variable __use_incremental_column__ to 'Y'.\n\nIf __truncate_tables__ is \"no\", __Decide Load Strategy__ will apply a strategy of APPEND INSERT by setting the variable __use_incremental_column__ to 'N'.\n\n- (__Decide Load Strategy__) Executes packaged up logic to determine the type of load this job is doing (UPDATE/APPEND INSERT/TRUNCATE INSERT). \nThis exports the __stage_sql__ variable which will be used to stage the dataset.\nThis exports the __load_strategy__ variable which will be used within Load Target","colour":"e6e63c"}},"variables":{"market":{"definition":{"name":"market","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PUBLIC"},"value":""},"api_key":{"definition":{"name":"api_key","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"stage_sql":{"definition":{"name":"stage_sql","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":""},"auto_debug":{"definition":{"name":"auto_debug","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"dist_style":{"definition":{"name":"dist_style","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"kms_key_id":{"definition":{"name":"kms_key_id","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"table_name":{"definition":{"name":"table_name","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"log_metrics":{"definition":{"name":"log_metrics","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"safe_search":{"definition":{"name":"safe_search","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"inc_col_type":{"definition":{"name":"inc_col_type","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":""},"search_terms":{"definition":{"name":"search_terms","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"stage_prefix":{"definition":{"name":"stage_prefix","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"stage_schema":{"definition":{"name":"stage_schema","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"staging_type":{"definition":{"name":"staging_type","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"error_message":{"definition":{"name":"error_message","type":"TEXT","scope":"BRANCH","description":"","visibility":"PRIVATE"},"value":""},"load_strategy":{"definition":{"name":"load_strategy","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":""},"stage_dataset":{"definition":{"name":"stage_dataset","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"stage_project":{"definition":{"name":"stage_project","type":"TEXT","scope":"BRANCH","description":"   ","visibility":"PUBLIC"},"value":""},"target_prefix":{"definition":{"name":"target_prefix","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"target_schema":{"definition":{"name":"target_schema","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"blob_container":{"definition":{"name":"blob_container","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"s3_bucket_name":{"definition":{"name":"s3_bucket_name","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"stage_database":{"definition":{"name":"stage_database","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"target_dataset":{"definition":{"name":"target_dataset","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"target_project":{"definition":{"name":"target_project","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"stage_warehouse":{"definition":{"name":"stage_warehouse","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"storage_account":{"definition":{"name":"storage_account","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"target_database":{"definition":{"name":"target_database","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"target_platform":{"definition":{"name":"target_platform","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"truncate_tables":{"definition":{"name":"truncate_tables","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PUBLIC"},"value":""},"auto_debug_level":{"definition":{"name":"auto_debug_level","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"get_metadata_sql":{"definition":{"name":"get_metadata_sql","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":""},"key_column_count":{"definition":{"name":"key_column_count","type":"DECIMAL","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":""},"stage_single_sql":{"definition":{"name":"stage_single_sql","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":""},"stage_table_name":{"definition":{"name":"stage_table_name","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PUBLIC"},"value":""},"static_row_count":{"definition":{"name":"static_row_count","type":"DECIMAL","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":""},"target_warehouse":{"definition":{"name":"target_warehouse","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"encryption_method":{"definition":{"name":"encryption_method","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"target_table_name":{"definition":{"name":"target_table_name","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":""},"cloud_storage_area":{"definition":{"name":"cloud_storage_area","type":"TEXT","scope":"BRANCH","description":"The cloud storage area name that should be used when loading bq tables.","visibility":"PUBLIC"},"value":""},"incremental_column":{"definition":{"name":"incremental_column","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":""},"source_column_list":{"definition":{"name":"source_column_list","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":""},"source_metadata_sql":{"definition":{"name":"source_metadata_sql","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":""},"timestamp_precision":{"definition":{"name":"timestamp_precision","type":"TEXT","scope":"BRANCH","description":"","visibility":"PRIVATE"},"value":"microseconds"},"use_incremental_column":{"definition":{"name":"use_incremental_column","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":null},"incremental_column_count":{"definition":{"name":"incremental_column_count","type":"DECIMAL","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":"0"},"metadata_stage_table_name":{"definition":{"name":"metadata_stage_table_name","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PUBLIC"},"value":""}},"grids":{"key_columns":{"definition":{"name":"key_columns","scope":"BRANCH","definitions":[{"name":"column_name","type":"TEXT"}],"description":"","visibility":"PRIVATE"},"values":[]},"source_list":{"definition":{"name":"source_list","scope":"BRANCH","definitions":[{"name":"table_name","type":"TEXT"},{"name":"column_name","type":"TEXT"},{"name":"incremental_column","type":"DECIMAL"},{"name":"key_column","type":"DECIMAL"}],"description":"","visibility":"PUBLIC"},"values":[]},"filter_columns":{"definition":{"name":"filter_columns","scope":"TASKBATCH","definitions":[{"name":"name","type":"TEXT"},{"name":"value","type":"TEXT"}],"description":"","visibility":"PRIVATE"},"values":[]},"stage_table_metadata":{"definition":{"name":"stage_table_metadata","scope":"TASKBATCH","definitions":[{"name":"column_name","type":"TEXT"},{"name":"type","type":"TEXT"},{"name":"size","type":"TEXT"},{"name":"precision","type":"TEXT"},{"name":"database_type","type":"TEXT"}],"description":"","visibility":"PRIVATE"},"values":[]},"source_table_metadata":{"definition":{"name":"source_table_metadata","scope":"BRANCH","definitions":[{"name":"column_name","type":"TEXT"},{"name":"data_type_name","type":"TEXT"},{"name":"length","type":"TEXT"},{"name":"numeric_scale","type":"TEXT"},{"name":"nullable","type":"TEXT"},{"name":"ordinal","type":"TEXT"},{"name":"is_key","type":"TEXT"}],"description":"","visibility":"PRIVATE"},"values":[]},"advanced_connection_options":{"definition":{"name":"advanced_connection_options","scope":"BRANCH","definitions":[{"name":"parameter","type":"TEXT"},{"name":"value","type":"TEXT"}],"description":"","visibility":"PUBLIC"},"values":[]}}},"Bing Search - 2 - Wrapper":{"id":-2,"revision":2,"created":1620657489306,"timestamp":1620657489306,"components":{"25565":{"id":25565,"inputCardinality":"ZERO","outputCardinality":"MANY","connectorHint":"UNCONDITIONAL","executionHint":"FLOW","implementationID":444132438,"x":-720,"y":0,"width":32,"height":32,"inputConnectorIDs":[],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[25571],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Start","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Start"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25566":{"id":25566,"inputCardinality":"ONE","outputCardinality":"ZERO","connectorHint":"UNCONDITIONAL","executionHint":"FLOW","implementationID":515156205,"x":-399,"y":161,"width":32,"height":32,"inputConnectorIDs":[25573],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Failure"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25567":{"id":25567,"inputCardinality":"ONE","outputCardinality":"ZERO","connectorHint":"UNCONDITIONAL","executionHint":"FLOW","implementationID":-1946388514,"x":80,"y":0,"width":32,"height":32,"inputConnectorIDs":[25570],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"End"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25568":{"id":25568,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"EXECUTE","implementationID":-1291941950,"x":-240,"y":0,"width":32,"height":32,"inputConnectorIDs":[25572],"outputSuccessConnectorIDs":[25570],"outputFailureConnectorIDs":[],"outputUnconditionalConnectorIDs":[],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Drop Stage Tables"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Database","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_database}"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"Schema","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${stage_schema}"}}}},"visible":true,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Target Table","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"${metadata_stage_table_name}, ${stage_table_name}"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"Ignore Missing","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Yes"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]},"25569":{"id":25569,"inputCardinality":"ONE","outputCardinality":"MANY","connectorHint":"SUCCESS_FAIL","executionHint":"TRANSFORM","implementationID":-1180878087,"x":-400,"y":0,"width":32,"height":32,"inputConnectorIDs":[25571],"outputSuccessConnectorIDs":[],"outputFailureConnectorIDs":[25573],"outputUnconditionalConnectorIDs":[25572],"outputTrueConnectorIDs":[],"outputFalseConnectorIDs":[],"exportMappings":{"1":{"slot":1,"fromId":null,"fromName":"metadata_stage_table_name","mapTo":"metadata_stage_table_name","validationStatus":null,"validationMessage":null},"2":{"slot":2,"fromId":null,"fromName":"stage_table_name","mapTo":"stage_table_name","validationStatus":null,"validationMessage":null}},"parameters":{"1":{"slot":1,"name":"Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Stage Object"}}}},"visible":true,"status":{},"columnNames":[]},"2":{"slot":2,"name":"Job Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Bing Search - 3 - Stage Object"}}}},"visible":true,"status":{},"columnNames":[]},"3":{"slot":3,"name":"","elements":{},"visible":false,"status":{},"columnNames":[]},"4":{"slot":4,"name":"Set Scalar Variables","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"target_platform"},"2":{"slot":2,"type":"STRING","value":"${target_platform}"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"STRING","value":"stage_warehouse"},"2":{"slot":2,"type":"STRING","value":"${stage_warehouse}"}}},"3":{"slot":3,"values":{"1":{"slot":1,"type":"STRING","value":"stage_database"},"2":{"slot":2,"type":"STRING","value":"${stage_database}"}}},"4":{"slot":4,"values":{"1":{"slot":1,"type":"STRING","value":"stage_schema"},"2":{"slot":2,"type":"STRING","value":"${stage_schema}"}}},"5":{"slot":5,"values":{"1":{"slot":1,"type":"STRING","value":"stage_project"},"2":{"slot":2,"type":"STRING","value":"${stage_project}"}}},"6":{"slot":6,"values":{"1":{"slot":1,"type":"STRING","value":"stage_dataset"},"2":{"slot":2,"type":"STRING","value":"${stage_dataset}"}}},"7":{"slot":7,"values":{"1":{"slot":1,"type":"STRING","value":"stage_prefix"},"2":{"slot":2,"type":"STRING","value":"${stage_prefix}"}}},"8":{"slot":8,"values":{"1":{"slot":1,"type":"STRING","value":"target_warehouse"},"2":{"slot":2,"type":"STRING","value":"${target_warehouse}"}}},"9":{"slot":9,"values":{"1":{"slot":1,"type":"STRING","value":"target_database"},"2":{"slot":2,"type":"STRING","value":"${target_database}"}}},"10":{"slot":10,"values":{"1":{"slot":1,"type":"STRING","value":"target_schema"},"2":{"slot":2,"type":"STRING","value":"${target_schema}"}}},"11":{"slot":11,"values":{"1":{"slot":1,"type":"STRING","value":"target_project"},"2":{"slot":2,"type":"STRING","value":"${target_project}"}}},"12":{"slot":12,"values":{"1":{"slot":1,"type":"STRING","value":"target_dataset"},"2":{"slot":2,"type":"STRING","value":"${target_dataset}"}}},"13":{"slot":13,"values":{"1":{"slot":1,"type":"STRING","value":"target_prefix"},"2":{"slot":2,"type":"STRING","value":"${target_prefix}"}}},"14":{"slot":14,"values":{"1":{"slot":1,"type":"STRING","value":"table_name"},"2":{"slot":2,"type":"STRING","value":"${table_name}"}}},"15":{"slot":15,"values":{"1":{"slot":1,"type":"STRING","value":"s3_bucket_name"},"2":{"slot":2,"type":"STRING","value":"${s3_bucket_name}"}}},"16":{"slot":16,"values":{"1":{"slot":1,"type":"STRING","value":"dist_style"},"2":{"slot":2,"type":"STRING","value":"${dist_style}"}}},"17":{"slot":17,"values":{"1":{"slot":1,"type":"STRING","value":"staging_type"},"2":{"slot":2,"type":"STRING","value":"${staging_type}"}}},"18":{"slot":18,"values":{"1":{"slot":1,"type":"STRING","value":"blob_container"},"2":{"slot":2,"type":"STRING","value":"${blob_container}"}}},"19":{"slot":19,"values":{"1":{"slot":1,"type":"STRING","value":"storage_account"},"2":{"slot":2,"type":"STRING","value":"${storage_account}"}}},"20":{"slot":20,"values":{"1":{"slot":1,"type":"STRING","value":"cloud_storage_area"},"2":{"slot":2,"type":"STRING","value":"${cloud_storage_area}"}}},"21":{"slot":21,"values":{"1":{"slot":1,"type":"STRING","value":"encryption_method"},"2":{"slot":2,"type":"STRING","value":"${encryption_method}"}}},"22":{"slot":22,"values":{"1":{"slot":1,"type":"STRING","value":"kms_key_id"},"2":{"slot":2,"type":"STRING","value":"${kms_key_id}"}}},"23":{"slot":23,"values":{"1":{"slot":1,"type":"STRING","value":"auto_debug"},"2":{"slot":2,"type":"STRING","value":"${auto_debug}"}}},"24":{"slot":24,"values":{"1":{"slot":1,"type":"STRING","value":"auto_debug_level"},"2":{"slot":2,"type":"STRING","value":"${auto_debug_level}"}}},"25":{"slot":25,"values":{"1":{"slot":1,"type":"STRING","value":"log_metrics"},"2":{"slot":2,"type":"STRING","value":"${log_metrics}"}}},"26":{"slot":26,"values":{"1":{"slot":1,"type":"STRING","value":"api_key"},"2":{"slot":2,"type":"STRING","value":"${api_key}"}}},"27":{"slot":27,"values":{"1":{"slot":1,"type":"STRING","value":"market"},"2":{"slot":2,"type":"STRING","value":"${market}"}}},"28":{"slot":28,"values":{"1":{"slot":1,"type":"STRING","value":"safe_search"},"2":{"slot":2,"type":"STRING","value":"${safe_search}"}}},"29":{"slot":29,"values":{"1":{"slot":1,"type":"STRING","value":"search_terms"},"2":{"slot":2,"type":"STRING","value":"${search_terms}"}}},"30":{"slot":30,"values":{"1":{"slot":1,"type":"STRING","value":"truncate_tables"},"2":{"slot":2,"type":"STRING","value":"${truncate_tables}"}}}},"visible":true,"status":{},"columnNames":[]},"5":{"slot":5,"name":"Set Grid Variables","elements":{"1":{"slot":1,"elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"GRID","value":"advanced_connection_options"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"GRID","value":"parameter"},"2":{"slot":2,"type":"GRID","value":"value"}}}},"values":{"1":{"slot":1,"type":"STRING","value":"advanced_connection_options"},"2":{"slot":2,"type":"STRING","value":"grid"}}},"2":{"slot":2,"elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"GRID","value":"source_list"}}},"2":{"slot":2,"values":{"1":{"slot":1,"type":"GRID","value":"table_name"},"2":{"slot":2,"type":"GRID","value":"column_name"},"3":{"slot":3,"type":"GRID","value":"incremental_column"},"4":{"slot":4,"type":"GRID","value":"key_column"}}}},"values":{"1":{"slot":1,"type":"STRING","value":"source_list"},"2":{"slot":2,"type":"STRING","value":"grid"}}}},"visible":true,"status":{},"columnNames":[]},"6":{"slot":6,"name":"Melt Name","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Bing Search Incremental Load"}}}},"visible":true,"status":{},"columnNames":[]},"7":{"slot":7,"name":"Melt Package","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"Matillion.Incremental"}}}},"visible":true,"status":{},"columnNames":[]},"8":{"slot":8,"name":"Melt Revision","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"INTEGER","value":"1"}}}},"visible":true,"status":{},"columnNames":[]},"9":{"slot":9,"name":"Melt Load Type","elements":{"1":{"slot":1,"values":{"1":{"slot":1,"type":"STRING","value":"DYNAMIC"}}}},"visible":true,"status":{},"columnNames":[]}},"validationStatus":{},"status":{},"expectedFailure":null,"activationStatus":"ENABLED","outputIterationConnectorIDs":[],"inputIterationConnectorIDs":[]}},"successConnectors":{"25570":{"id":25570,"sourceID":25568,"targetID":25567}},"failureConnectors":{"25573":{"id":25573,"sourceID":25569,"targetID":25566}},"unconditionalConnectors":{"25571":{"id":25571,"sourceID":25565,"targetID":25569},"25572":{"id":25572,"sourceID":25569,"targetID":25568}},"trueConnectors":{},"falseConnectors":{},"iterationConnectors":{},"noteConnectors":{},"canUndo":false,"undoCommand":"","undoCreated":-1,"canRedo":false,"redoCommand":"","redoCreated":-1,"notes":{"25564":{"id":25564,"x":-555,"y":-236,"width":470,"height":312,"text":"**Execute ETL Flow for specific source table**\n\nThis job wraps our Stage 3 job (Stage Object). This is done so that we can manage some exported variables from the Stage 3. If we were directly iterating over (Stage Object) we would not be able to handle these exported variables.\n\n- (__Stage Object__) Passes in the parameters and their values into the Stage - 3 job. Regardless of the result, the __metadata_stage_table_name__ and __stage_table_name__ are exported - this is to ensure these tables are dropped if the process fails before it has time to clean up these staging tables (when a run succeeds, these staging tables will no longer exist).\n\n- (__Drop Stage Tables__) Passes in the exported table names to be deleted. This flow is executed regardless of the result of Stage Object.","colour":"e6e63c"}},"variables":{"market":{"definition":{"name":"market","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PUBLIC"},"value":""},"api_key":{"definition":{"name":"api_key","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PUBLIC"},"value":null},"auto_debug":{"definition":{"name":"auto_debug","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"dist_style":{"definition":{"name":"dist_style","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"kms_key_id":{"definition":{"name":"kms_key_id","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"table_name":{"definition":{"name":"table_name","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"log_metrics":{"definition":{"name":"log_metrics","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"safe_search":{"definition":{"name":"safe_search","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PUBLIC"},"value":""},"inc_col_type":{"definition":{"name":"inc_col_type","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":""},"search_terms":{"definition":{"name":"search_terms","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PUBLIC"},"value":""},"stage_prefix":{"definition":{"name":"stage_prefix","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"stage_schema":{"definition":{"name":"stage_schema","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"[Environment Default]"},"staging_type":{"definition":{"name":"staging_type","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"stage_dataset":{"definition":{"name":"stage_dataset","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"[Environment Default]"},"stage_project":{"definition":{"name":"stage_project","type":"TEXT","scope":"BRANCH","description":"   ","visibility":"PUBLIC"},"value":"[Environment Default]"},"target_prefix":{"definition":{"name":"target_prefix","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":""},"target_schema":{"definition":{"name":"target_schema","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"blob_container":{"definition":{"name":"blob_container","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"s3_bucket_name":{"definition":{"name":"s3_bucket_name","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"stage_database":{"definition":{"name":"stage_database","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"[Environment Default]"},"target_dataset":{"definition":{"name":"target_dataset","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"target_project":{"definition":{"name":"target_project","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"stage_warehouse":{"definition":{"name":"stage_warehouse","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":"[Environment Default]"},"storage_account":{"definition":{"name":"storage_account","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"target_database":{"definition":{"name":"target_database","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"target_platform":{"definition":{"name":"target_platform","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"truncate_tables":{"definition":{"name":"truncate_tables","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PUBLIC"},"value":""},"auto_debug_level":{"definition":{"name":"auto_debug_level","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"stage_table_name":{"definition":{"name":"stage_table_name","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":""},"target_warehouse":{"definition":{"name":"target_warehouse","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"encryption_method":{"definition":{"name":"encryption_method","type":"TEXT","scope":"BRANCH","description":"","visibility":"PUBLIC"},"value":null},"cloud_storage_area":{"definition":{"name":"cloud_storage_area","type":"TEXT","scope":"BRANCH","description":"The cloud storage area name that should be used when loading bq tables.","visibility":"PUBLIC"},"value":null},"metadata_stage_table_name":{"definition":{"name":"metadata_stage_table_name","type":"TEXT","scope":"TASKBATCH","description":"","visibility":"PRIVATE"},"value":""}},"grids":{"source_list":{"definition":{"name":"source_list","scope":"BRANCH","definitions":[{"name":"table_name","type":"TEXT"},{"name":"column_name","type":"TEXT"},{"name":"incremental_column","type":"DECIMAL"},{"name":"key_column","type":"DECIMAL"}],"description":"","visibility":"PUBLIC"},"values":[]},"advanced_connection_options":{"definition":{"name":"advanced_connection_options","scope":"BRANCH","definitions":[{"name":"parameter","type":"TEXT"},{"name":"value","type":"TEXT"}],"description":"","visibility":"PUBLIC"},"values":[]}}}},"transformationJobs":{}}],"version":"EMD_21347","environment":"snowflake"}