GraphQL Custom Connector Example

Examples of custom Script Paging using the Rick and Morty GraphQL API.

This Data Productivity Cloud Custom Connector extracts and loads data from the Rick and Morty GraphQL API. The paging method requires customized paging, and the connector demonstrates how to do this for both POST and GET requests.

Once installed, you can use it in an Orchestration pipeline:

Endpoints

Both these endpoints extract the same data, demonstrating the two different approaches that are possible with this GraphQL API.

• Rick and Morty POST
• Rick and Morty GET

Integrating semi-structured data using vertical decomposition

In the screenshot above, you can see the data is extracted into a table named stg_rm_data.

Using Snowflake as an example, the SQL query to perform vertical decomposition requires a LATERAL FLATTEN clause:

SELECT f.Value:"id" :: INT AS Id, f.Value:"name" :: VARCHAR AS Name, f.Value:"type" :: VARCHAR AS Type
FROM "stg_rm_data" p,
LATERAL FLATTEN(input => p."data":"characters":"results") f;

In Data Productivity Cloud this can be done using an Extract Nested Data component in a Transformation pipeline:

Once the data has been converted to a relational format with rows and columns, it’s easy to work with further - for example to summarize, or to integrate with other data sets.

Downloads

Licensed under: Matillion Free Subscription License

• Download Rick-and-Morty-GraphQL.json
  • Type: Custom Connector

Installation Instructions