You leverage Fivetran for ETL into your data warehouse. The product is great, but unfortunately, they don't support the connector you need.
Fivetran function connectors are data pipelines that you create for Fivetran to sync data on your behalf within a broader ETL framework.
There are a few steps in the process of building a new connector in this manner.
You can dig deeper into the details, but the important thing to note is that as a developer, Fivetran has provided you with a template to organize your pipeline logic and metadata.
Yes, for custom connectors, you have to read API documentation.
Unlike the Fivetran connectors in their directory, custom connectors are managed by clients. So you need to find the API documentation and then navigate the nuances of authentication, request and response structure, and pagination.
If you are a data engineer who is well versed in reading docs, handling HTTP codes, version control with GitHub, managing config files, identifying primary keys, processing JSON, and using cloud solutions like GCP or AWS, you’re probably set up to get started.
If this is new to you, here is an example of API documentation for the common CRM system HubSpot.
It can take some time to learn how to navigate API documentation like HubSpot's documentation above, but at the highest level, you need to understand the following concepts:
Yes, you have to write code to build a custom integration in the platform.
That being said, you still need to create an account with AWS, Azure, or Google Cloud, you need to get the data integration stood up, and then you need to convert your understanding of the API documentation into code that can run within a function.
Your code needs to:
You need to write custom logic to understand the errors and address them programmatically.
Here is a list of the common HTTP error codes that you should consider.
While Fivetran doesn't help you to address the errors, or retry requests, they do offer the ability to expose errors into your Fivetran dashboard if you decide to add this logic to your function.
Yes. You have to pay Fivetran to move your data, and you have to pay the cloud charges associated with your custom ETL pipeline.
There are two major considerations around costs.
First, how much data are you processing? This matters not only because Fivetran offers pricing on monthly active rows, but also because data volume typically correlates closely with cloud compute costs for cloud functions.
Second, how efficient is your code to process the data? If your code is inefficient, it can cause your cloud bills to explode quickly at high data volumes. Even efficient code can lead to expensive cloud bills, but inefficiency can cause chaos.
It's always best practice to add maximum thresholds for billing to your cloud console and other safeguards. Why? There are certain death spirals around serverless functions that you don't want to end up in...
Yes. You have ONE other great option for custom integrations.
Clients with Fivetran in their data stack, and even resellers, commonly encounter use cases for custom integrations. Many of these companies trust Portable to build and maintain long-tail ETL connectors when Fivetran won't support them.
As a cloud-hosted SaaS solution just like Fivetran. Portable already supports 275+ hard-to-find data sources and loads the data into ready-to-query destination schemas.
Portable builds ETL connectors on-demand for clients. Development is free, and it's easy to sign up and manually sync data. Pricing is straightforward, and the Portable support team is on call if there are ever problems.
Want to spent your time writing SQL, managing models in DBT, building dashboards, and creating value from data?
We work with plenty of happy Fivetran clients. You'll be in good hands.