Need a Fivetran Custom Connector? We’ll Build It For You

Ethan
CEO, Portable

You leverage Fivetran for ETL into your data warehouse. The product is great, but unfortunately, they don't support the connector you need.

If you need a custom connector in Fivetran, what are your options?

  1. Script an ETL data pipeline using a function connector.
  2. Find a consultant that will handle replication for you. Luckily, we've pulled together a list of 175+ data consultants you can reach out to.
  3. Write code from scratch with your favorite programming language (Python is common). I'd recommend an orchestration tool like Airflow to help with scheduling.
  4. Use an open-source framework like Singer, Meltano, or Airbyte.
  5. Search for other ETL / ELT vendors to see if they have a prebuilt connector you can use.
  6. Get started for free with Portable today.

How do Fivetran Cloud Function connectors work?

Fivetran function connectors are data pipelines that you create for Fivetran to sync data on your behalf within a broader ETL framework.

There are a few steps in the process of building a new connector in this manner.

  1. Authorize
  2. Initial Sync
  3. Parse
  4. Process
  5. Load
  6. Update

You can dig deeper into the details, but the important thing to note is that as a developer, Fivetran has provided you with a template to organize your pipeline logic and metadata.

Do you need to read API documentation?

Yes, for custom connectors, you have to read API documentation.

Unlike the Fivetran connectors in their directory, custom connectors are managed by clients. So you need to find the API documentation and then navigate the nuances of authentication, request and response structure, and pagination.

If you are a data engineer who is well versed in reading docs, handling HTTP codes, version control with GitHub, managing config files, identifying primary keys, processing JSON, and using cloud solutions like GCP or AWS, you’re probably set up to get started.

If this is new to you, here is an example of API documentation for the common CRM system HubSpot.

How do you navigate API documentation to build a Fivetran custom connector?

It can take some time to learn how to navigate API documentation like HubSpot's documentation above, but at the highest level, you need to understand the following concepts:

  • Authentication - How do you authenticate with the API so they allow you to make requests?
  • Requests - Identify the endpoints you plan to make calls to, and research the format of how you need to make those requests. Are they REST APIs? GraphQL? Etc.
  • Responses - In what format is data returned from the API? How will your application process and store those responses?
  • Pagination - If you need to request multiple pages of data (i.e. the first 100 records, then the next 100), how do you do so?
  • Error handling - What could go wrong, and how will your system respond?
  • Rate limits - Many APIs don't want to be misused, so they restrict how frequently you can make requests. Are there rate limits in place, and how will your system respect them?

Once you understand how the API works, do you have to write code?

Yes, you have to write code to build a custom integration in the platform.

Luckily, technologies like AWS Lambda, Azure Functions, and Google Cloud Functions remove the complexity of managing infrastructure, which can be a massive headache when building data infrastructure.

That being said, you still need to create an account with AWS, Azure, or Google Cloud, you need to get the data integration stood up, and then you need to convert your understanding of the API documentation into code that can run within a function.

What does your code need to do to move data?

Your code needs to:

  1. Receive requests from Fivetran to initiate a sync
  2. Authenticate with an API key or other mechanism and make requests to the source system
  3. Receive and process the response from the API
  4. Return the response to Fivetran to load into your warehouse (Snowflake, BigQuery, Amazon Redshift, etc.)

How do you handle errors from the upstream API?

You need to write custom logic to understand the errors and address them programmatically.

Here is a list of the common HTTP error codes that you should consider.

While Fivetran doesn't help you to address the errors, or retry requests, they do offer the ability to expose errors into your Fivetran dashboard if you decide to add this logic to your function.

Do you have to pay for the custom integration?

Yes. You have to pay Fivetran to move your data, and you have to pay the cloud charges associated with your custom ETL pipeline.

There are two major considerations around costs.

First, how much data are you processing? This matters not only because Fivetran offers pricing on monthly active rows, but also because data volume typically correlates closely with cloud compute costs for cloud functions.

Second, how efficient is your code to process the data? If your code is inefficient, it can cause your cloud bills to explode quickly at high data volumes. Even efficient code can lead to expensive cloud bills, but inefficiency can cause chaos.

It's always best practice to add maximum thresholds for billing to your cloud console and other safeguards. Why? There are certain death spirals around serverless functions that you don't want to end up in...

Do you have any other options if Fivetran doesn't support the data connectors you need?

Yes. You have ONE other great option for custom integrations.

Clients with Fivetran in their data stack, and even resellers, commonly encounter use cases for custom integrations. Many of these companies trust Portable to build and maintain long-tail ETL connectors when Fivetran won't support them.

As a cloud-hosted SaaS solution just like Fivetran. Portable already supports 275+ hard-to-find data sources and loads the data into ready-to-query destination schemas.

Portable builds ETL connectors on-demand for clients. Development is free, and it's easy to sign up and manually sync data. Pricing is straightforward, and the Portable support team is on call if there are ever problems.

Want to spent your time writing SQL, managing models in DBT, building dashboards, and creating value from data?

Give Portable a try! Here’s how to get started.

Sign up, request a connector, or schedule time with me to learn more!

We work with plenty of happy Fivetran clients. You'll be in good hands.