Sign Up For Our Second Low-Key Data Conference15 speakers. 4 hours. Virtual event. No frills. Join us on 7/12/23 from 1-5pm EST - Click here to sign up!.
Fivetran is purpose-built for data teams that need a no-code SaaS ETL solution. Airbyte is a better fit for data engineers that want an open-source framework to customize integrations in-house. Portable fills the gaps - offering the simplicity of Fivetran for custom ETL connectors.
Portable is the first ELT platform to build connectors on-demand for data teams. We believe companies should have data from every business application at their fingertips - all with no code. We have more no-code ELT connectors than both Fivetran and Airbyte (300+ and counting).
|Notable Clients||Snowflake, Morgan Stanley, Intercom||Cart.com, AMARO, Prefect||EBTH, Orva, Droplette, Beacons|
|Custom Connectors||Build-Your-Own Cloud Functions||Build-Your-Own Connector Development Kit (CDK)||Free On-Demand Custom Integration Development|
Data engineers are automating data pipelines with ETL tools to integrate data from disparate business systems into their cloud data warehouse for various analytics use cases.
You've probably narrowed it down to two of the most popular data platforms -- Fivetran and Airbyte.
In this comparison, you'll learn the pros and cons of the two ETL platforms.
We'll outline the functionality and the pricing models for each ETL solution and even offer a simple framework to understand when (and how) to use each platform for your data integration strategy.
The two most common use cases for data integration tools are: data automation & data analytics.
Data integration solutions make it easy to extract data from APIs and databases, to then load the data into your data warehouse for business intelligence. Other companies in the space include Portable, Stitch Data, Meltano, and Matillion.
When using data for analytics use cases, data engineers typically utilize an ETL tool to load data from SaaS applications into Snowflake, Google BigQuery, Amazon Redshift, PostgreSQL or SQL Server.
From there, teams can build dashboards for better corporate decision making and business strategy.
On the other hand, automation replaces manual tasks with real-time, automated workflows that syncs data from one data source to another business application in a low-code or no-code manner. Reverse ETL solutions empower data teams with tooling to automate workflows directly from the data warehouse.
How should you get data integrated from business applications into a data warehouse or data lake for analytics? We'll examine how this can be done with Airbyte or Fivetran.
ETL platforms like Fivetran and Airbyte help business intelligence teams in three ways:
1. Self-service data extraction. With hundreds of pre-built data connectors to common SaaS applications and databases, both platforms make data replication simple.
2. Ready-to-query schemas for orchestration and data transformation. By syncing data into the warehouse, no-code solutions can be integrated with open source orchestration and transformation tools like Airflow and DBT to build data models, execute DAGs, and orchestrate complex pipelines.
3. Low maintenance data pipelines. Leveraging an out-of-the-box solution allows your data engineers to analyze data without having to worry about rate limits, errors, hardware failures, and scaling issues. Vendors like Fivetran and Airbyte offer a simple, low maintenance solution.
Fivetran was founded in 2012 as an ELT tool for hyper-growth companies.
Fivetran has a strong reputation for maintenance and support for databases (change data capture) and the largest business applications. When teams get started, they can't go wrong with Fivetran for scalable data pipelines.
Supporting a native integration with DBT, Fivetran can help turn raw data into insights with native transformation. For connectors that have complex data models, these off-the-shelf queries can save analytics teams significant work.
Support for enterprise data sources. Oracle, SAP, and Workday are a core focus of Fivetran's business model as they expand into enterprise clients. These connectors are extremely complicated and are difficult to find reliable solutions elsewhere. Fivetran spent $700m to acquire HVR to deepen these capabilities.
Cost is a big downside of the Fivetran platform. It's primarily why data teams seek out Fivetran alternatives. With pricing based on Monthly Active Rows, it's easy to end up in a scenario where a data source syncing large data volumes can cost a significant amount of money.
Not only are prices high, but they are very difficult to predict and forecast, adding additional stress to data teams, and additional scrutiny from finance stakeholders.
Fivetran takes months or years to build new connectors. And in many scenarios, they do not plan to build connectors at all.
In a scenario where you need a connector and Fivetran doesn't support it out-of-the-box, you aren't offered great options. Fivetran asks clients to script their own solutions using cloud functions.
Airbyte is a newer startup that was founded in 2020 as an open-source ELT tool for the modern data stack.
Airbyte offers an open-source platform for data integration. Unlike many other ELT vendors, clients can either use Airbyte Cloud (a SaaS solution) or an open-source version of Airbyte deployed on its own.
The platform is tailored to engineers. As an engineering team, if you want to build your own data integrations, Airbyte offers a CDK to accelerate development vs. writing code from scratch.
For destinations, Airbyte supports over 20 destinations (Snowflake, BigQuery, AWS Datalake, etc.). When you need information loaded into a bespoke platform, Airbyte can be a great solution when other ELT tools don't support your destination.
For long-tail integrations that you build on your own with the Airbyte CDK, your team is on the hook for maintenance, support, and fixing things when they break. Unlike Portable, writing your own integration with Airbyte can lead to ongoing maintenance efforts.
The Airbyte pricing model isn't straightforward. With a credit-based, volume-based consumption model, it can be difficult to predict usage and to mitigate costs as your data volumes increase.
Many of Airbyte's connectors are still currently in alpha, or not yet production-ready. In many scenarios, users can be on the hook for maintenance, flagging issues, and working with the Airbyte team on resolutions.
Now that we've outlined the pros and cons of the two platforms, let's outline Fivetran as an Airbyte alternative, and Airbyte as a Fivetran alternative.
It is important to dig into the true capabilities of the platforms we are considering. Let's dive into the features, functionality and pricing of the two platforms.
One of the most important criteria for selecting an ETL tool is whether or not the product supports the long-tail data sources you need.
Most vendors don't build many new data sources each year, so when you consider the offering, you're really purchasing access to the connectors they already have in their catalog. Breadth of connectors is a strong proxy for a vendor's ability to help your analytics team centralize data.
Fivetran's connector catalog is just over 160 data sources.
These connectors are typically the largest databases, file-based sources, and business applications that are helpful when your team is first standing up your data stack.
Airbyte's open source product includes 170+ data sources.
The cloud offering is newer and has some restrictions on what sources can be leveraged within the product; however, it's reasonable to expect that all 170+ would be available in the cloud product shortly.
When your team needs a new connector, you NEED the connector.
It's important to understand how both data integration platforms will help in these scenarios. Do they ask you to write code? To maintain the connector? To fix things when they break?
For custom connectors, Fivetran asks clients to use cloud functions to build their own connectors and maintain them when things break.
As part of the development process, users need to read API documentation, deploy a serverless cloud function, make sure things work, and then maintain the cloud function if things ever break. This can be a significant amount of work, and is a common scenario when data teams take a look at the Portable connector catalog for another approach.
Airbyte is open source (similar to Singer and Meltano), so engineers can develop their own custom integrations using Python or another programming language.
In these scenarios, developers still need to read the source documentation, learn the Airbyte protocol / CDK framework, and set up the integration. For custom connectors, the user that built the connectors has to maintain their own connector and troubleshoot issues as they arise.
Data integrations are living, breathing organisms. They evolve, they break, and they cause chaos with your queries and dashboards when they do.
It's critical to understand how both ETL vendors will support you when things go wrong, and what functionality each platform has in place for alerting, monitoring, and connector maintenance.
Fivetran offers a ticketing system for clients to flag issues when things go wrong.
Because Fivetran has a large client base, issues are typically picked up quickly, but support can feel impersonal at times.
Airbyte has over 12,000 users of the open source product, so users are asked to leverage the documentation, GitHub project board, community Slack and Discourse communities to troubleshoot issues.
Finally, let’s compare the pricing of Fivetran vs. Airbyte. There are both similarities and differences to be aware of.
Fivetran charges on Monthly Active Rows for data pipelines.
Airbyte has a [credit based pricing model].
Choosing an ETL solution for your Modern Data Stack really depends on your business use case and data management architecture.
We've outlined the pros and cons of both Fivetran and Airbyte to help frame out the scenarios in which each solution makes sense.
At Portable, we focus our efforts on a customer-first culture, a try-before-you-buy business model, and hands on support when things go wrong.
There's no downside to exploring our connector catalog, or even requesting the connector that's at the top of your backlog.
Want to learn more? Book time for a discussion or a demo directly on my calendar