Are you trying to build a custom ETL connector? It's important to make sure you understand your options.
There are a handful of different ways to build a custom ETL connector.
Option #1 - Write code - and manage infrastructure - from scratch.
You can always develop custom ETL connectors from scratch. Is it worthwhile? Very rarely. In most scenarios, someone else has already written a framework, created a scaffold, or is willing to take on the development work so you don't have to.
Option #2 - Use an open source framework.
Open source frameworks are great at providing structure when you want to build your own ETL connector. They are particularly useful for companies in regulated industries like Healthcare and Financial Services, where you have to build in-house (vs. using a SaaS solution), but when you need a starting point.
Option #3 - Hire a consultant.
Data consultants are great. Not only can they help you build custom ETL connectors, but they can also help create data models, develop dashboards, and assist in architecting your data stack. The problem with using a consultant for custom ETL connectors is that they are 1) expensive, and 2) ephemeral. This means if you move on, or stop paying them, you need to rebuild your custom ETL connector, or find someone to maintain it.
Option #4 - Use serverless infrastructure like cloud functions.
This doesn't really solve the entire problem. It's just a solution for not managing infrastructure. Serverless technology is simple, but you also need secure authentication, monitoring, alerting, retry logic, pagination logic, and more. Cloud functions can help if you have to build in-house, and don't want to manage infrastructure.
Option #5 - Use Airflow.
Airflow provides structure to an otherwise unstructured problem. Scheduling, orchestration, and stringing together requests, responses, and downstream actions can be handled with Airflow, but you still need actual integration logic. Airflow is simply a piece of the puzzle.
Option #6 - Just use Portable.
Portable specializes in building custom ETL connectors for clients. If you need a random, bespoke, long-tail system connected to your warehouse, just reach out. We build API to warehouse connectors on-demand, and can turn around production-grade SaaS hosted ETL connectors in a matter of hours or days. We handle development, monitoring, maintenance, alerting, troubleshooting, and support. If something goes wrong, we're on call, so you can sleep well.