Whirl Adds Local AirFlow Development Technique
Written by Kay Ewbank   
Tuesday, 23 April 2019

A test environment for Apache Airflow that lets you carry out local development and testing of Apache Airflow workflows has been made available as open source. Whirl uses Docker and Docker compose to start up Apache Airflow and the other components used in your workflow.

Whirl was developed at GoDataDriven as part of its Open Source Initiative.

whirl

Apache Airflow is a workflow automation and scheduling system that you can use to set up and manage data pipelines. It uses workflows made of directed acyclic graphs (DAGs) of tasks. Each task produces some output that is then used as the input to another task.

The idea of Whirl is to make it easy to run and develop Airflow workflows on your local machine. This gives you rapid feedback about whether the changes you made to your DAG work. The developers suggest you think of it as your integration test environment for developing Airflow DAGs.

The way Whirl works is that it uses Docker containers to start up Apache Airflow and the other components that your workflow relies on. This gives you a copy of your production environment that runs on your local machine.

This allows you to run your DAG locally all the way through using the actual code you would use on the production environment. You can try things out isolated from the actual workflows and see how it works with test data.  Whirl connects the code of your DAG and your test data to the Apache Airflow container that it spins up. By using volume mounts, you can make changes to your code in your IDE and immediately see the effect in the running Apache Airflow UI on your machine. This even works with custom Python modules that you are developing and using in your DAGs.

whirl 

 

More Information

Whirl On GitHub

Related Articles

Twitter Heron Donated To Apache

Gradle 2.0 Released

Apache Flink 1.5.0 Adds Support For Broadcast State

Flink Gets Event-time Streaming

Flink Reaches Top Level Status

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on, Twitter, Facebook or Linkedin.

Banner


Supreme Court Asks For Government Help In Oracle v Google
01/05/2019

The U.S. Supreme Court has again requested the views of the U.S.Solicitor General with regard to Google's petition asking for a judicial review of the copyright issues stemming from the Oracle v Googl [ ... ]



How Do Open Source Deep Learning Frameworks Stack Up?
24/04/2019

As the popularity of deep learning increases, finding application in all sorts of cases, so does the popularity of the various DL frameworks and libraries, making it difficult to choose between them.  [ ... ]


More News

Python

 



 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Tuesday, 23 April 2019 )