Home

Building a Multi-stage Data Ingestion Pipeline with Elixir Broadway

example app

• → hottest_city

First, I want to give thanks to Alex Koutmos for his excellent AppSignal blog post on this topic. Please check it out. Also, his and Hugo Baraúna’s book, Elixir Patterns is in my top three favorite books on Elixir. The book tackles Elixir concepts that I have yet to learn about anywhere else and has leveled up my understanding of the OTP ecosystem immensely.

As to this note on Broadway, looking at the very short video, reading through the example app code and reading the Broadway Docs will hopefully provide much clarity.

In the basic example app, there are a few modules that form the core data pipeline functionality

1. Producer → Acts as a producer/client that provides the app with a map of cities of which to ingest and make an external api calls with.
2. Processor → The main process that starts the Broadway process, handles and acknowledges GenStage messages. 3.TemperatureAgent → Holds state of the data returned from the external api call.

There are many additional features such as adding a buffer, however, this repo was intentionally kept simple to serve as a simple starting point when setting up Broadway. See this article for a nice example and explanation of adding a buffer to a Kafka pipeline.

Links, Resources & Citations

• → Alex Koutmos’s How to use Broadway
• → Broadway Docs