Home

Building a Multi-stage Data Streaming Pipeline with Elixir Broadway Kafka

example app

• → spc_data

Both Kafka and the application setup/startup instructions are described on the spc_data example repo. In addition, I took a lot of inspiration from the Dashbit Broadway Bikeshare App. To the Dashbit developer and creator of Floki Philip Sampaio, many 🙏🏼s.

The most pressing motivation for this note was to distill down initial steps to build out an event processing feature for a recent project. The data pipeline required support to stream a high-volume of data from various equipment sources. Subsequently, in the data processing stage, calculations such as Standard Deviation are then provided to downstream sources. All of this, in as “real-time” as possible so as to proactively correct and act upon errors.

Links, Resources & Citations

• → Dashbit Broadway Bikeshare App
• → Andrew Pett’s setting-kafkas-pace-with-broadway - ..nice example of adding a buffer
• → SPC & Standard Deviation
• → :brod - dealing with {:error, {:producer_not_found, "spc-data"}} blocker