Chris Engelbert

Bio

Christoph

Christoph Engelbert is a developer by heart, with strong bonds to the open source world. As a seasoned speaker on international conferences, he loves to share his experience and ideas, especially in the areas of scalable system architectures and back-end technologies, as well as all things programming languages.

Twitter: @noctarius2k

Data Pipeline Plumbing

For the last two decades, the amount of data we store, process, and analyze is ever growing. The last decade shows a higher focus on immediate feedback loop data pipeline, using technologies such as Complex Event Processing (CEP), Stream Processing, and Change Data Capture (CDC). Services such as Kafka or NATS are to be found in almost every new system (at least to some extent).

To build a data pipeline, the number of technologies, frameworks, and platforms are endless. Getting the initial grasp of it all was much harder than expected, but I learned a lot of things along the way. Let’s embark together on a retrospective of building a real-time analytics system for a farming IoT startup.