Shashank Mishra

In this talk, I will dive into a real-time data pipeline solution I worked on for the Travel & Hospitality domain, powered by Apache Flink Stateful Functions. This solution efficiently handled over 800,000 streaming events per day, addressing critical use cases. I will share insights into the challenges faced, the strategies used to overcome them, and how we successfully scaled the pipeline to meet demanding business requirements. For the second part of the session, I will explore the hot topic: Will Low-Code ETL Replace Data Engineers? I firmly believe that Low-Code ETL is not here to replace data engineers but to accelerate the development process. However, for this to succeed, ETL platforms must empower data engineers with the freedom and control to customize and adapt the platform to their specific needs. I’ll discuss examples like Prophecy, a Low-Code ETL platform that strikes a balance by acting as an IDE, generating open-source code, and keeping implementation transparent. This approach ensures that while development is expedited, the power remains firmly in the hands of data engineers.