Data engineers play a pivotal role in driving contextualized intelligence. They must ensure that data is enriched with the proper context to deliver meaningful insights. This tech talk will explore the practical steps and tools data engineers can use to build a contextual layer, ensuring AI models can deliver strategic insights based on relevant business contexts.

In this talk, I will dive into a real-time data pipeline solution I worked on for the Travel & Hospitality domain, powered by Apache Flink Stateful Functions. This solution efficiently handled over 800,000 streaming events per day, addressing critical use cases. I will share insights into the challenges faced, the strategies used to overcome them, and how we successfully scaled the pipeline to meet demanding business requirements. For the second part of the session, I will explore the hot topic: Will Low-Code ETL Replace Data Engineers? I firmly believe that Low-Code ETL is not here to replace data engineers but to accelerate the development process. However, for this to succeed, ETL platforms must empower data engineers with the freedom and control to customize and adapt the platform to their specific needs. I’ll discuss examples like Prophecy, a Low-Code ETL platform that strikes a balance by acting as an IDE, generating open-source code, and keeping implementation transparent. This approach ensures that while development is expedited, the power remains firmly in the hands of data engineers.

AI is reshaping the landscape of data engineering in transformative ways. One key trend is the rise of AI-powered data pipelines, where machine learning automates tasks like data cleaning, transformation, and feature engineering, significantly accelerating development cycles. Alongside this, AI-driven data observability tools are improving data reliability by detecting anomalies and quality issues in real time. Intelligent data catalogs are also evolving, using AI to automatically generate metadata, track lineage, and simplify data discovery. Natural language interfaces, such as Text-to-SQL systems powered by large language models, are making it easier for non-technical users to query databases using plain English. ETL and ELT processes are becoming smarter too, with AI automating schema mapping and transformation logic, reducing the need for manual coding. In parallel, synthetic data generation is gaining traction, enabling the creation of realistic datasets for testing, training, and privacy-sensitive use cases. Real-time data processing is becoming more intelligent as AI models integrate directly into streaming frameworks like Kafka and Spark, supporting use cases like fraud detection and personalization. Governance is also being augmented by AI, with tools now capable of automatically detecting sensitive data like PII and enforcing compliance with regulations.

The Unified Data platform is designed to address key limitations in data management for machine learning (ML) model development, training, and testing. Its primary focus is to improve turnaround time (TAT), enhance experimentation velocity, and provide a centralized solution for data needs.

Discover the engineering innovations behind KnowBe4's agentic platform, which utilizes Data and Generative AI to empower cybersecurity defense. Learn how KnowBe4 developed AIDA (Artificial Intelligence Defense Agents) to significantly reduce administrative overhead for customers by automating the configuration of KnowBe4 products. AIDA intelligently understands specific customer requirements based on user behavior, leading to automated risk reduction. This session will delve into the cutting-edge technologies that power this feature, including the latest advancements in Artificial Intelligence, Datamesh architecture, and big data engineering tools. We will explore how these elements are integrated to deliver current GenAI capabilities and lay the foundation for future intelligent features aimed at proactively mitigating human-related cyber risks.

Building a future-forward health insurance data platform focuses on leveraging emerging technologies like AI, machine learning, and blockchain to revolutionize data management in the insurance sector. By integrating diverse data sources such as medical records, wearable devices, and claims data, the platform enables personalized policies, faster claims processing, and enhanced risk management. This approach not only increases operational efficiency but also improves customer satisfaction by providing tailored coverage and real-time insights

This session explores how AI agents are transforming data engineering by automating complex workflows such as data ingestion, transformation, and pipeline orchestration. With real-time analytics and intelligent decision-making becoming critical, AI-driven automation is enabling greater efficiency, scalability, and accuracy in data processes. From automated anomaly detection to schema evolution and performance optimization, discover practical use cases that showcase the power of AI in simplifying and future-proofing data engineering strategies. Ideal for data engineers, architects, and AI enthusiasts, this talk offers insights into leveraging AI agents to reduce operational overhead and stay ahead in the era of intelligent automation.

In today’s data-driven financial ecosystem, speed, scale, and accuracy are non-negotiable. This session explores the key principles and real-world practices behind building scalable data pipelines that power meaningful financial insights. From ingesting high-volume transactional data to ensuring real-time processing and analytics, we'll dive into architectural patterns, technology choices, and performance considerations that enable organizations to unlock value from their data. Whether you're modernizing legacy systems or designing from the ground up, this talk will equip you with practical takeaways to build resilient, future-ready data pipelines tailored for the dynamic demands of the financial domain.

This session is to outline strategic data engineering practices that drive innovation and business growth. Emphasizing scalability, data quality, and operational efficiency. The session highlights how modern data platforms, automation, and governance empower data-driven decision-making across the organization.

In today’s data-driven world, ensuring resilience in data architectures is crucial for maintaining seamless operations, minimizing downtime, and enabling real-time decision-making. This session explores the key principles of designing fault-tolerant, scalable, and adaptive data ecosystems that can withstand failures and disruptions. Key focus areas include strategies for high availability, redundancy, disaster recovery planning, and leveraging distributed computing and cloud-native technologies to enhance resilience. Real-world insights will showcase how resilient data architectures can optimize performance, improve reliability, and support business continuity in an evolving technological landscape

Real-time data pipelines enable organizations to ingest, process, and act on streaming data with minimal latency, ensuring they can make informed decisions instantly. This session will explore the core components of real-time data pipelines, including event-driven architectures, stream processing frameworks, and scalable data ingestion technologies. We’ll discuss real-world applications such as fraud detection, predictive analytics, IoT monitoring, and personalized customer experiences. By the end of the session, attendees will gain a deeper understanding of how real-time data pipelines drive agility, enhance operational efficiency, and unlock new business opportunities. Whether you're a data engineer, architect, or business leader, this talk will equip you with the knowledge to harness the power of real-time data for smarter decision-making.

As organizations increasingly rely on complex data ecosystems, maintaining data quality and trust has become paramount to business success. This discussion explores the critical components and best practices for implementing robust data quality frameworks within modern data architectures. We examine how organizations can establish automated testing, monitoring, and governance processes to ensure data reliability across the entire pipeline - from ingestion to consumption. Special attention is given to emerging tools and methodologies that enable continuous data quality assessment while scaling with growing data volumes and complexity.

In the competitive OTT landscape, delivering an exceptional video Quality of Experience (QoE) is crucial for user satisfaction and retention. This session will explore how data-driven approaches can transform the way QoE is monitored, analyzed, and optimized. The talk will delve into the key metrics that define QoE, including startup time, buffering frequency, bitrate quality, and playback interruptions. By leveraging real-time analytics, machine learning, and predictive modeling, OTT platforms can proactively identify and resolve issues before they impact the user experience. Attendees will gain actionable insights into: Designing scalable data pipelines for QoE monitoring. Using predictive analytics to anticipate and mitigate streaming disruptions. Optimizing encoding and delivery processes to reduce latency and improve video quality. This session is ideal for data engineers, OTT platform architects, and video delivery professionals looking to elevate their platform’s user experience through the power of data.