Day 1 | HALL 2 - Practical Insights and Best Practices

This topic explores how data engineering is evolving to support modern AI and analytics. It emphasizes the importance of new data engineering techniques in building scalable and efficient systems on cloud data platforms. A key focus is on using Iceberg tables, an open table format that improves data reliability and performance. By leveraging Iceberg tables, organizations can enhance analytics capabilities and empower AI applications. This approach enables better data management, reduces costs, and accelerates the development of data-driven solutions.

As data-driven operations scale, data modeling has become increasingly complex and fast-paced, often clashing with the demands of agile delivery. This session explores how Generative AI is tackling long-standing challenges such as source data understanding and gaps in data catalogs. It highlights how GenAI is automating tasks like data profiling, metadata generation, schema design, and mapping, unlocking new levels of speed and accuracy. Real-world examples from Tiger Analytics illustrate the impact, along with a perspective on the evolving role of GenAI as a co-pilot in building scalable, business-aligned data architectures.

Data Engineering and Technical Product Capabilities at AMEX empowering critical business and risk decisioning at AMEX.

In his session, Koushik will reveal how organizations can transform from passive data collectors to strategic data orchestrators, unlocking unprecedented business growth and competitive advantage.
Drawing from his learnings of proven frameworks being adopted by industry-leading financial institutions and companies, he will outline the foundational pillars necessary to build a high-impact data strategy—starting with organizational alignment, where data teams are integrated into business units and held accountable for P&L outcomes.

Koushik will also highlight the role of experimentation, scalable infrastructure, and predictive analytics in optimizing every stage of the customer journey and the common pitfalls like misaligned KPIs, siloed teams, and resistance to change—offering practical solutions to overcome them.

The session will deliver an overall theme that winning with data is not about tools alone; it’s about aligning people, processes, and purpose.

In this session, Prakash Patidar will explore the architecture of next-gen data platforms that are unified, reliable, scalable, and AI-driven. He will discuss how real-time data processing and AI technologies are transforming data systems, enabling smarter decision-making at scale. With a focus on automation, data quality, and real-time insights, this talk will showcase how businesses can leverage AI to unlock the true potential of their data, especially in the fast-evolving world of crypto and DeF.

In a world overflowing with data, the true challenge lies not in access—but in articulation. This session explores the craft of transforming raw, noisy data into compelling, credible, and impactful narratives. From choosing the right metrics to visual storytelling and context-driven publishing, we’ll unpack techniques that help data professionals elevate insight over information. Whether you’re building dashboards, whitepapers, or public datasets, discover how to publish data that informs, engages, and drives action.

Data engineers play a pivotal role in driving contextualized intelligence. They must ensure that data is enriched with the proper context to deliver meaningful insights.

This tech talk will explore the practical steps and tools data engineers can use to build a contextual layer, ensuring AI models can deliver strategic insights based on relevant business contexts.

AI is reshaping the landscape of data engineering in transformative ways. One key trend is the rise of AI-powered data pipelines, where machine learning automates tasks like data cleaning, transformation, and feature engineering, significantly accelerating development cycles. Alongside this, AI-driven data observability tools are improving data reliability by detecting anomalies and quality issues in real time. Intelligent data catalogs are also evolving, using AI to automatically generate metadata, track lineage, and simplify data discovery.
Natural language interfaces, such as Text-to-SQL systems powered by large language models, are making it easier for non-technical users to query databases using plain English. ETL and ELT processes are becoming smarter too, with AI automating schema mapping and transformation logic, reducing the need for manual coding. In parallel, synthetic data generation is gaining traction, enabling the creation of realistic datasets for testing, training, and privacy-sensitive use cases.
Real-time data processing is becoming more intelligent as AI models integrate directly into streaming frameworks like Kafka and Spark, supporting use cases like fraud detection and personalization. Governance is also being augmented by AI, with tools now capable of automatically detecting sensitive data like PII and enforcing compliance with regulations.

In this talk, I will dive into a real-time data pipeline solution I worked on for the Travel & Hospitality domain, powered by Apache Flink Stateful Functions. This solution efficiently handled over 800,000 streaming events per day, addressing critical use cases. I will share insights into the challenges faced, the strategies used to overcome them, and how we successfully scaled the pipeline to meet demanding business requirements.

For the second part of the session, I will explore the hot topic: Will Low-Code ETL Replace Data Engineers? I firmly believe that Low-Code ETL is not here to replace data engineers but to accelerate the development process. However, for this to succeed, ETL platforms must empower data engineers with the freedom and control to customize and adapt the platform to their specific needs. I’ll discuss examples like Prophecy, a Low-Code ETL platform that strikes a balance by acting as an IDE, generating open-source code, and keeping implementation transparent. This approach ensures that while development is expedited, the power remains firmly in the hands of data engineers.

The Unified Data platform is designed to address key limitations in data management for machine learning (ML) model development, training, and testing. Its primary focus is to improve turnaround time (TAT), enhance experimentation velocity, and provide a centralized solution for data needs.