Day 2 | HALL 2 - Practical Insights and Best Practices

Unlock the potential of a high-performance, scalable, and cost-effective data pipeline. Learn how Kafka enables real-time data ingestion, Airflow streamlines workflow orchestration, Parquet optimizes data storage, and Superset enhances data visualization. Discover practical insights and real-world examples to elevate your data infrastructure, boost performance, and save costs.

This session will explore how data engineering practices come into play when operationalizing foundation models. We’ll move beyond the model itself to understand how data engineers prepare and manage the data these models rely on, ensuring they function effectively in real-world applications.

We will cover:
Foundation models as data representations: We’ll discuss how foundation models can be seen as powerful representations of underlying data, and how data engineering is crucial for leveraging this potential.
Data pipelines for foundation models: The focus will shift to the role of data engineers in building and maintaining data pipelines that continuously feed the model with high-quality data.
Fine-tuning foundation models: We’ll explore how data engineers can collaborate with data scientists to prepare data for fine-tuning foundation models for specific tasks.
Data management for foundation models: This section will delve into the data management challenges associated with foundation models, such as data versioning and monitoring.

This talk will explore two key perspectives:

What will be Generative AI’s role in shaping Data Engineering, and
How should Data Engineering practices evolve to support Generative AI.

Data engineering is a critical discipline to build capabilities that source, model and transform data to create usable information. This enables users to derive actionable insights, drive automation and create business value through Analytics, Reporting, AIML and now Generative AI. Generative AI will have significant implications on the discipline across its life cycle. It’ll play an instrumental role in enriching the existing data ecosystem and compel stakeholders to re-think end-to-end architecture and implementation. Primary focus areas include:

Data pipeline and workflow (Extract, Transform, Load) automation and modernization
Data governance – Metadata (and data) enrichment, Tagging, Cataloguing, Lineage, Usage and Documentation
Synthetic data generation and augmentation
Feature engineering automation
Data labelling and annotation assistance
Data re-usability
Natural language-based analytics

Additionally, data engineering practices need to evolve to support Generative AI adoption across Organizations, expanding to unstructured, multi-modal, and diverse data sources. This would create unique opportunities/challenges that must be addressed:

Storage and compute scalability
Frameworks and best practices for data processing – Indexing, Chunking, Embedding
Effective and efficient querying and retrieval
Standards for data quality, lineage, versioning (recency), drift (changes), metadata requirements
Standards for governance, security and regulatory compliance
Custom firm-specific labelling and annotation to fine-tune and/or evaluate Generative AI solutions
Unified data warehouse and feature store

In summary, data engineering discipline would require transformational changes to take advantage of Generative AI while also enabling organizations to leverage their data and processes to adopt Generative AI to derive efficiencies and create value.

Join this immersive workshop, delving into the intricacies of managing vast datasets for training generative models. Learn advanced techniques in data preprocessing, storage, and manipulation tailored for large-scale Generative AI projects. Equip yourself with the essential skills needed to drive innovation in AI. Join us and harness the potential of managing massive datasets to propel AI innovation forward, ensuring you’re ready to tackle the challenges of tomorrow’s AI landscape with confidence and expertise.

The evolution of data engineering encompasses transformative concepts driving advancements in knowledge graphs, data discovery, synthetic data generation, text-to-SQL transformations, data quality management and generative AI datasets.
Knowledge graphs facilitate semantic data modelling and interconnected data querying. Data discovery empowers insights extraction through metadata analysis and automated profiling. Synthetic data generation mitigates data scarcity using generative AI for realistic datasets. Text-to-SQL bridges natural language queries with structured databases, enhancing accessibility. Data quality management ensures integrity, consistency, and reliability through profiling and error remediation. Generative AI datasets fuel innovation and creativity in generative models. This talk will take the audience through real-world cases of the above-mentioned concepts.

MaaS, or Metrics as a Service, is a cloud-based model that provides organisations access to many metrics and analytics capabilities without building and maintaining complex data pipelines or infrastructure. In a MaaS platform, users can define, compute, visualise, and manage metrics and key performance indicators (KPIs) using a centralised and scalable solution.

Booking Holding is the world’s leading provider of online travel and related services and operates through 5 primary consumer facing brands : Booking,com, Priceline, Agoda, Kayak and Opentable, as well as through a network of subsidiary brands. The next phase of growth at booking will be driven by providing a unified connected trip experience and modernisation which will make it easier for everyone to experience the world. As part of this talk Ankur and Sucharita will highlight how booking is modernizing the data platform to Provide a modern, governed, self service Data and ML platform – facilitating data driven decision making and be future ready to support experimentation and AI/ML use cases while supporting the large booking business.

Data observability has garnered considerable interest in recent times. It has evolved from Observability, which relied on monitoring, logging, tracing, etc. However, the expectations associated with data observability and its key tenets have shifted significantly from traditional ways of thinking and implementing it. This session will delve into more practical aspects of Data Observability and how individuals, organisations, and platforms should think differently to gain maximum value from it in a cost-efficient manner.