Unstructured data such as customer conversations, meetings, social media contents, media files - images & audios etc. presents a significant analytics challenge for businesses. This presentation explores how Generative AI can transform such data into structured insights, enhancing decision-making and operational efficiency. We will discuss applications including real-time AI copilots and bots that automate tasks and improve services. Key highlights include case studies from NoBroker.com and ConvoZen AI, showcasing how these innovations lead to efficiency gains and better experiences. Attendees will leave with actionable strategies to harness Generative AI in their own organisations, turning data chaos into a competitive advantage.

During the session, Varun will be talking about how the process of ETL has evolved to ELT, and the different types of data storage/warehouse solutions.

This session will delve into our Gen AI capabilities seamlessly integrated into Microsoft Fabric. You will understand more about the hurdles confronting organizations and discuss customer scenarios showcasing our accelerators and solutions on Microsoft's cutting-edge SaaS Platform. Get ready to discover the tangible value we deliver to our global clients through these innovations.

Discuss how adopting a product mindset towards data encourages organizations to improve data accessibility and interpretability, ensuring that it meets customer needs much like any other product. Roles and responsibilities of data product managers—professionals who oversee the lifecycle of data products from creation to deployment and refinement. Examine case studies from leading companies that have successfully implemented this strategy, highlighting how they manage data as a product to enhance decision-making and customer satisfaction. Address the challenges of this approach, including the need for cultural shifts within organizations, the importance of cross-functional collaboration, and the continuous investment required to maintain and improve data products. Conclude with actionable insights on how attendees can start thinking about and treating their data as a product, setting the stage for enhanced innovation and efficiency in their processes. This shift not only improves service delivery but also ensures a competitive edge in the data-driven marketplace.

The data landscape is in a constant state of flux, demanding ever-more agile and scalable data processing solutions. While the Lambda architecture has served well, the Kappa architecture emerges as a powerful evolution. This session, designed for senior data leaders, explores the fundamentals and nuances of the Kappa architecture and its potential to revolutionize data pipelines. We'll embark on a journey beyond the Lambda architecture, unpacking the core principles of Kappa. You'll gain insights into how Kappa streamlines data processing by unifying batch and real-time processing into a single, continuous flow. This approach eliminates the complexity of managing separate Lambda layers, fostering a more agile and maintainable data pipeline. The session dives deep into the technical aspects of Kappa, including: Real-time Stream Processing: Leveraging powerful stream processing engines for low-latency data ingestion and transformation. Stateful Stream Processing: Enabling complex event processing and state management within the streaming pipeline itself. Simplified Data Pipelines: Reducing code duplication and operational overhead through a unified processing approach. Tailored Delivery Options: Providing flexibility to deliver data in real-time, near real-time, or in batches based on specific use cases. This session is ideal for data Leaders, data architects and data engineers seeking to push the boundaries of data processing agility. We'll explore real-world applications and best practices, empowering you to architect data pipelines that are not only scalable but also adaptable to the ever-changing needs of your organization.

Do you know about DataHub, the #1 Metadata Platform loved by data engineers? It's like a superpowered tool for finding and understanding data, keeping an eye on it, and making sure it follows the rules. And it’s open source! Big names like MYOB, DPG Media, Notion, PayPal, Airtel, Netflix, Expedia, and LinkedIn are already on board. DataHub plays nice with lots of other tools, like Snowflake, Redshift, BigQuery, S3, Airflow, NiFi, dbt, Great Expectations, Looker, Tableau, and more. It's super easy to plug into whatever setup you've got with a rich set of APIs for producing and consuming metadata. But of course, you have to run it yourself! With Acryl Cloud (the cloud hosted version of DataHub), DataHub gets even better with additional features for collaboration, governance and data quality. It's like having a Swiss Army knife for data management. We'll dive into why the Credit Saison team picked Acryl Cloud and how it's helping them. The talk will also cover cool new stuff coming to DataHub and Acryl Cloud soon! Think smarter ways to track where data comes from, searching using plain old English, a handy Slack bot, making sure data is what it says it is, using AI to make metadata even more useful, and a bunch more exciting features.

Over the past couple of years, we have experienced how Analytics and Machine learning have come into action in several sectors like E-commerce, health, education, Finance, and Agriculture, and organisations could see tremendous value out of data-driven decision-making. Although we adopted distributed computing platforms in building analytics services, the outlook on unlocking insights from analytics has still been traditional, that is, by building data warehouses and data marts. The Enterprise Data Warehouse (EDW) technologies were able to integrate and harmonize data, enabling BI analysts and users to extract information reliably, but flexibility and addressing the evolving data needs have been a constant challenge. In this talk, I would like to reveal how some hidden patterns could be extracted by realising the problems as Graphs. I'll in-brief state some of the limitations of the existing EDW and how these could be addressed through OLAP Graph technologies. OLAP graph technologies and their implementation, known as knowledge graphs, can link various heterogeneous data sources. I would also like to take you through some of the real-world challenges addressed by embedding Graph + AI design principles into our strategy. The real data challenges in our day-to-day work revolve around entities and their respective attributes. The talk will include: 1. Why Graph + analytics ? 2. What problems could we realise as a Graph? 3. Graph Technology for Data Integration 4. Defining consumable patterns for analysts and business stakeholders. 5. Use-cases Entity resolution in E-commerce 360-degree view of customers Improve enterprise decision-making by enabling cross-channel communication Deduplicating entities 6. OLAP Graph Data Warehouse

How Aquaconnect leverage satellite imagery to monitor coastal regions, enabling us to data driven decisions which drives sustainable aquaculture development. This bird's-eye view allows for precise and timely decision-making, ensuring optimal conditions for aquatic species. Approach integrates advanced remote sensing technologies with artificial intelligence to provide unparalleled insights into aquatic ecosystems

How can the integration of Gen AI & Data Engineering help to accelerate the growth of any technology organisation, what it takes to adopt the Gen AI ecosystem, and how to work towards a successful discovery & delivery model using the combination of 2 very different approaches

Embeddings are a ubiquitous term in discussions surrounding AI models, particularly in GenAi. While numerous resources explore the mathematical and theoretical underpinnings of embeddings and their significance in training Transformers or ML models in general, there remains a scarcity of material exploring their practical applications. Embeddings serve as data structures containing contextual information essential for executing intelligent tasks. Among the myriad applications of AI, semantic search stands out as one of the most prominent, directly influenced by the quality and scale of embeddings. The selection of embeddings significantly impacts search capabilities and the supported modalities, ranging from text-only to image search or even a fusion of both, known as Multi-Modality search. Recognizing embeddings as pivotal data structures, devising efficient storage mechanisms is imperative. Vector databases emerge as specialized solutions optimized for storing and retrieving embeddings. In this presentation, I will delineate establishing a reference architecture using Vector Database & Embeddings for intelligent search, encompassing options for text, image, and multi-modal searches. Also, I will highlight a few applications leveraging this architecture and outline potential avenues for future exploration.

Join this immersive workshop, delving into the intricacies of managing vast datasets for training generative models. Learn advanced techniques in data preprocessing, storage, and manipulation tailored for large-scale Generative AI projects. Equip yourself with the essential skills needed to drive innovation in AI. Join us and harness the potential of managing massive datasets to propel AI innovation forward, ensuring you're ready to tackle the challenges of tomorrow's AI landscape with confidence and expertise.

This session will explore how data engineering practices come into play when operationalizing foundation models. We'll move beyond the model itself to understand how data engineers prepare and manage the data these models rely on, ensuring they function effectively in real-world applications. We will cover: Foundation models as data representations: We'll discuss how foundation models can be seen as powerful representations of underlying data, and how data engineering is crucial for leveraging this potential. Data pipelines for foundation models: The focus will shift to the role of data engineers in building and maintaining data pipelines that continuously feed the model with high-quality data. Fine-tuning foundation models: We'll explore how data engineers can collaborate with data scientists to prepare data for fine-tuning foundation models for specific tasks. Data management for foundation models: This section will delve into the data management challenges associated with foundation models, such as data versioning and monitoring.

The evolution of data engineering encompasses transformative concepts driving advancements in knowledge graphs, data discovery, synthetic data generation, text-to-SQL transformations, data quality management and generative AI datasets. Knowledge graphs facilitate semantic data modelling and interconnected data querying. Data discovery empowers insights extraction through metadata analysis and automated profiling. Synthetic data generation mitigates data scarcity using generative AI for realistic datasets. Text-to-SQL bridges natural language queries with structured databases, enhancing accessibility. Data quality management ensures integrity, consistency, and reliability through profiling and error remediation. Generative AI datasets fuel innovation and creativity in generative models. This talk will take the audience through real-world cases of the above-mentioned concepts.

This talk will explore two key perspectives: What will be Generative AI's role in shaping Data Engineering, and How should Data Engineering practices evolve to support Generative AI. Data engineering is a critical discipline to build capabilities that source, model and transform data to create usable information. This enables users to derive actionable insights, drive automation and create business value through Analytics, Reporting, AIML and now Generative AI. Generative AI will have significant implications on the discipline across its life cycle. It'll play an instrumental role in enriching the existing data ecosystem and compel stakeholders to re-think end-to-end architecture and implementation. Primary focus areas include: Data pipeline and workflow (Extract, Transform, Load) automation and modernization Data governance – Metadata (and data) enrichment, Tagging, Cataloguing, Lineage, Usage and Documentation Synthetic data generation and augmentation Feature engineering automation Data labelling and annotation assistance Data re-usability Natural language-based analytics Additionally, data engineering practices need to evolve to support Generative AI adoption across Organizations, expanding to unstructured, multi-modal, and diverse data sources. This would create unique opportunities/challenges that must be addressed: Storage and compute scalability Frameworks and best practices for data processing – Indexing, Chunking, Embedding Effective and efficient querying and retrieval Standards for data quality, lineage, versioning (recency), drift (changes), metadata requirements Standards for governance, security and regulatory compliance Custom firm-specific labelling and annotation to fine-tune and/or evaluate Generative AI solutions Unified data warehouse and feature store In summary, data engineering discipline would require transformational changes to take advantage of Generative AI while also enabling organizations to leverage their data and processes to adopt Generative AI to derive efficiencies and create value.

MaaS, or Metrics as a Service, is a cloud-based model that provides organisations access to many metrics and analytics capabilities without building and maintaining complex data pipelines or infrastructure. In a MaaS platform, users can define, compute, visualise, and manage metrics and key performance indicators (KPIs) using a centralised and scalable solution.

Booking Holding is the world's leading provider of online travel and related services and operates through 5 primary consumer facing brands : Booking,com, Priceline, Agoda, Kayak and Opentable, as well as through a network of subsidiary brands. The next phase of growth at booking will be driven by providing a unified connected trip experience and modernisation which will make it easier for everyone to experience the world. As part of this talk Ankur and Sucharita will highlight how booking is modernizing the data platform to Provide a modern, governed, self service Data and ML platform - facilitating data driven decision making and be future ready to support experimentation and AI/ML use cases while supporting the large booking business.

Data observability has garnered considerable interest in recent times. It has evolved from Observability, which relied on monitoring, logging, tracing, etc. However, the expectations associated with data observability and its key tenets have shifted significantly from traditional ways of thinking and implementing it. This session will delve into more practical aspects of Data Observability and how individuals, organisations, and platforms should think differently to gain maximum value from it in a cost-efficient manner.

Unlock the potential of a high-performance, scalable, and cost-effective data pipeline. Learn how Kafka enables real-time data ingestion, Airflow streamlines workflow orchestration, Parquet optimizes data storage, and Superset enhances data visualization. Discover practical insights and real-world examples to elevate your data infrastructure, boost performance, and save costs.