The data landscape is in a constant state of flux, demanding ever-more agile and scalable data processing solutions. While the Lambda architecture has served well, the Kappa architecture emerges as a powerful evolution. This session, designed for senior data leaders, explores the fundamentals and nuances of the Kappa architecture and its potential to revolutionize data pipelines. We'll embark on a journey beyond the Lambda architecture, unpacking the core principles of Kappa. You'll gain insights into how Kappa streamlines data processing by unifying batch and real-time processing into a single, continuous flow. This approach eliminates the complexity of managing separate Lambda layers, fostering a more agile and maintainable data pipeline. The session dives deep into the technical aspects of Kappa, including: Real-time Stream Processing: Leveraging powerful stream processing engines for low-latency data ingestion and transformation. Stateful Stream Processing: Enabling complex event processing and state management within the streaming pipeline itself. Simplified Data Pipelines: Reducing code duplication and operational overhead through a unified processing approach. Tailored Delivery Options: Providing flexibility to deliver data in real-time, near real-time, or in batches based on specific use cases. This session is ideal for data Leaders, data architects and data engineers seeking to push the boundaries of data processing agility. We'll explore real-world applications and best practices, empowering you to architect data pipelines that are not only scalable but also adaptable to the ever-changing needs of your organization.

Discover the strategic advantages of modernizing data architecture on AWS in the age of big data. Explore best practices for architecting scalable data pipelines, optimizing storage and processing, and harnessing advanced analytics and machine learning. Gain insights into emerging trends and technologies and learn how to unlock new opportunities for data-driven growth on AWS. The talk will cover: - Introduction to Modern Data Architecture on AWS - Rationales for Modernizing Data Architecture on AWS - Architecting Scalable Data Pipelines on AWS - Optimizing Data Storage and Processing - Future Trends and Considerations

Over the past couple of years, we have experienced how Analytics and Machine learning have come into action in several sectors like E-commerce, health, education, Finance, and Agriculture, and organisations could see tremendous value out of data-driven decision-making. Although we adopted distributed computing platforms in building analytics services, the outlook on unlocking insights from analytics has still been traditional, that is, by building data warehouses and data marts. The Enterprise Data Warehouse (EDW) technologies were able to integrate and harmonize data, enabling BI analysts and users to extract information reliably, but flexibility and addressing the evolving data needs have been a constant challenge. In this talk, I would like to reveal how some hidden patterns could be extracted by realising the problems as Graphs. I'll in-brief state some of the limitations of the existing EDW and how these could be addressed through OLAP Graph technologies. OLAP graph technologies and their implementation, known as knowledge graphs, can link various heterogeneous data sources. I would also like to take you through some of the real-world challenges addressed by embedding Graph + AI design principles into our strategy. The real data challenges in our day-to-day work revolve around entities and their respective attributes. The talk will include: 1. Why Graph + analytics ? 2. What problems could we realise as a Graph? 3. Graph Technology for Data Integration 4. Defining consumable patterns for analysts and business stakeholders. 5. Use-cases Entity resolution in E-commerce 360-degree view of customers Improve enterprise decision-making by enabling cross-channel communication Deduplicating entities 6. OLAP Graph Data Warehouse

How Aquaconnect leverage satellite imagery to monitor coastal regions, enabling us to data driven decisions which drives sustainable aquaculture development. This bird's-eye view allows for precise and timely decision-making, ensuring optimal conditions for aquatic species. Approach integrates advanced remote sensing technologies with artificial intelligence to provide unparalleled insights into aquatic ecosystems

How can the integration of Gen AI & Data Engineering help to accelerate the growth of any technology organisation, what it takes to adopt the Gen AI ecosystem, and how to work towards a successful discovery & delivery model using the combination of 2 very different approaches

Embeddings are a ubiquitous term in discussions surrounding AI models, particularly in GenAi. While numerous resources explore the mathematical and theoretical underpinnings of embeddings and their significance in training Transformers or ML models in general, there remains a scarcity of material exploring their practical applications. Embeddings serve as data structures containing contextual information essential for executing intelligent tasks. Among the myriad applications of AI, semantic search stands out as one of the most prominent, directly influenced by the quality and scale of embeddings. The selection of embeddings significantly impacts search capabilities and the supported modalities, ranging from text-only to image search or even a fusion of both, known as Multi-Modality search. Recognizing embeddings as pivotal data structures, devising efficient storage mechanisms is imperative. Vector databases emerge as specialized solutions optimized for storing and retrieving embeddings. In this presentation, I will delineate establishing a reference architecture using Vector Database & Embeddings for intelligent search, encompassing options for text, image, and multi-modal searches. Also, I will highlight a few applications leveraging this architecture and outline potential avenues for future exploration.

The evolution of data engineering encompasses transformative concepts driving advancements in knowledge graphs, data discovery, synthetic data generation, text-to-SQL transformations, data quality management and generative AI datasets. Knowledge graphs facilitate semantic data modelling and interconnected data querying. Data discovery empowers insights extraction through metadata analysis and automated profiling. Synthetic data generation mitigates data scarcity using generative AI for realistic datasets. Text-to-SQL bridges natural language queries with structured databases, enhancing accessibility. Data quality management ensures integrity, consistency, and reliability through profiling and error remediation. Generative AI datasets fuel innovation and creativity in generative models. This talk will take the audience through real-world cases of the above-mentioned concepts.

MaaS, or Metrics as a Service, is a cloud-based model that provides organisations access to many metrics and analytics capabilities without building and maintaining complex data pipelines or infrastructure. In a MaaS platform, users can define, compute, visualise, and manage metrics and key performance indicators (KPIs) using a centralised and scalable solution.

Booking Holding is the world's leading provider of online travel and related services and operates through 5 primary consumer facing brands : Booking,com, Priceline, Agoda, Kayak and Opentable, as well as through a network of subsidiary brands. The next phase of growth at booking will be driven by providing a unified connected trip experience and modernisation which will make it easier for everyone to experience the world. As part of this talk Ankur and Sucharita will highlight how booking is modernizing the data platform to Provide a modern, governed, self service Data and ML platform - facilitating data driven decision making and be future ready to support experimentation and AI/ML use cases while supporting the large booking business.

Data observability has garnered considerable interest in recent times. It has evolved from Observability, which relied on monitoring, logging, tracing, etc. However, the expectations associated with data observability and its key tenets have shifted significantly from traditional ways of thinking and implementing it. This session will delve into more practical aspects of Data Observability and how individuals, organisations, and platforms should think differently to gain maximum value from it in a cost-efficient manner.