Day 1 | HALL 2 - Practical Insights and Best Practices

The data landscape is in a constant state of flux, demanding ever-more agile and scalable data processing solutions. While the Lambda architecture has served well, the Kappa architecture emerges as a powerful evolution. This session, designed for senior data leaders, explores the fundamentals and nuances of the Kappa architecture and its potential to revolutionize data pipelines.

We’ll embark on a journey beyond the Lambda architecture, unpacking the core principles of Kappa. You’ll gain insights into how Kappa streamlines data processing by unifying batch and real-time processing into a single, continuous flow. This approach eliminates the complexity of managing separate Lambda layers, fostering a more agile and maintainable data pipeline.

The session dives deep into the technical aspects of Kappa, including:
Real-time Stream Processing: Leveraging powerful stream processing engines for low-latency data ingestion and transformation.
Stateful Stream Processing: Enabling complex event processing and state management within the streaming pipeline itself.
Simplified Data Pipelines: Reducing code duplication and operational overhead through a unified processing approach.
Tailored Delivery Options: Providing flexibility to deliver data in real-time, near real-time, or in batches based on specific use cases.
This session is ideal for data Leaders, data architects and data engineers seeking to push the boundaries of data processing agility. We’ll explore real-world applications and best practices, empowering you to architect data pipelines that are not only scalable but also adaptable to the ever-changing needs of your organization.

Discover the strategic advantages of modernizing data architecture on AWS in the age of big data. Explore best practices for architecting scalable data pipelines, optimizing storage and processing, and harnessing advanced analytics and machine learning. Gain insights into emerging trends and technologies and learn how to unlock new opportunities for data-driven growth on AWS.

The talk will cover:

– Introduction to Modern Data Architecture on AWS
– Rationales for Modernizing Data Architecture on AWS
– Architecting Scalable Data Pipelines on AWS
– Optimizing Data Storage and Processing
– Future Trends and Considerations

Over the past couple of years, we have experienced how Analytics and Machine learning have come into action in several sectors like E-commerce, health, education, Finance, and Agriculture, and organisations could see tremendous value out of data-driven decision-making. Although we adopted distributed computing platforms in building analytics services, the outlook on unlocking insights from analytics has still been traditional, that is, by building data warehouses and data marts. The Enterprise Data Warehouse (EDW) technologies were able to integrate and harmonize data, enabling BI analysts and users to extract information reliably, but flexibility and addressing the evolving data needs have been a constant challenge.

In this talk, I would like to reveal how some hidden patterns could be extracted by
realising the problems as Graphs. I’ll in-brief state some of the limitations of the existing EDW and how these could be addressed through OLAP Graph technologies. OLAP graph
technologies and their implementation, known as knowledge graphs, can link various heterogeneous data sources. I would also like to take you through some of the real-world
challenges addressed by embedding Graph + AI design principles into our strategy.
The real data challenges in our day-to-day work revolve around entities and their
respective attributes.

The talk will include:

1. Why Graph + analytics ?
2. What problems could we realise as a Graph?
3. Graph Technology for Data Integration
4. Defining consumable patterns for analysts and business stakeholders.
5. Use-cases
Entity resolution in E-commerce
360-degree view of customers
Improve enterprise decision-making by enabling cross-channel communication
Deduplicating entities
6. OLAP Graph Data Warehouse

How Aquaconnect leverage satellite imagery to monitor coastal regions, enabling us to data driven decisions which drives sustainable aquaculture development. This bird’s-eye view allows for precise and timely decision-making, ensuring optimal conditions for aquatic species. Approach integrates advanced remote sensing technologies with artificial intelligence to provide unparalleled insights into aquatic ecosystems

How can the integration of Gen AI & Data Engineering help to accelerate the growth of any technology organisation, what it takes to adopt the Gen AI ecosystem, and how to work towards a successful discovery & delivery model using the combination of 2 very different approaches

Embeddings are a ubiquitous term in discussions surrounding AI models, particularly in GenAi. While numerous resources explore the mathematical and theoretical underpinnings of embeddings and their significance in training Transformers or ML models in general, there remains a scarcity of material exploring their practical applications. Embeddings serve as data structures containing contextual information essential for executing intelligent tasks.

Among the myriad applications of AI, semantic search stands out as one of the most prominent, directly influenced by the quality and scale of embeddings. The selection of embeddings significantly impacts search capabilities and the supported modalities, ranging from text-only to image search or even a fusion of both, known as Multi-Modality search.

Recognizing embeddings as pivotal data structures, devising efficient storage mechanisms is imperative. Vector databases emerge as specialized solutions optimized for storing and retrieving embeddings.

In this presentation, I will delineate establishing a reference architecture using Vector Database & Embeddings for intelligent search, encompassing options for text, image, and multi-modal searches. Also, I will highlight a few applications leveraging this architecture and outline potential avenues for future exploration.