Day 1 | HALL 2 - Practical Insights and Best Practices

During the session, Varun will be talking about how the process of ETL has evolved to ELT, and the different types of data storage/warehouse solutions.

Do you know about DataHub, the #1 Metadata Platform loved by data engineers? It’s like a superpowered tool for finding and understanding data, keeping an eye on it, and making sure it follows the rules. And it’s open source! Big names like MYOB, DPG Media, Notion, PayPal, Airtel, Netflix, Expedia, and LinkedIn are already on board.

DataHub plays nice with lots of other tools, like Snowflake, Redshift, BigQuery, S3, Airflow, NiFi, dbt, Great Expectations, Looker, Tableau, and more. It’s super easy to plug into whatever setup you’ve got with a rich set of APIs for producing and consuming metadata. But of course, you have to run it yourself!

With Acryl Cloud (the cloud hosted version of DataHub), DataHub gets even better with additional features for collaboration, governance and data quality. It’s like having a Swiss Army knife for data management. We’ll dive into why the Credit Saison team picked Acryl Cloud and how it’s helping them.

The talk will also cover cool new stuff coming to DataHub and Acryl Cloud soon! Think smarter ways to track where data comes from, searching using plain old English, a handy Slack bot, making sure data is what it says it is, using AI to make metadata even more useful, and a bunch more exciting features.

This session will delve into our Gen AI capabilities seamlessly integrated into Microsoft Fabric. You will understand more about the hurdles confronting organizations and discuss customer scenarios showcasing our accelerators and solutions on Microsoft’s cutting-edge SaaS Platform. Get ready to discover the tangible value we deliver to our global clients through these innovations.

Unstructured data such as customer conversations, meetings, social media contents, media files – images & audios etc. presents a significant analytics challenge for businesses. This presentation explores how Generative AI can transform such data into structured insights, enhancing decision-making and operational efficiency. We will discuss applications including real-time AI copilots and bots that automate tasks and improve services.

Key highlights include case studies from NoBroker.com and ConvoZen AI, showcasing how these innovations lead to efficiency gains and better experiences. Attendees will leave with actionable strategies to harness Generative AI in their own organisations, turning data chaos into a competitive advantage.

Discuss how adopting a product mindset towards data encourages organizations to improve data accessibility and interpretability, ensuring that it meets customer needs much like any other product.
Roles and responsibilities of data product managers—professionals who oversee the lifecycle of data products from creation to deployment and refinement. Examine case studies from leading companies that have successfully implemented this strategy, highlighting how they manage data as a product to enhance decision-making and customer satisfaction.
Address the challenges of this approach, including the need for cultural shifts within organizations, the importance of cross-functional collaboration, and the continuous investment required to maintain and improve data products.
Conclude with actionable insights on how attendees can start thinking about and treating their data as a product, setting the stage for enhanced innovation and efficiency in their processes. This shift not only improves service delivery but also ensures a competitive edge in the data-driven marketplace.

The data landscape is in a constant state of flux, demanding ever-more agile and scalable data processing solutions. While the Lambda architecture has served well, the Kappa architecture emerges as a powerful evolution. This session, designed for senior data leaders, explores the fundamentals and nuances of the Kappa architecture and its potential to revolutionize data pipelines.

We’ll embark on a journey beyond the Lambda architecture, unpacking the core principles of Kappa. You’ll gain insights into how Kappa streamlines data processing by unifying batch and real-time processing into a single, continuous flow. This approach eliminates the complexity of managing separate Lambda layers, fostering a more agile and maintainable data pipeline.

The session dives deep into the technical aspects of Kappa, including:
Real-time Stream Processing: Leveraging powerful stream processing engines for low-latency data ingestion and transformation.
Stateful Stream Processing: Enabling complex event processing and state management within the streaming pipeline itself.
Simplified Data Pipelines: Reducing code duplication and operational overhead through a unified processing approach.
Tailored Delivery Options: Providing flexibility to deliver data in real-time, near real-time, or in batches based on specific use cases.
This session is ideal for data Leaders, data architects and data engineers seeking to push the boundaries of data processing agility. We’ll explore real-world applications and best practices, empowering you to architect data pipelines that are not only scalable but also adaptable to the ever-changing needs of your organization.

Over the past couple of years, we have experienced how Analytics and Machine learning have come into action in several sectors like E-commerce, health, education, Finance, and Agriculture, and organisations could see tremendous value out of data-driven decision-making. Although we adopted distributed computing platforms in building analytics services, the outlook on unlocking insights from analytics has still been traditional, that is, by building data warehouses and data marts. The Enterprise Data Warehouse (EDW) technologies were able to integrate and harmonize data, enabling BI analysts and users to extract information reliably, but flexibility and addressing the evolving data needs have been a constant challenge.

In this talk, I would like to reveal how some hidden patterns could be extracted by
realising the problems as Graphs. I’ll in-brief state some of the limitations of the existing EDW and how these could be addressed through OLAP Graph technologies. OLAP graph
technologies and their implementation, known as knowledge graphs, can link various heterogeneous data sources. I would also like to take you through some of the real-world
challenges addressed by embedding Graph + AI design principles into our strategy.
The real data challenges in our day-to-day work revolve around entities and their
respective attributes.

The talk will include:

1. Why Graph + analytics ?
2. What problems could we realise as a Graph?
3. Graph Technology for Data Integration
4. Defining consumable patterns for analysts and business stakeholders.
5. Use-cases
Entity resolution in E-commerce
360-degree view of customers
Improve enterprise decision-making by enabling cross-channel communication
Deduplicating entities
6. OLAP Graph Data Warehouse

How Aquaconnect leverage satellite imagery to monitor coastal regions, enabling us to data driven decisions which drives sustainable aquaculture development. This bird’s-eye view allows for precise and timely decision-making, ensuring optimal conditions for aquatic species. Approach integrates advanced remote sensing technologies with artificial intelligence to provide unparalleled insights into aquatic ecosystems

How can the integration of Gen AI & Data Engineering help to accelerate the growth of any technology organisation, what it takes to adopt the Gen AI ecosystem, and how to work towards a successful discovery & delivery model using the combination of 2 very different approaches