Day 2 | HALL 2 - Practical Insights and Best Practices

In today’s fast-paced world, where every millisecond counts, the ability to predict outcomes quickly and accurately is crucial. This is particularly true in fields such as finance, healthcare, and e-commerce, where timely decisions can mean the difference between success and failure. In this talk, we will explore the challenges associated with achieving millisecond prediction latencies, and discuss strategies for cost-effective training of predictive models. We will also examine techniques for ensuring the sanity of offline versus online performance of prediction models, as well as ways to enable faster experimentation through multiple versions of prediction models. Additionally, we will discuss the problem of model degradation due to data drifts or assumption changes, and explore solutions for addressing this issue. Finally, we will delve into the complexities of solving for batch and real-time use cases of prediction models, and discuss best practices for achieving optimal performance in both scenarios. Join us for an informative and thought-provoking discussion on the latest developments in predictive modelling, and learn how to stay ahead of the curve in this rapidly evolving field.

The data access layer, which plays a crucial role in enterprise applications reliant on relational databases, has evolved significantly over the past two decades. Various innovative solutions have emerged, enhancing the security, efficiency, and productivity of applications and relational databases. Notably, SQL Mappers such as Spring JDBC and MyBatis, as well as JPA (Hibernate ORM), have dominated the market for an extended period. However, in today’s cloud-native landscape, fresh challenges have emerged, and Compile Time ORM has emerged as a novel perspective on these established solutions. In this talk, we will explore the latest trends in the data access layer, taking into account the changing requirements of modern application development.

Probabilistic data structures are a type of statistical algorithm designed to optimize the use of memory in storing and querying large datasets. These structures employ probabilistic algorithms to estimate the presence of elements in a dataset with a high degree of accuracy while minimizing the amount of memory required for storage. In this session, we will explore some fundamental analytical questions that, when answered accurately for very large datasets, require substantial resources and cost. This is particularly relevant in streaming data use cases such as real-time monitoring, fraud detection, social media analytics, and online advertising, where the timely availability of analytics takes precedence over their accuracy.

The ‘T’ in ETL or ELT Pipeline

A tool to equip the business to build its own data marts using SQL/Dbt and create dashboards to drive insights much faster with almost no dependency on the data engineers. A Data Transformation Pipeline to transform the data, which is built using DBT(data build tool) running on serverless containers (AWS Fargate) using Airflow as a workflow orchestrator.

The global market for cloud migration services is expected to grow at a CAGR of
24.3%, reaching USD 340.7 billion by 2028 from USD 92.4 billion in 2021. However,
traditional data management methods are often inefficient, costly, and error-prone,
and organizations must keep up with evolving trends in cloud services. Major
challenges include streamlining the end-to-end data lifecycle process, collaboration
between different teams, creating a conceptualized framework for Ops-Governance
of data, managing and scaling infrastructure, securing data and ensuring
compliance with regulations, monitoring and alerting systems, along with adapting
to new technologies and evolving business needs. Interestingly, data only accounts
for 49% of enterprise decisions, while insights account for 10%; there is clearly room
for growth in both categories.
In this session, we take you through the journey on how to manage enormous
amounts of data effectively and proactively, for identifying relevant opportunities,
driving visibility, building reliability, and achieving scalability in an organization.

By implementing a semantic data layer, new businesses can achieve
several benefits:
– Improved data quality: Faster time-to-insight: A semantic layer can
help reduce the time it takes to extract insights from data by making
it easier to access and use.
– Greater agility.
– Improved collaboration.
Overall, a data semantic layer can be a valuable investment for new
businesses that want to make the most of their data and gain a
competitive advantage in their industry.