This whitepaper presents how AI-powered smart aggregation technology enables interactive analysis on billions of data points on Azure while controlling analytics costs. Organizations can use their existing business intelligence tools for ad hoc analysis and experience exceptionally high performance, regardless of the size of their data.
Moving data workloads to Azure helps organizations scale their storage limitlessly. Most enterprises begin by migrating data from traditional EDW or on-premise storage platforms to either Azure Blob Storage or Azure Data Lake Storage (ADLS).
Once the initial storage is set up, the next challenge is to conduct interactive analytics on this data. This whitepaper explains how businesses can take their business intelligence to the next level by changing how users interact with data on Azure.
Enterprises today deal with limitless data volumes incoming from varied sources such as traditional point-of-sale systems, e-commerce sites, social media platforms, or IoT sensors streaming data in real-time. Once this data comes into the cloud, the next issue is making it available to users for accessibility, ease of use and delivering high-performance, interactive responses required by BI and analytics teams.
Though it has become easier to store and manage massive volumes of distributed data on Azure storage, the real challenge is meeting the growing expectations of business users and analysts. Reporting today is more complex than ever; timelines are aggressive and there is no tolerance for latency. Achieving interactive speeds becomes critical for enterprises. They need tools that can bring the benefits of analytics to business users as seamlessly and transparently as possible without disrupting their day-to-day activities.
When business users and analysts try to analyze large datasets on Azure across hundreds of dimensions and measures, response times are typically slow in a complex business scenario. As a result, there are limitations on the amount of data that can be processed or the depth of analysis that can be performed. The problem intensifies as more and more data lands into the cloud and concurrency increases.
Let us examine some of the popular techniques used for advanced analytics on Azure, discuss their limitations and then see how Kyvos’ AI-powered smart aggregation technology overcomes those challenges.
One option is that users query the Azure data warehouse directly. However, in this case, a considerable amount of coding effort is involved in writing queries to transform data & extract insights. This makes it unsuitable for business users.
Besides, complex on-the-fly SQL operations result in performance slowdowns as all heavy-lifting is done at query time. The problem escalates further for business scenarios with complex queries and higher concurrencies.
Another option is to use Azure Analysis Services (AAS) for querying. Given AAS works on a single instance, there are limitations on the extent to which it can scale out. In-memory processing brings in resource dependencies and performance suffers when the load increases. Also, when memory requirements are high, costs can become prohibitive, making it unsuitable for complex calculations and high-cardinality use cases.
Unlike SSAS, AAS does not support multidimensional models. As a result, critical business use cases that require interactive analysis on massive datasets become difficult to support.
Kyvos’ AI-powered smart aggregation technology eliminates the limitations of cloud analytics, offering the best of both worlds—the speed and interactivity of OLAP combined with the scalability and flexibility of Azure. This kind of performance cannot be achieved at this scale with any other solution.
Kyvos can build data models on hundreds of dimensions and measures, enabling quick multidimensional analytics on the cloud without any coding effort. ML-powered smart recommendation engine brings the intelligence needed to build smarter aggregates in the cloud, helping enterprises deal with the scale and complexity of today’s data. Users can query, filter, slice & dice, drill down and visually explore extremely large datasets on Azure with instant response times. They can delve into the most granular details and get instant answers to all their business questions. In addition, they have the flexibility to use their BI tool of choice for analytics.
Kyvos creates a semantic layer that provides a consistent data view for users across the enterprise. All metadata can be defined in Kyvos, eliminating the need to write complex logic in multiple tools. Users can then view the dimensions and measures available to them in their BI tool and drag and drop them into their visualizations intuitively. They can define all kinds of hierarchies (e.g., multiple, alternate, parent-child), different ways of aggregating, custom roll-ups and more. Once the semantic model is complete, users have a consistent path to query and drill down through a hierarchy.
With no data movement, Kyvos maintains the native data security of the storage layer itself. The Kyvos data models are stored and cached in Azure Data Lake Storage Gen2. It has a built-in security model and supports standard security frameworks and protocols. Enterprise-class data security features, including row and column-level security, allow users to see only what they are authorized to see. Kyvos works seamlessly with the security and access management features of the Azure platform.
Auto-scaling provides consistent performance during spikes and shrinks costs during lean periods. As more data gets added to the BI environment, Kyvos can be configured to scale out to build data models on that data. This enables extremely fast builds, even on billions of rows of data. Similarly, querying capacity can be scaled up or down depending upon expected loads. Users can configure the day and time when additional resources need to be added and then release them when they are not in use.
As data volumes rise further and the analytical ecosystem evolves, enterprises need a BI environment that has built-in elasticity and can deliver high performance on Azure workloads, irrespective of reporting complexity or data size.
Kyvos eliminates the complexity of big data analytics and enables direct access to massive data on Azure without the need to write any code. It fits seamlessly into the Azure stack, leveraging its scalability and flexibility to build massively scalable data model that deliver interactive performance on billions of rows of data with high concurrency.