Today, organizations are generating vast amounts of data and they consider it the most strategic asset to improve operations and customer experiences. To unlock the true potential of their data, many organizations have migrated their data workloads to cloud data warehouses like Snowflake.
Snowflake is an industry leader in making massive data easy to ingest and it offers relational, scale-out capabilities that help derive value from fast-growing data. Having spent a fortune in migrating their data workloads to Snowflake, it becomes critical for enterprises to manage and control their recurring analytics cost.
However, here are some common cost-related challenges organizations face on Snowflake.
As data volumes rise, the resources required to process it also increase. Analyzing months or years of data becomes expensive, especially when resource-intensive queries that include multiple joins or group bys are fired on massive datasets.
As all the heavy processing is done at query time, it leads to increased resource consumption and a significant rise in costs. Enterprises find it challenging to strike a balance between exploding BI costs and the growing analytical needs of their business users.
Businesses often need to perform ad hoc analysis to find answers at the moment. Cloud data warehouses like Snowflake use caching mechanisms for hot queries to retrieve results faster. But when it comes to ad hoc queries, it takes longer to bring back the results from remote storage.
This implies that every ad hoc query fired scans complete data, does on-the-fly calculations and enterprises have to bear the cost of excessive processing at query time. Now, in case thousands of users from different departments fire ad hoc queries for their day-to-day decisions, then for each query, additional resources are consumed, leading to a huge bill at the end of the month.
Data isn’t used to its full potential if it is unavailable to users across the enterprise. To make it more accessible, organizations consolidated data from multiple sources into a cloud data warehouse like Snowflake.
However, there is a significant degradation in performance when hundreds of users send complex queries to Snowflake simultaneously. It takes longer than expected to bring back results. Due to this, businesses need to launch additional warehouses to improve query performance and each new warehouse will impact your costs and increase their monthly bill.
To overcome these challenges, organizations need a solution that helps them control expenditure without compromising on the analytical needs of their users.
Kyvos helps enterprises cut down the querying costs on Snowflake. It reads the data from Snowflake and processes massive datasets across hundreds of dimensions. Using its AI-powered smart aggregation technology, it builds highly optimized data models. Once built, queries are served directly from the Kyvos layer instead of going back to Snowflake to fetch results. This minimizes query-time computations and ensures minimal resource consumption per query, helping save costs.
As all the required combinations are processed in advance, queries remain lightweight, irrespective of the data size.
With built-in intelligence to create smart aggregates based on data and query patterns, Kyvos enables enterprises to perform cost-effective analytics of any scale of data.
With Kyvos, both warm and cold queries are served equally fast and their runtime resource consumption is minimal. This helps save costs as well as delivers exceptionally high performance.
Besides, advanced OLAP capabilities ensure that complex analysis (such as YoY, cohort and other resource-intensive analysis) can be performed cost-effectively.
Kyvos helps enterprises cut down on the costs incurred due to higher concurrencies.
Our price-performant querying approach ensures that enterprises don’t have to rely on their data warehouse to serve every query fired by every user.
The cost per user decreases substantially and there is no performance degradation as concurrency increases. Enterprises can scale out for as many users as they need without impacting their bottom line.
The benchmarking study below demonstrates how Kyvos can help save up to 50% of the cloud data warehouse cost while delivering exceptionally high performance. Snowflake charges based on warehouse size and running time. Each query that comes to Snowflake will have to perform joins and aggregates. This results in higher query execution time and eventually, higher costs. Using Kyvos between Snowflake and BI tool, organizations can significantly reduce costs and improve query performance. In this scenario, the BI tool will connect with Kyvos rather than Snowflake so that the user can experience many other benefits of Kyvos apart from faster queries at lower cost.
The tests were carried out to analyze response time and cost for AdventureWorks (a standard sample dataset) queries. Two scenarios were compared where: in first case, BI queries were executed directly on Snowflake and in the second case, BI queries were executed on Kyvos, which was connected to Snowflake to create smart aggregates.
The following table gives details of the row count for each table used in this exercise
| Table Name | No. of Rows |
| FACTINTERNETSALES | 5,000,000 |
| DIMCUSTOMER | 10,018,484 |
| DIMGEOGRAPHY | 655 |
| DIMPRODUCT | 184,671 |
| DIMPRODUCT | 184,671 |
| DIMPRODUCTSUBCATEGORY | 1785 |
| DIMSALESTERRITORY | 11 |
| Snowflake Costs | |
| Snowflake cluster size X-Large | X-Large |
| Number of credits consumed per hour for X-Large size cluster | 16 |
| Snowflake cost per hour per credit | $3 |
| Total per-hour cost | $48 |
| Maximum number of concurrent queries per warehouse | 60 |
| Number of Snowflake warehouses required | 8 |
| Cost per hour | $376 |
| Cost per day | $4,512 |
| Cost per month | $99,264 |
| Total yearly cost | $1,191,168 |
Average query response time: 15-20 seconds.
| Kyvos Costs | |
| No of nodes (m6g.16xlarge – 64 cores and 256G RAM) | 10 |
| Yearly license cost per node | $50,000 |
| Cost of an AWS node per hour | $2.464 |
| Total hourly cost of nodes | $25.626 |
| Cost of 100 GB disk per month (EBS SSD gp2) | $10 |
| Cost per day (without EBS SSD) | $307.51 |
| Monthly cost (with EBS SSD) | $6,869 |
| Yearly hardware cost | $82,430 |
| Yearly license cost | $312,000 |
| Yearly Snowflake cost to refresh the model | $6,000 |
| Total yearly cost | $608,430 |
Average query response time: <5 seconds
When the data volumes on the cloud rise, enterprises often go off-budget to meet their BI needs. It’s high time to move on to a better approach that enables resource optimization and cost savings with no-compromise analytics on the cloud.
Moving your query workload to Kyvos enables you to analyze years of data across hundreds of dimensions and measures, with more users, all at a fraction of the cost. With Kyvos’ price-performant querying approach, enterprises can eliminate a lot of query processing costs from their BI architecture.
The combined capabilities of Snowflake and Kyvos, two powerful platforms, can help enterprises scale their cloud BI cost-effectively.