24/07/2024 | Press release | Distributed by Public on 24/07/2024 13:14
It's been a busy spring and summer here at Dell Technologies. In March, on the first day of spring, we launched the Dell Data Lakehouse. This turnkey solution features a powerful query engine powered by Starburst, which provides high-performance, high-concurrency access to distributed data, regardless of the data source. The foundation of Dell Data Lakehouse is built on Dell S3-compatible storage providing a highly performant, high availability storage layer to store and query data in open formats like Iceberg.
In today's fast-paced world, IT and data leaders face a tough challenge, accelerating analytics and AI while keeping costs in check. The tradeoff between speed and cost can prove expensive on either end. While adopting the data lakehouse offers performance advantages at lower costs, data engineering and IT teams still grapple with deciding which data to optimize and cache and which to leave as is in the data lake.
Consider this feedback from data teams:
This is why we're thrilled to introduce Warp Speed in the Dell Data Lakehouse for data on Dell's S3-compatible storage.
Warp Speed is a new feature in the Dell Data Lakehouse that autonomously learns query patterns and identifies frequently accessed data to create optimal indexes and caches while keeping infrequently accessed data where it is.
It delivers on the promise of accelerating query performance while keeping costs in the check. With Warp Speed, the same cluster can run data lake queries 3x to 5x faster without requiring any change in the query by the end user. It can also help reduce cluster sizes by up to 40%.
More simply put, organizations can run more queries on large clusters or run the same volume of queries on smaller clusters.
Warp Speed employs a combination of acceleration technologies to achieve its remarkable performance:
Starting July 17, Warp Speed will be available to all Dell Data Lakehouse customers and supported for those who are using Dell S3-compatible storage as their data lake. There is no change to the software license-this is now built-in. The configuration of the compute nodes will be modified to include SSDs that have been fully tested and benchmarked by Dell to support the Warp Speed index and cache.
Dell Data Lakehouse with Warp Speed sets a new benchmark in data lake analytics, empowering organizations to derive insights from their data more quickly and efficiently than ever before. Warp Speed unlocks the full potential of the Dell Data Lakehouse, paving the way for accelerated and budget-friendly innovation and growth in the AI era.
To get a full, hands-on experience, visit the Dell Demo Center to interactively explore the Dell Data Lakehouse with labs hand-picked for you by Dell Technologies' experts. You can also contact your Dell account executive to explore the Dell Data Lakehouse for your data needs.
And check out this blog to find out more about the latest release of the Dell Data Lakehouse!
Note on Performance Benchmarking
These performance benchmarks are based on testing conducted by Dell in July 2024 using TPC-DS 1TB and 10TB datasets stored on Dell ECS S3-Compatible Object Storage against a variety of Dell Data Lakehouse cluster sizes (6 workers, 11 workers and 16 worker nodes) with each worker node configured with 64 vCPUs and 256GB RAM. The TPC-DS benchmark queries cover a wide range of query scenarios including reporting, ad hoc and interactive.
Our results show that Warp Speed provides performance improvement generally across such scenarios, and between 3x to 5x for the top 20% of queries.
Compute savings are estimated by comparing the total queries executed per 10 mins by a 6- worker node with Warp Speed to that of a 11-worker node without Warp Speed and extrapolating how many Warp Speed-enabled nodes could provide the same level of performance as an 11-worker node cluster without Warp Speed. We repeated this test for both 1TB and 10TB datasets.