Spotify、データレイク上で低遅延ポイントクエリを可能にする外部インデックス構築

Spotify has introducedRandom Access Parquet (RAP), a storage architecture that enables low-latency point queries directly against data stored in its data lake, allowing online services and AI applications to retrieve individual records without replicating datasets into operational databases. RAP adds an external indexing layer over Apache Parquet files, enabling interactive lookups while continuing to use the same datasets for analytics, machine learning, and online serving.

Spotify explained that modern data lakes have become the central repository for analytical and AI workloads, but retrieving individual records remains inefficient because distributed query engines such as Trino andBigQuery are optimized for analytical scans rather than key-based lookups. Although cloud object stores such asGoogle Cloud Storage now provide millisecond access latency, query planning, metadata traversal, and file discovery can add significant overhead for point queries. Spotify noted that it stores petabytes of online data in Bigtable while exabytes reside in its Google Cloud Storage-based data lake, making large-scale replication into serving databases increasingly costly.

RAP addresses this challenge by introducing an external index that maps lookup keys, such as user IDs, directly to Parquet files and row locations. Instead of scanning thousands of files, a query resolves the key through the index before issuing a targeted ranged read against object storage. As new data is written intoApache Iceberg tables, an index builder generates append-only index fragments without modifying immutable Parquet files. Spotify said the approach allows the same datasets to support analytical processing, machine learning pipelines, notebooks, AI agents, and latency-sensitive online applications without maintaining duplicate storage systems.

Spotify's announcement follows broader efforts to extend open data lake technologies beyond analytical processing. Google Cloud recently described anApache Iceberg based lakehouse architecture for AI applications that similarly seeks to reduce data duplication while enabling operational access to data. Unlike that approach, RAP introduces a dedicated external indexing layer optimized for point lookups while remaining compatible with existing Parquet files and Iceberg tables.

The architecture also generated discussion within the data engineering community.Andrew Lamb highlighted RAP as an example of extending open data formats for interactive workloads. In a separate LinkedIn discussion, Vikas Singh argued that improvements in cloud object storage performance have shifted more of the latency associated with point queries toward query planning and metadata access, an area that RAP is designed to reduce through precomputed indexes.

Spotify also described several storage layout optimizations that reduce point query latency. These include sorting data by lookup key to reduce the number of files accessed, grouping related records together, interleaving value columns so multiple attributes can be retrieved through a single contiguous read, and using covering indexes that can satisfy some queries without reading Parquet files. According to Spotify, these techniques trade modest increases in file or index size for fewer storage operations, allowing some point queries to be served through a single ranged read of only a few kilobytes.

Spotify also supports secondary indexes, enabling efficient querying across multiple lookup dimensions, such as buyer ID or seller ID, without rewriting Parquet files. Hash-based indexes support exact lookups, while sorted indexes enable range queries. Spotify said secondary indexes are managed at the serving layer, allowing new access paths without changing data pipelines while continuing to use the same Parquet datasets for both analytical scans and interactive point lookups. Storage layout techniques such asZ ordering andHilbert curves can further improve data locality for secondary lookup dimensions.

About the Author

Leela Kumili

Leela is a Lead Software Engineer at Starbucks with deep expertise in building scalable, cloud-native systems and distributed platforms. She drives architecture, delivery, and operational excellence across the Rewards Platform, leading efforts to modernize systems, improve scalability, and enhance reliability.

In addition to her technical leadership, Leela serves as an AI Champion for the organization, identifying opportunities to improve developer productivity and workflows using LLM-based tools and establishing best practices for AI adoption. She is passionate about building production-ready systems, enhancing developer experience, and mentoring engineers to grow in both technical and strategic impact. Her interests include platform engineering, distributed systems, developer productivity, and bridging technical solutions with business and product goals.

Show moreShow less