How we cut our vector search bill by nearly $10,000 a month with Lance

Earth Index searches geospatial embeddings to find what you’re looking for. Embeddings are numerical representations of what a satellite sees, generated by AI models. A vector search looks through all the embeddings in a search area and finds the embeddings closest to a specific query. Earth Index covers the whole world, so we have to keep a lot of data available to search through.

Until September, our vector searches ran on Atlas, three PostgreSQL shards behind a PgDog sharding proxy. Each shard held one region: the Americas, Europe and Africa, or Oceania and Asia. The shards used the VectorChord extension with RaBitQ indexes, which compress each vector into a compact code so the index is small and fast to scan. Atlas kept about 3.5 billion embeddings a year, for 2024 and 2025, on the local NVMe disks of three large hosts. It was incredibly performant at scale, but it cost about $12,000 a month.

We replaced it with Lance files on S3 and one server that keeps only the indexes locally on disk. The bill is now about $2,000 a month. This is how we built it.

Search without a database server

Lance is an open-source columnar format that stores vector indexes in flat files next to the data. Lance reads byte ranges straight from object storage such as S3, so the files need no database server. In order to run a search, you still have to pull a fair amount of index data, especially as the size of your dataset grows.

This summer we built a prototype that served Earth Index search from Lance files on S3, but we immediately ran into a problem: cold-start searches had to load so much index data that they were taking up to 30 seconds or more. But subsequent searches sped up dramatically, so we knew we were on to something if we could find a way to solve those cold starts.

After experimenting with various sharding approaches that we thought might reduce the amount of index data we had to load, we landed on a wonderfully simple solution. We realized that we could speed everything up by just storing the indexes locally on disk.

Enter: Lancelot, our Rust-based search service powered by Lance files on S3 with local copies of their indexes.

Indexes on NVMe, data on S3

Lance's vector indexes use a k-means clustering algorithm to group vectors into clusters around centroids. A search compares the query with the centroids and then scans only the closest clusters. As Lancelot's design notes put it, "The index is touched on every query while data is touched only for the final top-k rows."

So at boot, Lancelot copies every index file to the instance's local NVMe, the SSDs attached to the EC2 host. It leaves the data fragments on S3 and reads them only for the top results. In-memory caches sit in front of both. The indexes are about a third of the bytes. For the two years of embeddings data we serve at the moment, that is about 2.6 TiB on local disk. To make this split-storage approach work, we had to implement our own hybrid object store for Lance using a Rust trait injection hook. That lets us read indexes from the local disk and vector data from S3 without forking Lance itself.

In our testing, first searches dropped from over 30s to just a couple of seconds. We also put a cache in front of the S3 data reads, which brought warm searches from 83–170 ms to 29–31 ms on a single-shard benchmark.

One of the trade-offs is that a new instance of the search service starts with empty disks and has to download all the index data. We handle that with a blue-green deployment approach, setting up a new, ready instance with all the indexes copied before tearing down the old one.

Lancelot also shards its data by geohash. A geohash is a short code for a grid cell on the globe, and its first character divides the world into 32 cells. We split each year by that character into separate Lance tables, each with its own index. A search picks the tables that overlap its area of interest, which is usually just one or two. It queries them in parallel and merges the hits by distance. We split by geography because a random split would send every query to every table.

One server instead of three

In production, we cut over from Atlas to Lancelot in September 2026, and we shut Atlas down on September 15. Now, instead of running Atlas’s three beefy servers, we can get away with just one machine that’s actually smaller than any of the Atlas servers. We have about 10.8 TiB of Lance files on S3. But it’s a lot cheaper to house our data there than in a live PostgreSQL server.

At on-demand prices, Atlas cost $12,025.73 a month. Lancelot costs $2,053.60, made up of $1,794.05 for compute, $243.00 for S3 and $16.54 for the network load balancer. That is a saving of $9,972 a month. The savings come from no longer paying for three large hosts to keep everything on local disk. Only the indexes need fast local storage, and the rest sits on S3 for about $243 a month.

‍

At Earth Genome, we love finding new and exciting ways that cloud-native technologies can help make our work more efficient and more impactful. And when that comes with a massive cost cut, that’s not bad either!

Earth Index search now runs on Lancelot, for a fraction of what Atlas cost. Try it out at app.earthindex.ai and let us know how your searches go.