“Vitess for PostgreSQL” is a useful shorthand for the ambition behind Neki, but it significantly understates the architecture we have built. Neki carries forward what we learned from Vitess in a new sharding architecture designed around PostgreSQL’s internals and semantics from the start.
PlanetScale has spent years building and operating Vitess at scale. That experience has shaped how we approach query routing, data placement, and the challenge of making many database servers behave like one.
We have previously written about how Neki makes hundreds of database servers behave like one. Here, we take a closer look at the mechanism that controls where data lives and how queries are routed.
One of Neki’s core design goals is flexibility in data placement. There are many ways engineers need to map their logical database schemas onto physical shards in order to optimize for performance and scalability.
That configuration is expressed as a data topology.
What is a data topology?
Sharding spreads data across multiple servers, but it also creates a unique routing problem. How does each query find the shard or shards that hold the data it needs?
Without a routing layer, the application would need to keep track of where its data lives and choose the correct shard for each query.
A data topology is how Neki answers that question. It is a JSON configuration that describes a PostgreSQL sharding scheme by mapping logical PostgreSQL tables to groups of physical shards and giving routers the information they need to route queries.
A data topology has three building blocks:
- Shard indexes specify the column or expression Neki uses to route a row and how its value is transformed for routing. The selected column or expression is called the shard key, and its transformed value is the routing key that Neki uses to select a shard.
- Shard groups define a set of physical shards and assign a range of routing keys to each one. They also specify the default shard index for tables in the group. Related tables that use the same shard key can be colocated, allowing joins and transactions between them to stay local.
- Databases bind tables to shard groups, following the PostgreSQL hierarchy of databases, schemas, and tables. A table can name its shard group directly or inherit a default from its schema, its database, or the cluster.
Together, these building blocks tell Neki how to route queries.
The data topology does not provision or manage physical shards. Those shards are provisioned separately, and the topology refers to each one by a stable ID. In Neki, a shard consists of a PostgreSQL primary and its replicas.
Example of a data topology
Consider a store
database that partitions related data around its customers.
CREATE TABLE customers (
customer_id bigint PRIMARY KEY,
name text NOT NULL,
address text NOT NULL
);
Suppose the customers
table needs to be distributed across two shards. Here's how a data topology might look:
Shard indexes
The topology names this shard index customer_id_xxhash
. It tells Neki to convert each customer_id
into a routing key using xxhash
.
Neki supports three shard-index strategies:
- Hash-based indexes use a hash function to map a value into the routing space. Neki currently uses
xxhash
. - Modulo-based indexes divide an integer by a configured modulus and use the remainder.
- Range-based indexes use an integer value directly, without hashing it.
Each strategy takes a different path from customer_id
to a physical shard:
In this example, customer_id
is the shard key, and its value is 42
. Each strategy transforms its value into a different routing key. xxhash
produces 0x2e4fe982b68910ac
, which falls in the range assigned to customer-shard-1
. modulo
calculates 42 mod 4
, producing 2
, which falls in the range assigned to customer-shard-2
. range
uses 42
directly, which falls in the range assigned to customer-shard-1
. In each case, the shard group selects the physical shard whose range contains the routing key.
Shard groups
The customer_data
shard group divides the possible xxhash
results between two shards. This example focuses on hash-based routing, but shard groups work similarly with modulo-based and range-based strategies. Key-range boundaries are written as hexadecimal prefixes, so 80
expands to 0x8000000000000000
, the midpoint of the routing space:
customer-shard-1: 0x0000000000000000 through 0x7fffffffffffffff
customer-shard-2: 0x8000000000000000 through 0xffffffffffffffff
For example, these two customer IDs land on different shards because their hashes fall in different ranges.
customer_id 42 → 0x2e4fe982b68910ac → customer-shard-1
customer_id 1 → 0xf36b4a1a44f78bf3 → customer-shard-2
Neki first hashes the raw customer ID, then finds the range containing the resulting routing key.
Every data topology also names an authoritative
group, a single-shard home for database-wide metadata such as sequences and schema information. Queries against customers
still use customer_data
.
Databases
The databases
section follows the familiar PostgreSQL hierarchy of databases, schemas, and tables.
Most tables do not need to appear in the data topology. An unlisted table inherits the default shard group from its schema, then its database, then the cluster, allowing a topology to shard an entire database with one default and list only the exceptions.
Put together, the explicit entry for store.public.customers
maps it to customer_data
, where it uses the customer_id_xxhash
shard index and the group’s key ranges.
The foundation of Neki internals
On one side of a Neki router is the user's application, speaking the PostgreSQL wire protocol. On the other, the application's data can be spread across an ever-growing number of physical shards.
The data topology connects the two, informing the router how data is divided and which shard or shards should receive a query. The result is a live configuration that maps logical databases to physical locations. Neki updates it during resharding, table moves, and imports, all while the application continues serving traffic.
From here, we can explore the rest of Neki’s architecture and how it helps make 768 servers look like 1.
Running PostgreSQL at scale? Request access to Neki.