The architecture of Neki

* Neki is a sharding and scaling solution for real Postgres.
* PostgresManager manages starting/stopping Postgres, data directories, and replication configuration.
* The Sidecar pools connections for each Postgres instance.
* The Router accepts client connections and communicates with Postgres nodes via Sidecars.
* The Router reports instance health and primary/replica status to the cluster.
* Object Identifiers (OIDs) are used by Postgres to track objects internally.
* The authoritative shard translates custom type OIDs across shards for consistency.
* The Admin detects failures, promotes replicas, and maintains durability policies.
* The Operator manages the deployment, updates, and replacement of Neki components via a hierarchy model.
* The Router coordinates distributed queries, potentially executing joins itself across shards.

The architecture reveals a deep concern for decoupling operational concerns from the data layer while maintaining Postgres compatibility. The system distributes responsibilities across specialized layers—Manager, Sidecar, Router, Admin, Operator—to handle disparate functions like infrastructure management, connection pooling, query routing, failure response, and deployment lifecycle. A key tension exists between the desired state of a sharded database (logical consistency via OIDs and Data Topology) and the physical realities of distributed failure, managed by the Admin's failover mechanisms. The Replicator addresses the complex challenge of data migration during scaling events, introducing workflow concepts like MoveTables and OnlineDDL to facilitate atomic data relocation while maintaining query availability through buffering by the Router. This structure suggests a pattern where operational overhead is internalized into the control plane (Operator/Admin) so that application-facing concerns are simplified via the Router, even when underlying persistence involves complex distributed state management. The reliance on etcd for the authoritative Data Topology underscores an architectural commitment to a single source of truth for shard placement, which is foundational to resisting potential systemic drift in a highly distributed environment.
Bridge Questions: How does the cost or complexity associated with maintaining the consistency between OID translation across shards influence the overall latency profile of cross-shard queries? What are the emergent failure modes when the Operator's replacement process interacts concurrently with the Admin's failover coordination during a system-wide outage? What is the boundary between optimizing the physical state of Postgres (Admin) and ensuring the logical presentation of that state to applications (Router)?