Database benchmarks are imperfect. They are also useful.
No benchmark can tell you exactly how a database will perform for your application. Workload shape, data size, region placement, storage, configuration, and cost all matter. But fair benchmarks help customers understand tradeoffs, compare options, and ask better questions before choosing infrastructure.
The DeWitt clause
Many cloud vendors include language in their terms that restricts comparative benchmarking. These restrictions are called "DeWitt clauses", named after database researcher David DeWitt. That is a strange legacy for someone whose work helped move the database industry forward by measuring real systems and publishing results.
Previously, PlanetScale also included a DeWitt clause in our Acceptable Use Policy (AUP). Today, we are removing this in favor of a more open "Benchmarking" section in our AUP.
The new section reads:
You may perform benchmark tests (“Benchmark”) of the Services, provided that the Benchmark is conducted in good faith and uses a fair and transparent methodology. Please refer to PlanetScale's published benchmarking best practices. Except with respect to Beta Features, you may disclose the results of the Benchmark. If you perform or disclose, or direct or permit any third party to perform or disclose, any Benchmark of the Services, you (i) will include in such disclosure, and will disclose to PlanetScale, all information necessary to replicate such Benchmark, and (ii) agree that PlanetScale may perform and disclose the results of Benchmarks of your products or services, irrespective of any restrictions on Benchmarks in the terms governing your products and services.
Any Benchmark must be conducted in accordance with the Agreement, including this Acceptable Use Policy. The Benchmark must not interfere with the Services or misrepresent the configuration, methodology, results, or cost of the Services or any compared service.
Benchmarks have gained a bad reputation because they are frequently conducted poorly. Sometimes this is done with malicious intent, often referred to as "benchmarketing." At other times it is done out of ignorance. Many engineers are not trained in all aspects of fair benchmarking.
A new standard
Anyone benchmarking PlanetScale should follow the best practices outlined in our benchmarking guide. These practices come from our deep experience benchmarking databases in the cloud where topology, server location, region, workload, and instance type differences materially impact the result.
We encourage other vendors, analysts, and practitioners to use the same standard. Benchmarks should be deep, thorough, technically sound, and transparent enough for others to understand and reproduce.
Our ask
We invite other vendors to adopt this same language and standard in their own AUPs. Allow public benchmarking, remove DeWitt clauses, and hold benchmarks to clear expectations for fairness and transparency.
Customers should be able to compare the systems they rely on.
Facts Only
PlanetScale has removed a "DeWitt clause" from its Acceptable Use Policy (AUP).
The DeWitt clause previously restricted comparative benchmarking of PlanetScale's services.
The new AUP includes a "Benchmarking" section allowing benchmark tests under specific conditions.
Benchmarks must be conducted in good faith with fair and transparent methodology.
Results of benchmarks can be disclosed, except for Beta Features.
Disclosures must include all information necessary to replicate the benchmark.
PlanetScale reserves the right to benchmark competitors' products, regardless of their terms.
Benchmarks must not interfere with services or misrepresent configurations, methodologies, results, or costs.
PlanetScale has published benchmarking best practices based on its experience with cloud databases.
The company encourages other vendors to adopt similar benchmarking policies.
DeWitt clauses are named after database researcher David DeWitt.
Many cloud vendors include DeWitt clauses in their terms of service.
Executive Summary
PlanetScale has revised its Acceptable Use Policy (AUP) to remove a "DeWitt clause," which previously restricted comparative benchmarking of its database services. The new policy explicitly permits benchmarking, provided it is conducted in good faith with transparent methodology and full disclosure of replication details. The company also reserves the right to benchmark competitors' services, even if their terms prohibit it. This shift reflects a broader industry debate about the role of benchmarks in evaluating cloud databases, where factors like workload, configuration, and cost significantly impact performance. While benchmarks are often criticized for being misleading or poorly executed, PlanetScale argues that fair, transparent comparisons help customers make informed decisions. The company has published benchmarking best practices and encourages other vendors to adopt similar policies, advocating for an industry standard that prioritizes openness and reproducibility.
The move highlights tensions between vendor control and customer autonomy in cloud infrastructure. DeWitt clauses, named after database researcher David DeWitt, have been controversial for limiting independent performance evaluations. PlanetScale's policy change positions it as an advocate for transparency, though the effectiveness of this approach depends on widespread adoption and adherence to rigorous benchmarking standards. The announcement acknowledges that benchmarks are imperfect but emphasizes their utility when conducted responsibly, framing the issue as one of customer empowerment in a complex technical landscape.
Full Take
PlanetScale’s policy shift is a strategic move to position itself as a proponent of transparency in an industry often criticized for opacity. The removal of the DeWitt clause is a direct challenge to the status quo, where vendors frequently restrict independent benchmarking to control narratives about performance. This aligns with a broader push for customer-centric practices, but it also raises questions about whether such policies will be adopted widely or remain isolated gestures. The company’s insistence on reproducibility and methodological rigor is commendable, yet the effectiveness of these standards hinges on enforcement and peer scrutiny—something the industry lacks a formal mechanism for.
The deeper implication here is about power dynamics in cloud infrastructure. Vendors historically wield significant control over how their services are evaluated, often using legal clauses to suppress unfavorable comparisons. PlanetScale’s policy disrupts this by not only allowing benchmarks but also asserting the right to benchmark competitors, even if their terms prohibit it. This could set a precedent, but it also risks escalating tensions if other vendors retaliate with stricter clauses or legal challenges. The move reflects a growing recognition that customers—especially enterprises—demand verifiable data to justify infrastructure decisions, particularly as costs and performance trade-offs become more complex.
However, the announcement leans heavily on the assumption that benchmarks can be made "fair" and "transparent" through best practices alone. While this is a step forward, it overlooks systemic issues like vendor-specific optimizations, proprietary configurations, and the inherent difficulty of apples-to-apples comparisons in distributed systems. The call for industry-wide adoption is aspirational, but without collective agreement on standards, it may devolve into a marketing arms race where each vendor claims superiority based on selectively favorable benchmarks.
**Patterns detected: none**
**Bridge questions:**
How might competitors respond to PlanetScale’s policy, and could this lead to a fragmentation of benchmarking standards rather than unification?
What mechanisms could ensure that benchmarks are not only transparent but also methodologically sound across different vendors?
If benchmarking becomes a competitive tool rather than a neutral evaluation, how can customers distinguish genuine performance insights from "benchmarketing"?
Sentinel — Human
The text reads as a well-structured advocacy piece, exhibiting a human-like persuasive voice grounded in specific technical and legal concepts, making synthetic generation unlikely.