Data collaboration archetypes

2024/10/30

There are a few ways to organize around data problems.1

Centralized: data warehouses

Historically, the barriers to working with data meant you needed lots of resources, effectively limiting it to institutions. This is the most straightforward to monetize and still largely the shape of the data ecosystem today. Data vendors sell into organizations top-down, emphasizing uniformity and centralized administration. ETL systems consolidate all of our data into singular data warehouses. This attempt at highly structured systematization is usually an ongoing pipe dream and requires heavy coordination across an organization.2

Distributed: web3

Some have pushed back against these centralized systems and pursued the complete opposite through blockchains (and other Merkle trees) and distributed data stores.3 These come at great costs though. Distributed systems, while robust and scalable, are much less efficient.4 They rely on consensus of complicated protocols, making them time consuming to change and generally lacking agility to add new functionality.

Decentralized:5 community-supported data

Resourceful individuals have found interesting ways to build datasets in public.6 OpenFreeMap is developed and paid for by one person (of course built on other open technologies and datasets). Hosting costs have become manageable enough that motivated people can donate the storage and computation for non-trivial data projects.

By continuing to push down the economics of data, we can lower the barriers to allow motivated individuals to accomplish even more. The value of open source code eclipses even the most valuable corporations, and there’s no reason open data shouldn’t be even more valuable.

There are still complicated coordination challenges to overcome,7 but I believe this is the future of data. It can’t come soon enough.

  1. A parallel can be drawn between each of these models of data organization and the three classic network architectures. ↩︎

  2. Other inadequacies of the traditional data warehouse have been highlighted in explanations of data mesh. ↩︎

  3. Examples of distributed storage technologies include IPFS and Filecoin. ↩︎

  4. Inefficiencies of distributed systems have been highlighted by DuckDB, which is able to perform many analytics on a single machine that previously required a distributed computing cluster. ↩︎

  5. Or “federated.” ↩︎

  6. For example, the git-based data strategy used by Simon Willison. ↩︎

  7. This is why I’m working on knit. ↩︎