Skip to main content

Data Storage: A Primer

There are many options how one could store user information in a decentralized context. First, think of how storage and persistence is typically solved in permissioned environments like AWS RDS / S3 or ORM / SQL providers like Prisma, Neon or Supabase. If you were to start some patient record management system from scratch you'd define some database schema, write mappers (or let an ORM handle that for you) and attach query APIs like REST or Graphql. Rolling this out means migrating a certain object relational or document oriented schema to some infrastructure that you might or might not control yourself.

The Security Bottlenecks in Classical Databases​

At some point many applications, developers, users or product owners will stumpble upon an obvious question: who can actually access that data? And we're not talking about an authentication issue here; user information is usually sent over encrypted channels but could be intercepted at some debug log sink (when it's decrypted to be inserted into the database).

Not only your Kafka stream processor but also most relational databases keep around a write log with cleartext data that can be replayed to recover from failures or to provision replicas. Administrators with root access can gain read access to the filesystem or might even be able to unmount drives and read their data at the convenience of their own home.

Long ago database vendors started ideating good encryption concepts to avoid this situation, and the solution landscape addressing the fundamental security gaps has evolved significantly.

Modern Database Encryption: The State of the Art​

Contemporary database management systems employ a multi-layered encryption approach that addresses data protection at every stage of its lifecycle. Encryption in transit secures data moving between clients against man-in-the-middle attacks. Encryption at rest protects data when it's stored on disk. This encompasses several sophisticated techniques: transparent data encryption (TDE) encrypts entire database files, including data files, log files, and backup files, while column-level encryption allows for granular protection of sensitive fields like social security numbers or credit card information. File-level encryption extends protection to the operating system level, ensuring that even if someone gains access to the underlying storage, the data remains unintelligible without proper decryption keys.

Who holds the keys and knows the data?​

The encryption key management landscape has also matured considerably. Hardware Security Modules (HSMs) provide tamper-resistant environments for storing and managing encryption keys, while key rotation policies ensure that compromised keys can be replaced without data loss.

In cloud provider scenarios it's not uncommon to Bring Your Own Key (BYOK) model that allows organizations to maintain control over their encryption keys even when using cloud database services, addressing the fundamental trust issue of cloud providers having access to customer data.

Adding scifi cryptography​

Query-level encryption represents the cutting edge, where individual queries can be encrypted before reaching the database, and results can be returned in encrypted form. This approach, combined with homomorphic encryption techniques, allows for certain types of computations to be performed on encrypted data without decryption. Zero-knowledge proofs enable verification of data properties without revealing the underlying information, opening possibilities for privacy-preserving analytics and compliance verification.

Trusted Parties can Intercept Data​

Despite these advances, traditional encryption approaches still face fundamental limitations. Backup and replication systems often require decryption for processing, creating temporary vulnerabilities. Database administrators with sufficient privileges can still access decrypted data during maintenance operations.

Application-level encryption can provide additional protection, but it complicates querying and indexing, often requiring architectural trade-offs between security and performance. These limitations highlight why decentralized storage solutions, with their inherent cryptographic guarantees and elimination of trusted intermediaries, represent such a compelling alternative for truly sensitive data.

The Complexity of Decentralization​

The Beauty of Permissionless End to End Encryption​

This is where decentralized storage solutions come into play. They shift the paradigm from a traditional "trust but verify" model to a fundamentally trustless architecture. Permissionless end-to-end encryption eliminates the need for trusted intermediaries entirely - no database administrators, no cloud providers, no backup systems that could eventually hold keys to the plaintext. User data is encrypted with keys that only they control, stored across a distributed network where no single node has access to the complete, decrypted dataset.

This approach not only solves the fundamental trust issues of centralized systems but also provides censorship resistance - data cannot be taken down, modified, or accessed by any authority, regardless of their technical capabilities or legal jurisdiction. The result is a storage system where security is not a feature you configure, but a fundamental property of the architecture itself.

The solution domain is manifold, and it can be approached from different angles. Welshare chooses to rely on Nillion's "blindfold" key share-distribution idea and their TEE based compute and database engines. The next chapter explains how Welshare utilizes Nillion technology in detail.