Obsoleted IPFS + Filecoin & Mapped Out How to Leverage $ETH For Distributed File Storage

$ETH Aug 31, 2020
Preface: This was the product of me spitballing / musing to myself the other day. Giving this away because that's how simple this is.
(DHT + Kademlia + Sidetree Protocol Combination to Substitute IPFS as a File Storage Solution)
Premised Upon / Inspired By the Eth Light Client Code Repository and the Content in its README = https://github.com/ConsenSys/eth-lightwallet

Basic Idea / Description  of the Ethereum Light Client Project

^^ Allows you to protect your private keys within the context of the site or whatever its deployed. Very simple, just a javascript file with some other attachments appended to it as well.

This does not replace Metamask though.

According to the Project's Premise:

"Lightwallet is an HD wallet that can store your private keys encrypted in the browser to allow you to run Ethereum dapps even if you're not running a local Ethereum node."

Only uses BIP32 & BIP39 (we're looking for BIP38 for stronger wallet generation).

Additional Use (Important to Note)

"LightWallet is primarily intended to be a signing provider for the Hooked Web3 provider (like Metamask) through the keystore module."


"This allows you to have full control over your private keys  while still connecting to a remote node to relay signed transactions."

So this handles your wallet address rather than leaving it up to Metamask or another middleware Web3 wallet connector to serve that purpose for you ; this must be instantiated within the webpage as actual Javascript though.

Code Snippet For ETH Lite Client Integration in HTML / JS Webpages

The code they provide for someone looking to install this directly in their webpage:

    <script src="lightwallet.min.js"></script>

Sample Code That Provides a Pre-Configured Script to Interoperate with a User's Web3 Provider (Likely Metamask)

// the seed is stored encrypted by a user-defined password
var password = prompt('Enter password for encryption', 'password');

  password: password,
  // seedPhrase: seedPhrase, // Optionally provide a 12-word seed phrase
  // salt: fixture.salt,     // Optionally provide a salt.
                             // A unique salt will be generated otherwise.
  // hdPathString: hdPath    // Optional custom HD Path String
}, function (err, ks) {

  // Some methods will require providing the `pwDerivedKey`,
  // Allowing you to only decrypt private keys on an as-needed basis.
  // You can generate that value with this convenient method:
  ks.keyFromPassword(password, function (err, pwDerivedKey) {
    if (err) throw err;

    // generate five new address/private key pairs
    // the corresponding private keys are also encrypted
    ks.generateNewAddress(pwDerivedKey, 5);
    var addr = ks.getAddresses();

    ks.passwordProvider = function (callback) {
      var pw = prompt("Please enter password", "Password");
      callback(null, pw);

    // Now set ks as transaction_signer in the hooked web3 provider
    // and you can start using web3 using the keys/addresses in ks!

Tabling This For Now, But This Deserves Way More Inspection

There is a lot more that's included within this repo ($ETH Lite Client).

What they have here is setup for:

  1. Creating a 'keystore' where the private / public keys would be held at.
  2. The keystore in question would be encrypted (using a password + a salt on it); we need to see how we can outfit the code to either leverage Scrypt or Argon2id as the KDF (this would remove the redundant / unecessary step of including a salt to encrypt something differently). The code for the 'MimeWallet' literally outlines how to do this (argon2id for the password + pbkdf2 for the salt [provided for by the email]). Haven't looked under the hood at this to see the code to know for myself how the passwords are used to facilitate the encryption / decryption here - but doubtful that they use the same methods as the 'MimeWallet'.
  3. There's still a mnemonic phrase that's generated (this feels dangerous and this seems wholly unnecessary in the grand scheme of things). Why not use the BIP38 method of generating wallet addresses? That woudl ensure that no password has to be stored at all. If we integrate / combine this with the stateless password generator that I found a week or two ago, then this could be the ultimate authentication workflow because a user wouldn't even need to come up with their own password at that point - that app would handle it for them (but since its stateless & deterministic the user would just need to integrate the same unique credentials into the app in order to generate said password again.

^^^ As long as someone isn't dumb enough to outline through the credentials that they provide to the stateless password manager exactly what service the credentials are for (i.e,. blablah@gmail.com / 'Wells Fargo Banking Login' / etc.), then they should be good.

Need to look at that deployment again to get a better idea of how that works.

It has a "master password" that must also be used in tandem with the generated passwords that it spits out for users (which is mad fucking useful because this is almost like an additional hash of some sort).

If this is funneled directly into something like a 'Mimewallet', which is then transforming that output into a generated BItcoin / Ethereum / Monero / Litecoin address, then we have a super fucking bulletproof means of generating a REALLY secure wallet address that doesn't require the storage of private keys ANYWHERE ON planet earth.

The only thing that a user would need to remember is the master password to the stateless password manager in question that we're discussing here.

And we have endless password managers (not to mention the Hashicorp Vault which is basically built for shit like this + that can be provisioned alongside solutions like the Ethereum Lite Client Wallet).

Encrypted Messages?

Wait, didn't get to this final part where they dsecribe how their keystore vault solution (already built into the code repo) is ultimately intended to facilitate the encryption of messages through leveraging the Ethereum protocol itself (which is something that's super fucking unique and if done correctly on top of the protocol itself in some way, this could be leveraged as a messaging protocol of some sort).

That's Thinking Way Too Small About This

Etheruem protocol isn't designed to facilitate messages back and forth on top of its architecture.

However, when combining this with the Sidetree Protocol implementations (specifically the ones that integrate JWT token authentication schemes) + the Zero-Knowledge libraries that I'm finding (which already have deployable code designed to work with WASM (which means offline / client-side operation) - we could create an alternative network (something like Tor) that exists on top of Ethereum.

Quick Notes on This Fleeting Idea Before it Leaves Me:

  1. I'm thinking that IM (messages) won't work because that needs to be rapid and most times, stateful (i.e., you want it immediately and it matters if you're online or not).
  2. However, we can still encrypt things to another address on the protocol - so that address becomes akin to an endpoint of some sourt (i.e., URI resource)
  3. We can't relay this over the blockchain (directly on Ethereum) itself for obvious reasons - this would be way too massive (and bog down the chain infinitely with unwieldy gas fees).
  4. The Sidetree protocol is designed to create some sort of external signficance to an anchored hash on the protocol. Creating an ETH DID registry also creates an end point on the Ethereum protocol.
  5. That endpoint can be used to store private / public key pairs (a keystore) among other things.
  6. Back to the Ethereum Lite Client - all it does is encrypt a message to a given public key - it doesn't handle the actual delivery of said message. Neither does the Sidetree Protocol.
  7. However, we could create an alternative protocol / network built on top of Ethereum (or really any other blockchain like Ethereum), by leveraging DHT technology - just like Bittorrent. This network would not run on top of the blockchain or via a smart contract of any sort (that would be mad fucking dumb for obvious reasons). Rather instead, this will just be another network that users can access with an Ethereum client (that's what the Light Client would be really useful for).
  8. Rather than users seeding certain content / files on the network, connection to the network will require inherent support (content won't be hosted in one fixed location, but rather accessed by its URI endpoint - no different than a magnet URI).
  9. This is where the Sidetree Protocol would come in. Instead of using a magnet URI, users would use the DID URI that's generated (that URI will correspond with a resource that was hosted on the network).
  10. The creation of the URI endpoint should still involve the blockchain itself (because this will anchor the URI with whatever resource in question that needs to be accessed).
  11. ETH nodes that are plugged into our ETH-DHT network (pseudo-bittorrent) will use those URI endpoints to 'locate' certain resources (and that will be what ultimately provides any level of 'order' to what's being published on the network). This will also help us get around the issue of content being positively identified via metadata (which is probably an issue with alternative protocols).
  12. This can also remove the need for any kind of 'key server' due to the manner in which the DID registry URI endpoint will be integrated into the overall scheme.

In Relation to What I Stated in Point Twelve Above:

A) Creating a DID registry (if done for the purposes of this protocol) will only grant one ownership over that URI resource.

B) Once that URI endpoint is created, that user will copy / paste said endpoint and then provide it to our pseudo-DHT protocol (accompanied with whatever content that they are looking to upload to the network)

C) Before the network accepts content being uploaded under that URI identifier, the user must prove that they are the URI endpoint's original creator (obviously this can be done by signing whatever file / content is being uploaded). Upon verification, the network will confirm upload.

This Creates an Avenue For Monetization (Necessary For Sustainability)

This opens up the possibility for payment (and it should require it) because:

A) There must be some sort of quick way to auto-filter bogus requests to the network (i.e., DoS attempts). Normally this is taken care of by transaction fees on the regular blockchain, but we're talking about a different protocol here so we need a way to account for that.

B) Additionally, the network only has X amount of resources at a given point of time. There will of course be no upper bound limit to the amount of content that is stored on the network at any given time (it will be sharded and distributed per node as they are logged on to the network via DHT - specifically, using the famous Kademlia algorithm). However, there must be an upperbound limit on the network's throughput (i.e., how much content can be validated / verified at a given endpoint at a time).

This creates a situation where we're not literally transacting money of any sort, but we are attempting to upload content to the network under a certain URI resolver endpoint.

This is Where Another Gas Network Would Work Perfectly

We also should create yet another Proof of Work-based chain.

However, the big difference here is that this chain should be designed to favor ASIC mining equipment.

This is because in 'confirming' these uploads, we will need to hash the URI endpoint in a way that's digestible for the network nodes so that they can very quickly find requested URIs. This will also allow the network to feasibly shard and distribute content to other users.

'Blocks' May Not Be Necessary

Initially when I wrote this, I envisioned circumventing the need for a perpetual, long running blockchain with blocks appended in a chronological order containing data that can only be valid if the appended block does not contradict the previously established 'version of truth' (i.e., latest, valid version of the blockchain with the greatest 'Proof of Work' exhibited).

Thought by cutting out this facet of the proposed protocol I'm shelling out in these notes that I would make the final product more 'lean' (less overhead / technical debt).

Post-Modification of Mapped Out Plans, the Following Implementation Seems Like a Tentative 'Solution'

  1. Since we're leveraging Ethereum nodes, 'time frames' will be determined by the actual blockchain's block height.
  2. However, this is only for the sake of creating some sense of chronological time for the uploads on the network (to create a higher sense of ordering).
  3. There will be no blocks on the network though. Instead, when the upload is finished, the block height of the Ethereum network (as polled by the node ; this is their job , after all) will be included within the URI endpoint.

Mining Will Have to Be Different With This Prospective Protocol Design

  1. It can't just be hashing node results. Perhaps it could be those that facilitate the upload and propagation of the content.
  2. Its not as important that the content be propagated ASAP as it is with regular blockchain payments (you want that to go through as soon as possible for obvious reasons).
  3. I imagine that the content being uploaded here does not need to exist immediately (so perhaps we could do the same mining setup as what we see with normal blockchain payments).

There Should Be an Upper Bound Limit to the Volume of Data That Can Be Added to the Network

There actually should be a difficulty targeting algorithm (that adjusts), but not for the purposes of keeping a chain together (perhaps there will be a 'loose chain' that's created for the sake of ensuring that the data can be authenticated ; i.e., to prevent a bad actor from proposing an alternative version of the chain that re-distributes URI identifiers to malicious endpoints - like a payload for a massive virus or something like that).

Fuck it, we're just going to mirror what's already in existence.

Additional Ideas to Be Considered as Augmentations (Future-Facing)

Content could also be hosted this way too. It would be encrypted to an endpoint and decryption would be contingent on appropriate access.

That access could be distributed via a smart contract. If access is contingent on payment (and only payment ; nothing else matters for accessing that URI resource), then there can be tokens that are distributed to users from a smart contract.

The transaction hash of the token distribution event should serve as an anchor (using DID again) to generate a unique bearer authentication (API) for an entity looking to access said content.

The token will no longer be needed after it is distributed (this is just to generate the transaction hash) - but the wallet used to generate said token WILL be needed as it will be attached to the transaction ID / hash. Authentication / access to the encrypted resource should be a trivial matter at that point. This doesn't need to necessarily leverage any part of the DID / Sidetree protocol but if that's possible, that could make things more convenient.

Kilt.io appears to have created a scheme that essentially addresses this very use case. They're also open source as well so deployment is free and it can leverage all of the apps that are available on Polkadot for users that are looking to gather more information about the network (just deploying this by itself would be a high-level IT enterprise service that companies would honestly pay fucking bank for with ease...).



Happy to serve and help wherever I'm needed in the blockchain space. #Education #EthicalContent #BringingLibretotheForefront

Great! You've successfully subscribed.
Great! Next, complete checkout for full access.
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.