A project from the future and fantastic readme to boot at https://github.com/dnr/styx
It's essentially a layer of the nix package manager that uses linux userspace file-system mechanisms (but not FUSE) to download data on demand as they're read. As in at posix read time.
And not whole files, mind you, but chunks of files. I.e. segments of a file that are common across others will be deduplicated on disk. Not only that but large files can be partially downloaded as needed.
For example, I rarely use most of the fonts bundled in the widely used google noto fonts package which is above 1GiB on disk right now with the version I'm using. A lot LLM related packages like the cuda family and ollama are pretty big as they tend to bundle together code for each hardware/architecture that they support in one package.
On the other hand, nixos upgrades tend to leave lots of versions of the same package in the store. This is such a big issue that I'm usually forced to "garbage collect" the store or update flakes of unrelated projects in lock-step to benefit from version dedup. The chunking aspect would help here a lot as for most packages, different versions are mostly identical chunkwise.
Hearsay against #IPFS has me suspicious of the metadata overhead but intuition says package management is a good usecase for this.
I was previously aware of https://github.com/containers/composefs-rs which tries to do something similar for OCI images but very cool to run across this despite how...early days it is.