so we've actually had to turn off the use of our s3 cache in our nix deployments, for now
a couple months ago we did a lot of work that made it appear to work reliably, so we turned it on everywhere
then we got to the next substantive rebuild (that involved more than a couple packages) and discovered that builds now spend about an hour checking the cache for packages before attempting to build any the normal way