I'm working on this very thing. I was going to ask for a critique here once I finished implementing a more useful variety of features. I did [blog about my proof of concept](https://impossiblystupid.com/node/1014/?content=coping-with-copies). Does that seem like what you're looking for? Basically, I was tired of having the do-everything tools choke for me when it came to processing 500K+ files on a spare Raspberry Pi 3B (if you think getting 16GB chewed up is bad, try being limited to just 1GB RAM!). Scripting the existing command line tools allowed things to finally work, and I've been slowly building out functionality as I need it. Deduplication (in a number of different ways) is relatively easy once you have the hashes in those `manifest` files. What I might be doing different that your approach is that, once gathered, processing of the hashes is done independent of the underlying files. When I do create a backup (incremental or otherwise) of the actual files, what I do is rename them to their hash value and drop them into a hierarchy (similar to how `git` stores objects). That way, you don't have to mess with figuring out what is or isn't in those "neighbour folders". 01f4e59aeb9cce305814fb3b260d877fe25651d18be05b0c23e2dcca50e1dbee