#Critique my archiving/backup strategy ##Preamble What I'm hoping to hear is if this approach would be useful to others, if similar software already exists, or if there are any obvious shortcomings that I've overlooked. I already have some proof-of-concept software (shell scripts) for much of what I discuss, so I'd like some idea whether or not I should go farther with it, or abandon it in favor of an established project on GitHub that aims to do the same thing. The main reason I'm even bothering is because I tried all the existing tools I could find (`git-annex`, `bup`, `borg`, etc.) and they all choked on my 1GB RAM Raspberry Pi test machine on even a small-ish file set of ~500K items taking less than 1TB of space. It seems like they all try to do just too much, so I tackled the problem in the classic Unix way of using small, dedicated tools. In that, part of my aim is also to create a declarative system that can be used portably (i.e., without requiring a special filesystem or otherwise having a file format for the archive that needs dedicated software to be usable). ##Catalog the file list I start by generating a `manifest` file, which is a hash of the file contents followed by the filepath (essentially the output of the `sha256sum` command). For the time being, I'm not concerning myself with other file metadata. With a single `manifest` I can easily find duplicates in my data set, as well as rename or remove entries. With multiple `manifest` files I can do other set operations, like joining them into a larger catalog, or finding duplicates across devices, or showing you what files one catalog has that another is missing. All of these actions can be done on just the `manifest` files themselves, allowing you to manage data in cold storage (or otherwise offline/offsite). Those operations could then be batch applied to the files once they were accessable, or alternatively the updated `manifest` could be sent to the remote system to apply the changes. ##Put files in a data warehouse The true utility of `manifest` files comes from the hash of the file contents. I create an archive of all the unique data simply by naming a copy of the file by that hash value. I call this archive a `destiny` (**manifest destiny**; get it?). To keep any one directory from becoming overwhelmed, I created a hierarchy several levels deep to store the files (e.g., Warehouse/cc/78/26/376d9edc57bd9a5d52a7e48fcbbf086236f8e28478defd890ff514f5e0). This is somewhat inspired by how `git` operates. It should be easy to see how you could then go in reverse and use the `destiny` along with a `manifest` file to restore the original filepath hierarchy. It should also be obvious how more than one `destiny` can be combined or incrementally extended. And from that, how different versions/snapshots of a filesystem can be recreated from a single `destiny` using different `manifest` files. The `destiny` is also easily verified independent of `manifest` files or custom software. All you have to do is re-hash a file and compare it to it's name. ##Conclusion I think I have a fairly portable, resource-lite structure. Fundamentally it's based on the philosophy of dealing with content data separate from metadata. Does this sort of approach interest anyone else? What important things would you want that are missing? Am I just reinventing someone else's wheel? 5272497dda2386b141e106d3205e9ab0df87280e68c65daa2248f98878eb7bf5