I only have a small collection of bits in true regularly updated offline backup (my authentication/other DB, some important documents).
An extra protection against the possibility of malicious access such as ransom attacks is a two-step “soft-offline” backup.
The source machines backup to a central place, and the backup machines pull copies from there. The important part is that the source machines can not connect (or at least can not authenticate against) the final backups and vice-versa, so a malicious process/person getting into one can not affect the other and vice versa. Obviously if I don't notice the damage immediately then the most recent backups may be corrupted because damaged data was pulled, but past snapshots (taken after each pull at the backup side) will still be clean.
You have to be very careful about storing credentials to make sure source credentials don't ever end up on the backup machines and backup credentials don't ever end up on the source machines, even well out of the way of normal places like ~/.ssh, because a targetted attack might find them, but it gives almost the assurance of an offline backup while still being fully automatable. My backup site credentials are in my true offline backups, I need to refer to them for maintenance access, and then they get cycled after that use.
Restores can be mediated the same way. Verifying backups can be done by both sides running hashes on the files and posting the list back to the central machine for comparison - anything that differs without having a timestamp after the previous snapshot is likely corruption.
As a side note about corruption: if doing snapshots in the filesystem (“cp -al after rsync” or one of the many similar options) make sure you have more than one snapshot chains (on separate storage if you have resource for that). If you have a file that hasn't changed in years so every one of your snapshots points to that one version, it could only take one random filesystem error to completely lose the file.
An extra protection against the possibility of malicious access such as ransom attacks is a two-step “soft-offline” backup.
The source machines backup to a central place, and the backup machines pull copies from there. The important part is that the source machines can not connect (or at least can not authenticate against) the final backups and vice-versa, so a malicious process/person getting into one can not affect the other and vice versa. Obviously if I don't notice the damage immediately then the most recent backups may be corrupted because damaged data was pulled, but past snapshots (taken after each pull at the backup side) will still be clean.
You have to be very careful about storing credentials to make sure source credentials don't ever end up on the backup machines and backup credentials don't ever end up on the source machines, even well out of the way of normal places like ~/.ssh, because a targetted attack might find them, but it gives almost the assurance of an offline backup while still being fully automatable. My backup site credentials are in my true offline backups, I need to refer to them for maintenance access, and then they get cycled after that use.
Restores can be mediated the same way. Verifying backups can be done by both sides running hashes on the files and posting the list back to the central machine for comparison - anything that differs without having a timestamp after the previous snapshot is likely corruption.
As a side note about corruption: if doing snapshots in the filesystem (“cp -al after rsync” or one of the many similar options) make sure you have more than one snapshot chains (on separate storage if you have resource for that). If you have a file that hasn't changed in years so every one of your snapshots points to that one version, it could only take one random filesystem error to completely lose the file.