Is your feature request related to a problem? Please describe.
Recently some of my docker-volume-backup sidecars started getting killed by OOM. The issue seems to be a combination of low container memory limits (128 MiB), size of the backup archive, and buffer size used by the SSH backend. I've resolved this by increasing the memory limits relative to the archive size.
Describe the solution you'd like
Instead of copying the archive by sequentially reading a large (1 GiB) chunk of it into memory and then writing it to the remote file, consider using File.ReadFrom. I've found the recommendation on File.Write docs while digging through the source.
At a glance it seems that all the other backends, except Dropbox, upload the archive by some form of streaming. This would make the SSH backend behave similarly.
Alternatively, using io.Copy or drastically reducing the buffer size (<1 MiB) might be easier to implement.
Describe alternatives you've considered
Increasing the memory limits to at least 1 GiB and staggering individual backups by using BACKUP_JITTER, or individual schedules.
Additional context
I've started with the 128 MiB limits because the container idles around ~10 MiB, so it seemed to be more than enough of a margin
Is your feature request related to a problem? Please describe.
Recently some of my
docker-volume-backupsidecars started getting killed by OOM. The issue seems to be a combination of low container memory limits (128 MiB), size of the backup archive, and buffer size used by the SSH backend. I've resolved this by increasing the memory limits relative to the archive size.Describe the solution you'd like
Instead of copying the archive by sequentially reading a large (1 GiB) chunk of it into memory and then writing it to the remote file, consider using File.ReadFrom. I've found the recommendation on File.Write docs while digging through the source.
At a glance it seems that all the other backends, except Dropbox, upload the archive by some form of streaming. This would make the SSH backend behave similarly.
Alternatively, using io.Copy or drastically reducing the buffer size (<1 MiB) might be easier to implement.
Describe alternatives you've considered
Increasing the memory limits to at least 1 GiB and staggering individual backups by using
BACKUP_JITTER, or individual schedules.Additional context
I've started with the 128 MiB limits because the container idles around ~10 MiB, so it seemed to be more than enough of a margin