Backup time and throttling

It is a solution that I use with i-mscp and maybe it can help someone, or just parts of what I'm using. If someone thinks different, please ignore my post.

My dear @Delta04

Of course, I've no doubts about the purpose of your solution but the fact is that we are not talking about external solution implemented by hand here. We are talking about "How to make the ready-to use i-MSCP backup solution more viable on busy systems).

Hello ;

I've worked a few about the new implementation on my paper. Here come the result:

Daily backup task

The daily backup will be a cron task. This task will be responsible to do an incremental backup of a specific domain using the rdiff-backup tool, and put the resulting data either on a local directory¹ or a remote server using rsync².
This backup task will not create any archive³ for the end users (customers). This will be simply the daily incremental backup (with or without compression) of the domain data.

Each domain will have it own daily backup task run at different time, which includes incremental backup of any data related to the domain:

• Metadata: This is the data which are pulled from the database (domain properties, htaccess, sql users, mail users, ftp users...)
• Web data: This is the web data which belong to the domain
• Mail data: This is the data of mailboxes which belong to the domain
• SQL data: This is the SQL databases data (dumps) which belong to the domain

Note: Here, a domain refers to a single domain (either a domain, a domain alias or subdomain). This is a major difference with the current i-MSCP backup feature implementation which does an all-in-one backup. Also, you can see that now, we'll save all data (including those from the i-MSCP database)

Backup archive requested by a customer through the GUI interface

As stated above the daily backup task will not produce any backup archive. Instead the backup will be processed incrementally.

The backup action as provided through the GUI, will allow the customer to request a backup archive for a specific domain (this archive will be the actual state of the last incremental backup including full data).

When a customer will request for a backup archive, a new task will be put in the i-MSCP job queue for processing and then, when this task will be achieved, the customer will be noticed. The resulting backup archive will be available through the usual backup directory (inside the customer home directory).

It must be noted that the administrator will be able to set a limit for the maximum number of requests that a customer can do (per day and per domain). This will prevent abuses.

Backup restoration requested by a customer through the GUI interface

A customer will be able to request a restoration of its backup archives through the GUI interface. For this, the customer will have to upload the backup archive in the usual backup directory. Once this task will be requested, a new job will be added in the i-MSCP job queue. Finally, when the restoration task will be finished, the customer will be notified.

It should be noted here that a restoration will remove any current data (including database data).

Note: Each archive will be signed to mitigate any compromission and the backup manager will keep trace of those signatures.⁴

• This will allow to mitigate CPU time and disk access problems because no archive will be created. Also, the admin will be able to setup the local directory path (eg, on another disk).
• This will allow to mitigate CPU time and disk access problems.
• Same as above, this will allow to mitigate the CPU time and disk access because the backup archive will be created only when really needed.
• This will allow to mitigate security problems about backup archive which can be compromised by customers

Any suggestion?

@Smallserver

You think too fast...

The bandwidth problem can occur only in case of a remote server usage. Anyway, we can set the --bwlimit option with rsync. Furthermor, the NAS servers (remote server) are generally on the same network and so, the synchronization should be fast. We can also minimise the resource consumption with the job queue