Multiple backups

Quote from cbxk1xg

I wrote a patched version of the imscp backup script. I modified the script to keep the backups for seven days.

The original file is located at: /var/www/imscp/engine/backup/imscp-backup-all
Make a backup of the file, before you patch it.

Perl

1. #!/usr/bin/perl
4. # i-MSCP - internet Multi Server Control Panel
5. # Copyright (C) 2010-2013 by internet Multi Server Control Panel
6. #
7. # This program is free software; you can redistribute it and/or
8. # modify it under the terms of the GNU General Public License
9. # as published by the Free Software Foundation; either version 2
10. # of the License, or (at your option) any later version.
11. #
12. # This program is distributed in the hope that it will be useful,
13. # but WITHOUT ANY WARRANTY; without even the implied warranty of
14. # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15. # GNU General Public License for more details.
16. #
17. # You should have received a copy of the GNU General Public License
18. # along with this program; if not, write to the Free Software
19. # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
20. #
21. # @category i-MSCP
22. # @copyright 2010-2013 by i-MSCP | http://i-mscp.net
23. # @author Daniel Andreca <[email protected]>
24. # @link http://i-mscp.net i-MSCP Home Site
25. # @license http://www.gnu.org/licenses/gpl-2.0.html GPL v2
28. use strict;
29. use warnings;
32. use FindBin;
33. use lib "$FindBin::Bin/..";
34. use lib "$FindBin::Bin/../PerlLib";
37. use iMSCP::Debug;
38. use iMSCP::Boot;
39. use iMSCP::Execute;
40. use iMSCP::Mail;
41. use iMSCP::Dir;
42. use Data::Dumper;
45. newDebug('imscp-backup-all.log');
48. our $command = {
49. pbzip2 => {extension => 'bz2', command => 'CMD_PBZIP'},
50. bzip2 => {extension => 'bz2', command => 'CMD_BZIP'},
51. gzip => {extension => 'gz', command => 'CMD_GZIP'},
52. lzma => {extension => 'lzma', command => 'CMD_LZMA'},
53. xz => {extension => 'xz', command => 'CMD_XZ'}
54. };
57. sub backup_start_up {
60. umask(027);
61. iMSCP::Boot->new()->init({nolock => 'yes'});
64. 0;
65. }
68. sub backup{
71. my $rs = 0;
72. my ($stdout, $stderr);
75. if ($main::imscpConfig{'BACKUP_DOMAINS'} ne 'yes') {
76. return 0;
77. }
82. unless ($main::imscpConfig{ZIP} =~ /^(pbzip2|bzip2|gzip|lzma|xz)$/i) {
83. my $msg = "Backup algorithm not supported: $main::imscpConfig{ZIP}!";
84. error($msg);
85. return 1;
86. }
89. my $sql = "
90. SELECT
91. *
92. FROM
93. `domain` AS t1,
94. `admin` AS t2
95. WHERE
96. t1.domain_status = 'ok'
97. AND
98. t1.domain_admin_id = t2.admin_id
99. AND
100. t1.allowbackup != 'no'
101. ";
102. my $rdata = iMSCP::Database->new()->factory()->doQuery('domain_name', $sql);
105. if (ref $rdata ne 'HASH'){
106. error("$rdata");
107. $rs = 1;
108. } else {
111. for (keys %$rdata) {
112. iMSCP::Boot->new()->lock();
113. if ($rdata->{$_}->{allowbackup} !~ m/^no$/i) {
116. my $hDir = $main::imscpConfig{USER_HOME_DIR}."/$_";
117. my $bkpDir = "$hDir/backups";
120. my $rv |= iMSCP::Dir->new( dirname => $bkpDir)->make({
121. user => $main::imscpConfig{ROOT_USER},
122. group => $main::imscpConfig{ROOT_GROUP},
123. mode => 0755
124. });
125. $rs |= $rv;
126. if($rv){
127. iMSCP::Boot->new()->unlock();
128. next ;
129. }
132. # cbxk1xg patch to save backups for 7 days.
133. $rs |= execute("find $bkpDir/* -maxdepth 0 -type f -mtime +7 -print | xargs -r /bin/rm -v", \$stdout, \$stderr);
134. # $rs |= execute("$main::imscpConfig{'CMD_RM'} -rf $bkpDir/*", \$stdout, \$stderr);
135. debug($stdout) if $stdout;
136. error($stderr) if $stderr;
139. use POSIX qw(strftime);
140. my $ext = $command->{lc($main::imscpConfig{ZIP})}->{extension};
141. my $date = strftime "%Y.%m.%d-%H-%M", localtime;
144. if ($rdata->{$_}->{allowbackup} =~ m/full|dmn/i) {
147. my @cmd = (
148. lc($main::imscpConfig{CMD_TAR}),
149. "--create --directory=$hDir",
150. "--use-compress-program=" . lc($main::imscpConfig{ZIP}),
151. "--file=$bkpDir/$_-backup-$date.tar.$ext",
152. "--exclude=logs",
153. "--exclude=phptmp",
154. "--exclude=backups",
155. "--exclude=domain_disable_page",
156. ". "
157. );
160. my $rs |= execute(join(' ', @cmd), \$stdout, \$stderr);
161. debug($stdout) if $stdout;
162. error($stderr) if $stderr;
165. my $file = iMSCP::File->new(filename => "$bkpDir/$_-backup-$date.tar.$ext");
166. $rs |= $file->mode(0666);
167. $rs |= $file->owner($main::imscpConfig{ROOT_USER}, $main::imscpConfig{ROOT_GROUP});
168. }
171. if (
172. $rdata->{$_}->{allowbackup} =~ m/full|sql/i
173. &&
174. iMSCP::Database->new()->factory()->can('dumpdb')
175. ) {
178. $sql = "SELECT `sqld_id`, `sqld_name` FROM `sql_database` WHERE `domain_id` = ?";
179. my $dbData = iMSCP::Database->new()->factory()->doQuery('sqld_name', $sql, $rdata->{$_}->{domain_id});
182. if (ref $dbData ne 'HASH'){
183. error("$dbData");
184. $rs |= 1;
185. iMSCP::Boot->new()->unlock();
186. next;
187. }
190. for (keys %$dbData) {
193. my $date = strftime "%Y.%m.%d-%H-%M", localtime; # cbxk1xg patch to include the date in the file name
196. my $cmd = $main::imscpConfig{$command->{$main::imscpConfig{ZIP}}->{command}}." --force \'$bkpDir/$_-backup-$date.sql\'";
197. $rs |= iMSCP::Database->new()->factory()->dumpdb($_, "$bkpDir/$_-backup-$date.sql");
200. my $file = iMSCP::File->new(filename => "$bkpDir/$_-backup-$date.sql");
201. $rs |= $file->mode(0666);
202. $rs |= $file->owner($main::imscpConfig{ROOT_USER}, $main::imscpConfig{ROOT_GROUP});
205. $rs |= execute($cmd, \$stdout, \$stderr);
206. debug("$stdout") if $stdout;
207. error("$stderr") if $stderr;
208. error("Can not dump $_") if (!$stderr && $rs);
209. }
210. }
211. }
212. iMSCP::Boot->new()->unlock();
213. }
214. }
217. my @warnings = getMessageByType('WARNING');
218. my @errors = getMessageByType('ERROR');
219. my @all = (@warnings, @errors);
222. iMSCP::Mail->new()->errmsg("@all") if(scalar @all > 0 );
225. $rs;
226. }
229. my $rs = 0;
232. exit 1 if backup_start_up();
235. $rs |= backup();
238. exit $rs;

Display More

Hello ;

The problem is that we have some customers with bunch of data. Imagine an user with a total of 5 Go to backup. So 5 * 7 = 35 Go... Now let imagine many users with such amount of data to backup...

The whole script must be rewritten but before, we must take some decisions on how to re-implement backup feature. From my side, I always thought that any backup task should be left to the administrator care. Any solution implemented directly in i-MSCP will never correspond to all administrators needs.

Thank you for using i-MSCP.

Not for User's you're right, but i don't know any Server which have unlimited HDD Space

ANd they we are back at this point Nuxwin Explained

If we ignore the other problems and the HDD size is the only problem, you can use duply as backup engine. It's an incremental backup tool with backup encryption, so if the user not change the whole content all day, only the first backup's size is 5GB and the rest is a couple of megabytes, because it's same as in the first backup. So you can store the backup for 30 days, and make a full backup every month. I use this method on all of my servers. Only the SQL needs to be saved all time (Because without binlog replication it's not support incremental backups).

yes, ideally we can work duplicity into being the backup engine, and incorporate archiving, and allow people to restore any day of the backup. duplicity is the best rsync + compression option so far.