new backup script

    Hi guys


    I would like to put the question here before opening a ticket.


    The rewritten (somewhere before beta 5) backup script works slightly different than before. The biggest difference seems to me that the old backups are removed before the new backup is created + finished.


    This leads to the question if this is a good idea. I often was happy to have the backup from yesterday ready when the backup from today could not finish because tar threw an error (changing files during backup etc)...


    What do you think?


    Koxi

    That is something obvious! If he isn't working like that, the backup system is useless.


    A backup system (automated) should never delete a backup before ending the current one.


    I also want to see those options for backup system:
    - How many backups to keep.
    - Complete and differencial backups.
    - What to backup (only mail, only web, all)
    - Domain backup/Full Server backup (for the full server backup when backing up the domains go backup domains or backup only the backups from the domains)
    - Restore option for the domain and for the whole server.

    Hi


    well - of course it can be built up to a general backup/restore system for the whole server and all the other things - but I just wanted to discuss the basics - no removal of older backups if the new backup is not finished.
    This used to work like this until summer... with beta5 many engine scripts were rewritten and this detail has changed.


    imscp-backup-all:
    line 108: $rs |= execute("$main::imscpConfig{'CMD_RM'} -rf $bkpDir/*", \$stdout, \$stderr);
    line 130: my $rs |= execute(join(' ', @cmd), \$stdout, \$stderr);
    where $cmd is the tar command....


    I know that if during the backup the (typo3) website is being visited, then it often happens that tar ends with a message
    (/bin/tar: ./htdocs/typo3temp: file changed as we read it) which is recognized as error - new backup is removed and a mail is sent.
    At least it seems that now this type of "error-"backup is not deleted - and should be usable.


    To the other backup wishes:


    - How many backups to keep.
    -> imscp backups are kept for 14 days.
    - Complete and differencial backups.
    -> hm... So this looks rather like you want to integrate a full backup software into imscp
    - What to backup (only mail, only web, all)
    -> maybe an option... - but then with: "mail" every hour, "web" once a day etc...
    - Domain backup/Full Server backup (for the full server backup when backing up the domains go backup domains or backup only the backups from the domains)
    - Restore option for the domain and for the whole server.


    For now a rsync backup makes sense for the parts of the server you wish to have backed up (without the backups made by imscp). For an archive you may need a more sophisticated software.


    I included a small feature to let the user define some files/folders which should not be backed up - e.g. for people who have a copy of a dvd on the server...


    /J

    Hi,


    first, I totally agree with Joximu. The previous backup must be deleted when only after the new one had be created successfully. Think of a power outage or so, where the new backup is not being dumped out. Then we are left with nothing.


    Okay, for the full view... I'd like to partition it into the different work areas of the system:
    Server hardware:
    - use backup software + RAID + UPS or whatever, but this is absolutely out of the scope of the software. Think of service updates (e.g. MySQL) which break some startup script. Should i-mscp daemon monitor this and allow for recovery? To me an absolute no.


    Control panel:
    - this resides in the i-mscp database. We have transactions to keep the thing clean. Either dump the db before every change (impractical) or keep daily backups for 14 days (enough for my needs).


    Servers (the machines running postfix, dovecot, apache, ...):
    - we have that /etc/imscp directory where working configs are generated based on the database changes. So these could be moved/copied before every change. I don't know if the system generates ALL/every configs after any change or if it recognizes individual changes (e.g. mailbox added -> update postfix/dovecot or courier only). I would like to have review possibility in a folder structure based on changes made:
    2011-12-13/19-21/(configs from that date)
    2012-01-04/12-07/(configs from that date)


    then it would be very easy to review changes to the server configuration. When were they made? What changed? Perhaps a log from manager could also be a good idea. Like a changelog:
    Administrator altered mailusers -> generating new configs for services postfix, dovecot
    Reseller 'horst' altered domains -> generating new configs for apache2, awstats


    This to me is very important with multiple admins hacking the server(s). Also think of the m in i-mscp, multiple servers. A hell hard thing to monitor, especially when something breaks. I want a quick way to return to last known good state.


    Last point, the users:
    - x number daily backups. Select based on hosting plan, databases, sites, mailboxes. Selectable daily, weekly, monthly. This can create lots of traffic (tar 3000 users every night) for the hosts and lots of webspace to waste (keep 30 copies of 1 website is 30 times the space).
    - a default for me is to backup the sites/databases only. Once daily, I guess it's what has been talked about in this thread here initially. Train the users to make backups between changes and all is good.
    - When the customer is allowed to have multiple backup steps, then account this into his quota.
    - I'm curious, can customers already run a restore from the web panel? Perhaps this could be useful.
    - I do not make individual email backups. Most users still have POP boxes and download everything. I do nightly backups of /var/mail and can restore individual boxes via shell on demand, but that has never been the case in the last years.


    Okay, I hope I haven't bloated this thread to much with my reply.

    Well... Can i suggest the same backup system used in ipbrick servers (http://www.ipbrick.com)?


    How it works, all changes are saved into the DB.
    Example:
    - Add a mailbox.
    - Add a subdomain.
    - Click "apply" (this will save the changes into a DB row and apply the changes to the system)


    This way, if i want to go back to a date/time, i just have to go on that row and click the restore button.
    [hr]
    Note: We have a ipbrick server @ our enterprise and i can assure that i never see a so good backup/restore system like that!
    If your server has been stolen (this is a example) and you only have the files daily backed up to a NAS system. Pick a new server, install the server with default values (this takes 10 minutes +/-), pick the configuration backup (it is also mailed to somewhere outside the server, in our case it's a gmail account), upload it on the control panel, and 5 minutes later the whole server is configured with all mailboxes, domains, subdomains, (in ipbrick case: voip accounts, antivirus license, antispam license, firewall settings, LAN domain with all hosts, Domain users, domain configs, etc...) and he also asks if you want to restore all files from the last configured destination backup to their original place.

    Edited once, last by xclanet ().

    One big ask I have is to not force the backups under the users web directory. It would be nice to be able to configure this to a different partition, so it could be on a slower raid array, Nfs mounted volume, etc.


    Let the user define files to exclude from the backup would be nice too.


    Differential backups would be amazing. :)