Backups

    Hi All,


    So, I realized a new behavior with sci2tech's sweet backup script, that, for me, is not ideal, and I think maybe others may want it changed in the release... I will let you decide...


    The backup script deletes any folders in the /backups folder. For me, I have some domains with an archive script backing up to /backups/archive/ ... customers may also store something else there, who knows..


    To modify this, I just modified /var/www/imscp/engine/backup/imscp-backup-all at line 108.


    i changed

    Code
    1. $rs |= execute("$main::imscpConfig{'CMD_RM'} -rf $bkpDir/*", \$stdout, \$stderr);


    to

    Code
    1. $rs |= execute("$main::imscpConfig{'CMD_RM'} -rf $bkpDir/*.*", \$stdout, \$stderr);


    so that it is no longer recursive, and doesn't send the inconsequential "can't delete folder" error to the imscp error mailer


    Also, (and I know this has been brought up before) what do we think about implementing something a little less disk intensive than deleting everything and recreating with tar+zip?


    Perhaps if we went with an initial gzip --rsyncable and then did rsync backups from then on? Our servers would be much happier.


    We could integrate the initial gzip --rsyncable with the domain creation process, that way we don't have to write any strange checks in the backup script, and that would be quick since it's only doing a couple files to start with.


    How does this sound to everyone?


    I think I'm going to begin implementing it myself anyways...

    Edited once, last by anarking ().

    Is old backup script with new code. Way of process is same. gzip --rsyncable is GREAT idea. If you implement will be added to core. If not, after multiserver will be done I will try do it.

    I'll see what I can do :) It will be a couple weeks before I can work on it, but it shouldn't take too long to implement. I do not know Perl very well, but bash is my friend :P so we will see.


    The basic formula I think is this...


    At domain creation, do gzip -rsyncable to backups directory.
    For backup-all, rsync... (but it will need a check to make sure a gz is there, if it is not, it will need to run gzip). the behavior for this... what do you think?


    if no gzip file found... run gzip? immediately? or wait until after other rsyncs are done? send error?

    Edited once, last by anarking ().

    okay so I am getting to work on this finally after virtualizing my infrastructure. the backup script is not a problem in imscp-backup-all, no problem there, i am familiar with it, but...


    which file and where should the initial gzip rsyncable creation script go? It needs to happen at the end of the domain add script, as soon as that completes, so where is best for me to put the gzip creation lines? i am not familiar with the domain add engine files. (user_add#.php files?)


    just let me know and i will finish the new backup system and let you guys know.

    Problem is that script never make distinction between first account creation and following recreations caused by upgrade (add vs change).
    So as suggestion script should check if initial archive exists, if not run script to create else rsync existing archive, done in existing script.
    Also creating / modifying accounts should end task as soon as possible since require some server restart, a backup is more time consuming so i would keep it in a separate script like it is now (triggered by cron).

    hey sci.


    good suggestion for recreations from upgrade... i was planning on changing the filename structure (just removing hour:minute or just minute), as the hour:minute format now is problematic/annoying for some scripts, etc., because linux has to put escape characters inbetween the ":". So I will make it to check for that format filename and else if it is the old format.


    the reason why i want to add the initial gzip rsyncable to the create account (create only, modify is not affected), is because then the backup job can just be rsync. otherwise it has to run logic every time to if no backup exist, run gzip rsyncable.


    to run a gzip rsyncable at the end of create account should take less than 1 second because the directory is very small at creation. this seems more efficient than adding more logic to run with every cron.


    where is the end of the account creation script? at least for my own testing :)


    thanks for the help sci.

    Entry point and end point for user creation is in User.pm function process to add a code just before ending you should add it like:

    thanks sci.


    one question, if adding it there, can i be sure that the files will have already been created by the time the backup function runs at that point you say to add lines, or does the function before just set it to 'toadd' and the files may or may not be there in time?


    i may not need this, if i continue a different way...


    i think, to upgrade-proof and to make sure that in case anything happens to a backup file, the if-else for the nightly script will check if the backup file matches the format (or doesn't exist), and if not, delete and then create a new gzip rsyncable, but if it matches, to continue with normal rsync. this should be the safest way to do it. agreed?


    i have to do this by name matching. current imscp backup script creates datestamp with hours and minutes, so i could use datestamp with just date, same as ispcp was, but then anyone migrating from ispcp would match. i guess i will just make it a slightly different format from both.

    Hi anarking


    maybe just do it like this:
    if a file "domain.tar.gz" exists then do rsync - otherwise remove everything in backupdir and do a "first new backup"...


    And another thing: I have some hostings withe either some big data (eg. a dvd iso image) or with heavy load so temp files are a problem for backup - so I changed the backup script a little bit to allow having a "exclude" file:



    so if there's a file domain-home/backup.exclude then add to the parameters: --exclude-from=backup.exclude


    Maybe this is also interesting for others...


    /Joxi