Request: Tutorial for installing Varnish with i-mscp

  • Hello


    I am interested in trying Varnish with i-mscp RC Release: 1.1.0-rc2.3 . I have searched the forum for some information on how to install it but I have not found out on how this can be accomplished. I would like to request please a quick tutorial on how this can be done with i-mscp. It would be a very useful tutorial for everyone using i-mscp and the apache2 web server or vps.


    Thanks in advance for your time and help.

  • Okay I'll put one together when I have some free time, as I've set it up a few times and it works very well. There are a feww things to do to make sure all functions keep working when creating/modifying sites so it doesn't revert to old settings (mainly the port change is all). It's not too difficult, and runs very quickly :)

  • Thank you anarking that would be most appreciated :shy:


  • Okay I'll put one together when I have some free time, as I've set it up a few times and it works very well. There are a feww things to do to make sure all functions keep working when creating/modifying sites so it doesn't revert to old settings (mainly the port change is all). It's not too difficult, and runs very quickly :)


    Hello ;


    If you write a good howto for i-MSCP, I can create a plugin easily. ;)


    Thank you for using i-MSCP

    badge.php?id=1239063037&bid=2518&key=1747635596&format=png&z=547451206

    Edited once, last by Nuxwin ().

  • Pardon the wait, it is not a short tutorial ;) let's begin...


    apt-get install varnish libapache2-mod-rpaf


    (rpaf is used for correct recognition of user ips that are proxied back to apache via varnish, otherwise authentication issues)


    edit /etc/default/varnish


    make sure START=yes (line 8)


    change

    Code
    1. DAEMON_OPTS="-a :6081 \


    to

    Code
    1. DAEMON_OPTS="-a :80 \


    (note: there are many other options here, amount of memory to allocate, whether to use file as storage or direct-to-memory, how many thread pools/workers, and what config files to use)


    Example:
    default is:

    Code
    1. DAEMON_OPTS="-a :6081 \ -T localhost:6082 \ -f /etc/varnish/default.vcl \ -S /etc/varnish/secret \ -s malloc,256m"


    modified (big server):

    Code
    1. DAEMON_OPTS="-a :80 \ -T localhost:6082 \ -f /etc/varnish/default.vcl \ -S /etc/varnish/secret \ -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,2G \ -p thread_pool_add_delay=2 \ -p thread_pools=8 \ -p thread_pool_min=200 \ -p thread_pool_max=5000 \ -p session_linger=50 \ -p sess_workspace=262144"


    thread pools should normally be the # of CPU cores. but the only requirement here is port changed to 80.


    modify /etc/apache2/ports.conf
    change:

    Code
    1. Listen 80


    to

    Code
    1. Listen 81


    edit all site config files in /etc/apache2/sites-available/ including 00_master.conf and 00_nameserver.conf (00_modcband.conf will not be edited)
    change:

    Code
    1. NameVirtualHost <theserversip>:80


    to

    Code
    1. NameVirtualHost *:81


    for scripting, perhaps use:

    Code
    1. sed -i "s/$(ifconfig eth0 | grep 'inet addr:' | cut -d: -f2 | cut -d' ' -f1):80/*:81/g" /etc/apache2/sites-available/*


    NOTE: for servers with multiple IPs being hosted, there will be different configurations needed in Varnish and in apache


    edit all site config files in /etc/apache2/sites-available/ EXCLUDING 00_nameserver.conf and 00_modcband.conf
    add: (start at line 2?)

    Code
    1. RPAFenable OnRPAFsethostname OnRPAFproxy_ips 127.0.0.1


    edit /etc/apache2/mods-available/rpaf.conf
    file should be:

    Code
    1. <IfModule mod_rpaf.c>RPAFenable OnRPAFsethostname OnRPAFproxy_ips 127.0.0.1 ::1</IfModule>


    change /etc/varnish/default.vcl
    make sure backend default is this:

    Code
    1. backend default { .host = "127.0.0.1"; .port = "81";}include "/etc/varnish/imscp/master.vcl";


    this can be the entire file, nothing else is needed. you must mkdir /etc/varnish/imscp and create master.vcl. see following post for example master.vcl


    we include master.vcl, as many settings can be put in there, and must, for caching to work and Varnish to know what domains to accept. this is where it gets tricky.


    the proper way to then configure is to include other specific config files, which can then have the caching specifics that say what to cache for each site, and MUST declare the URL of each site that is being passed back to Apache. and for Varnish to be effective, there must be more config to set what static content to strip of cookies and cache (images, video, archives), what pages to bypass (login), etc. etc. That is a complete other article of reading :P


    Quote

    NOTE FOR DEV:
    For complete integration with i-MSCP, I believe we should have the option "Varnish enabled?" for each customer. If enabled, this would add an entry in /etc/varnish/default.vcl that says

    Code
    1. include "/etc/varnish/imscp/customerdomain.vcl";


    with a newline of the same for every customer. then each of those configuration files can then be edited by a customer from within imscp. (maybe /etc/varnish/imscp/customerdomain/config.vcl if needed to better separate permissions)
    It should use the same method as others handled by i-MSCP (parts/working in /etc/imscp/varnish) to have a base config that inserts {DOMAIN_NAME}.


    Now, to make i-MSCP respect the new port changes when using the panel...


    change .pm files for apache types in /var/www/imscp/engine/PerlLib/Servers/httpd/
    for each type (php-fpm, fcgi, itk) change (for example) ../httpd/apache_php_fpm.pm and ../httpd/apache_php_fpm/installer.pm
    so...
    change apache_php_fpm.pm
    from

    Code
    1. $content.= "NameVirtualHost $_:443\n" for @{$data->{'SSLIPS'}}; $content.= "NameVirtualHost $_:80\n" for @{$data->{'IPS'}};


    to

    Code
    1. $content.= "NameVirtualHost \*:443\n" for @{$data->{'SSLIPS'}}; $content.= "NameVirtualHost \*:81\n" for @{$data->{'IPS'}};


    change apache_php_fpm/installer.pm
    from

    Code
    1. $rdata =~ s/^(NameVirtualHost\s+\*:80)/#$1/gmi;


    to

    Code
    1. $rdata =~ s/^(NameVirtualHost\s+\*:81)/#$1/gmi;


    TO SCRIPT:

    Code
    1. sed -i "s/:80/:81/g" /var/www/imscp/engine/PerlLib/Servers/httpd/*/installer.pm


    Code
    1. sed -i 's/$_/\\\*/g' /var/www/imscp/engine/PerlLib/Servers/httpd/*.pm


    change /etc/imscp/apache/parts/domain*
    from:

    Code
    1. <VirtualHost {DOMAIN_IP}:80>


    to:

    Code
    1. <VirtualHost *:81>RPAFenable OnRPAFsethostname OnRPAFproxy_ips 127.0.0.1


    change /etc/imscp/apache/parts/*_ssl.tpl
    from:

    Code
    1. <VirtualHost {DOMAIN_IP}:443>


    to:

    Code
    1. <VirtualHost *:443>RPAFenable OnRPAFsethostname OnRPAFproxy_ips 127.0.0.1


    change /etc/imscp/apache/01_awstats.conf
    from:

    Code
    1. NameVirtualHost 127.0.0.1:80<VirtualHost 127.0.0.1:80>


    to:

    Code
    1. NameVirtualHost 127.0.0.1:81<VirtualHost 127.0.0.1:81>


    change /etc/imscp/apache2/00_master.conf
    from:

    Code
    1. <VirtualHost {DOMAIN_IP}:80>


    to:

    Code
    1. <VirtualHost *:81>RPAFenable OnRPAFsethostname OnRPAFproxy_ips 127.0.0.1


    change /etc/imscp/apache2/00_master_ssl.conf
    from:

    Code
    1. <VirtualHost {DOMAIN_IP}:443>


    to:

    Code
    1. <VirtualHost *:443>RPAFenable OnRPAFsethostname OnRPAFproxy_ips 127.0.0.1


    cp -p /etc/apache2/sites-available/* /etc/imscp/apache/working/
    service apache restart
    service varnish restart


    Then it should all be fully functional. I think that's everything. Like I said, for use with multiple IP's, it would need a slightly different setup with the variables, I'll think about that some other time ;)
    [hr]
    Here is the good master.vcl varnish config file for use that covers most caching scenarios, you can see, just replace with your domain name(s).


    return(pass); means to pass it through and do not cache at all.


    also includes an example of not caching a particular page (login.php).


    This could easily be ported to i-MSCP handling for customers' varnish .vcl files by replacing the domain names with {DOMAIN_NAME} (as well as ALIAS and subdomain), to generate the new /etc/varnish/imscp/{DOMAIN_NAME}/config.vcl for each customer


    Edited once, last by anarking ().

  • WOW thats one big tutorial!


    Thank you so much for your help. I will give it a go and see what sort of results I get.


    My main aim is to squeeze (as in Debian (no pun intended, well not really)) as much out of a 1 gig ram VPS as I can. So my ideal option would be apache optimized with nginx as proxy but that is still under development. Or just plain nginx but I dont know how my script would work as it is heavy htaccess relient. So for now I am looking at alternatives and I have read good things about Varnish but never knew how to set it up with i-mscp. Until now ;)


    Again thank you so much for what you have done anarking it is most appreciated. I hope it helps many users of i-mscp as well as myself :shy: When I have a little spare time I will give this ago :shy:


    Thanks :shy:

    Edited once, last by veg-grower ().

  • i run a number of 1GB VPS' with i-MSCP for people (I run a small "datacenter"). with basic optimization and no varnish, it's very powerful. with varnish, you have to be careful and very specifically configure it, when using small RAM systems. it will depend on what kind of datasets and things on your site. images, js, css, html, or a lot of database activity. if database activity, varnish doesn't help very much, you want an opcode cacher, I recommend Xcache. When using varnish and any caching, 1GB of ram isn't very much, so like I said, be careful, 256mb for apache, 256mb for varnish, 128mb for Xcache. Varnish handles static files (a lot like nginx, so it doesn't make much sense to use nginx with it when already using apache, speed gains would be minimal) and so will generally use more memory to keep the files in, where Xcache doesn't need much, as data out of the database is usually small.


    apache + varnish will do what you want the best. if you can get it tweaked well and keep most in memory, and the provider's pipe is good enough, you can serve a hundred thousand visitors off a 1GB VPS. it becomes more about CPU than ram at that point (apache likes cpu, p.s. make sure to use php-fpm for xcache to work)

  • Again thanks for the detailed and informative advice it is most appreciated.


    I will be using a VPS for mainly static html/php files and maybe a 'few' wordpress sites. At the moment it is a bit of a challenge I have set myself as I am learning to setup a good VPS and then migrate away from shared hosting. I have installed i-mscp but not really done any major optimization so that is something I will try first and see what results that gives me first before I start to think about anything more advanced if you can run a good number of static sites on a optimized i-mscp with good optimization. Your advice has been very informative. Thanks :shy:

    Edited once, last by veg-grower ().

  • i'm using my own tutorial to make sure i get all the little pieces haha. will also check and make sure nothing has changed with it for latest git master.

  • Waoowww......... its really fast on my VPS, but somehow i need to do something with SEO to be run better with varnish.
    My forum running like F1 speed amazing.