Varnish

    Varnish Cache is a web application accelerator (also known as a caching HTTP reverse proxy) that is installed and configured in front of any HTTP and takes care of caching its contents. Varnish Cache is really fast,  typically speeding up delivery with a factor of 300 - 1000x depending on the architecture.

    Apart from its performance, another of its key features is the flexibility of its configuration language, VCL. VCL enables you to write policies on how incoming requests should be handled. In such a policy you can decide what content you want to serve, from where you want to get the content and how the request or response should be altered.

    Which requirements does it have?

    Varnish requires a working compiler (such as gcc) to compile its configuration file, which is then dynamically linked into the server process.

    How to enable Varnish?

    Varnish is disabled by default so it cannot be started using the ctlscripts. Enabling it is as simple as renaming its script from ctl.sh.disabled to ctl.sh:

    cd <installdir>/
    mv varnish/scripts/ctl.sh.disabled varnish/scripts/ctl.sh
    ./ctlscript.sh start varnish

    In case you are using BitNami Cloud Hosting, a Virtual Appliance, or a Amazon Cloud Image, the installation directory is /opt/bitnami and you have to use "sudo" to move the file:

    sudo mv /opt/bitnami/varnish/scripts/ctl.sh.disabled /opt/bitnami/varnish/scripts/ctl.sh
    sudo /opt/bitnami/ctlscript.sh start varnish
    

    If you enable Varnish, please make sure that PageSpeed is disabled. Varnish with PageSpeed is not currently supported in Bitnami.

    How to start Varnish?

    Varnish is included with a default port configured as the first free port after the selected Apache port. Thant should be enough to do a preliminary test to check all is in place. You just need to execute:

    $ ./ctlscript.sh start varnish
    SMF.s0: filename: /opt/bitnami/varnish/var/rubystack-2.3.14-9/varnish_storage.bin size 1024 MB.
    /opt/bitnami/varnish/scripts/ctl.sh : varnish started at port 81

    And ensure Apache is running:

    $ ./ctlscript.sh start apache
    Syntax OK
    /opt/bitnami/apache2/scripts/ctl.sh : httpd started at port 80

    Now you should be able to access the index page in both ports, 80 and 81. Accessing the server through the 80 port will behave as if Varnish were not enabled, retrieving all the data from the server. Accessing the 81 port however will use Varnish as a reverse proxy, serving cached contents and asking to Apache for uncached ones. 

    How to check if Varnish is working?

    You can check what Varnish is doing under the hood using the `varnishlog` command. To indicate which instance of Varnish you are interested in, you must specify the Varnish working directory, which is by default `${installdir}/varnish/var/varnish/`:

     
    varnishlog
        0 CLI          - Rd ping
        0 CLI          - Wr 200 19 PONG 1340840690 1.0
        0 CLI          - Rd ping
        0 CLI          - Wr 200 19 PONG 1340840693 1.0
        0 CLI          - Rd ping
        0 CLI          - Wr 200 19 PONG 1340840696 1.0

     

    If you visit your server url though the configured Varnish port (81 in our example) you will see a more interesting output:

     15 Hash         c /favicon.ico
       15 Hash         c 75.101.208.108
       15 VCL_return   c hash
       15 VCL_call     c pass pass
       15 Backend      c 14 default default
       15 TTL          c 1976586397 RFC 120 -1 -1 1340840847 0 1340840847 0 0
       15 VCL_call     c fetch
       15 TTL          c 1976586397 VCL 120 -1 -1 1340840847 -0
    ....
       15 TxResponse   c OK
       15 TxHeader     c Server: Apache
       15 TxHeader     c X-Powered-By: PHP/5.3.13
       15 TxHeader     c Content-Type: image/vnd.microsoft.icon
       15 TxHeader     c Content-Length: 0
       15 TxHeader     c Accept-Ranges: bytes
       15 TxHeader     c Date: Wed, 27 Jun 2012 23:47:27 GMT
       15 TxHeader     c X-Varnish: 1976586397

    To get a clearer idea of what is happening, you can use `varnishstat` instead:

    varnishstat
    Hitrate ratio:        1        1        1
    Hitrate avg:     0.0000   0.0000   0.0000
              35         0.00         0.02 client_conn - Client connections accepted
              23         0.00         0.01 client_req - Client requests received
               8         0.00         0.00 cache_miss - Cache misses
    

    The command shows much more information but the clearer indication of if it is working is by checking the Hitrate ratio (how often Varnish finds the contents in its cache) and the cache_misses (how much times it failed and had to contact Apache). If you navigate your site for a while, you may find something like the below:

     

    0+00:38:06                                                                                                                     /opt/bitnami/varnish/var/varnish
    Hitrate ratio:       10       62       62
    Hitrate avg:     0.9990   0.9677   0.9677
            7393         0.00         3.23 client_conn - Client connections accepted
            7380         0.00         3.23 client_req - Client requests received
            7354         0.00         3.22 cache_hit - Cache hits
     
    There is also a webpage that allows you check the Varnish status and it gives more info about the configuration. You can test your application at http://www.isvarnishworking.com/

    How to change the Varnish and Apache ports?

    After checking all is working properly, you may want to change Varnish port to a standard one, usually 80. If it was free at installation time, it should be already taken by Apache.
     
    Before starting with the process, we have to stop Apache and Varnish:
    cd <installdir>
    ./ctlscript.sh stop apache
    ./ctlscript.sh stop varnish
     
     
    The configuration of the ports will involve first changing the Apache port and then the Varnish port:
     
    Move Apache to a different port so 80 is available:
     
      To change it,  you have to edit the "Listen" directive in the Apache configuration file under <installdir>/apache2/conf/httpd.conf. In our example, the relevant section of the file will look like the below snippet:
    ...
    # Change this to Listen on specific IP addresses as shown below to 
    # prevent Apache from glomming onto all bound IP addresses.
    # 
    #Listen 12.34.56.78:80
    Listen 80
    ...
     
    Then we can move it to a different one, for example the `81` port:
     
    ...
    # Change this to Listen on specific IP addresses as shown below to 
    # prevent Apache from glomming onto all bound IP addresses.
    # 
    #Listen 12.34.56.78:80
    Listen 81
    ...
     
    Update your application configuration for Apache:
    If your applications are configured with virtual host, you will need to change the port in the applications configuration file for apache as well in /opt/bitnami/apps/<appname>/conf/httpd-vhosts.conf. To know which applications are you running as virtual host check the file  /opt/bitnami/apache2/conf/bitnami/bitnami-apps-vhosts.conf. In that file you will find the list of applications that you need to update, if any.
     
    We are using this port because is the one currently in use by Varnish, so we know it will be available after changing Varnish to listen in the new freed port (80). To get the current Varnish port you check its VARNISH_PORT variable in its ctl.sh script file under <installdir>/varnish/scripts/ctl.sh:
    #! /bin/sh  
    ...
    VARNISH_PORT=81
    ...
    
     

     Configure Varnish to use the old Apache port (80) and specify the new port in which Apache will be listening in the configuration file (81).

     
    After changing Apache port, we have available the port 80. To make varnish listen in that port, you have to edit its ctl.sh script file under <installdir>/varnish/scripts/ctl.sh. In our example, it looks like:
     
    #! /bin/sh  
    ...
    VARNISH_PORT=81
    ...
    VARNISH_CONFIG_FILE=/opt/bitnami/varnish/etc/varnish/default.vcl
    ...

    So we change it to 80:

    #! /bin/sh  
    ...
    VARNISH_PORT=80
    ...
    VARNISH_CONFIG_FILE=/opt/bitnami/varnish/etc/varnish/default.vcl
    ...
     
    Please note we also included the VARNISH_CONFIG_FILE details in the snippet. This file should be also edited as it contains the port in which Apache was listening, which is now outdated. You have to open this file (/opt/bitnami/varnish/etc/varnish/default.vcl in our example) and look for the below section:
     
    backend default {
        .host = "127.0.0.1";
        .port = "80";
    }      

    And we change the port from 80 to the new Apache port, 81:

    backend default {
        .host = "127.0.0.1";
        .port = "81";
    }      

    After restarting, Apache uncached should be available at port 81 and Varnish at 80 as revese proxy for Apache:

    cd <installdir>
    ./ctlscript.sh restart

    How to change the configuration file?

    Varnish is installed with a default configuration file, agnostic to the web application being cached. Using this configuration file, although achieving high performance, could lead to some contents not properly refreshed in the cache so your users would get an outdated version of your site.

    The solution is to use a particularized VCL configuration file. There are multiple sources on the Internet that provides customized configuration files for different applications. A good source is the Varnish examples page. In this example, we will change our default.vcl configuration file to a WordPress-specific one (you can find the source file here).

     

    The file requires a little modification to register the port in which your Apache server will be running. This port can be read either from the Apache configuration, looking for the "Listen" directive (in <installdir>/apache2/conf/httpd.conf):

     

    ...
    # Change this to Listen on specific IP addresses as shown below to 
    # prevent Apache from glomming onto all bound IP addresses.
    # 
    #Listen 12.34.56.78:80
    Listen 80
    ...

    Or executing in a console:

    $ egrep '^Listen ' <installdir>/apache2/conf/httpd.conf
    Listen 80

     

    With this value (80), we edit the dowloaded wordpres.vcl and look for the section below, usually at the top of the file :

     
    backend default {
        .host = "127.0.0.1";
        .port = "8080";
    }      

     

    And we change the port:

    backend default {
        .host = "127.0.0.1";
        .port = "81";
    }      
    
     

    In the case of the BitNami stacks, Varnish is installed in the same server as Apache so the .host can be configured as `127.0.0.1` but you could also use Varnish to cache a remote server. In that case you should also provide the host IP.

     

    After performing this modification, we copy the file to the varnish directory:

    cp /path/to/the/wordpress.vcl  <installdir>/varnish/etc/varnish/

     

    And then change the varnish ctl scripts to load the appropriate file. To do that, we first ensure Varnish is stopped:

    cd <installdir>
    ./ctlscript.sh stop varnish

     

    And then edit the file `<installdir>/varnish/scripts/ctl.sh`  and change the `VARNISH_CONFIG_FILE` variable to point to the new file:

    #! /bin/sh
    ...
    VARNISH_CONFIG_FILE=/opt/bitnami/varnish/etc/varnish/wordpress.vcl
    ...

     

    And now we restart Varnish (and apache if needed):

    ./ctlscript.sh start varnish
    ./ctlscript.sh start apache
    

     

    IMPORTANT: Varnish does not cache the content if Apache PageSpeed module is enabled. Please check how to disable this module here.

    How to block phpmyadmin, phppgadmin and server-status for being cached?

    It is strongly advisable to block phpmyadmin, phppgadmin and/or server-status to being cached and public, you will need to add this code at the end of the default varnish configuration file (/opt/bitnami/varnish/etc/varnish/default.vcl) on in the configuration file that you are using for your application:

    sub vcl_recv {
        if (req.url ~ "^/phpmyadmin/.*$" || req.url ~ "^/phppgadmin/.*$" || req.url ~ "^/server-status.*$") {
            error 403 "For security reasons, this URL is only accesible using localhost (127.0.0.1) as the hostname";
        }
    }
    
     
    With this code, you will block any request to that URLs, so for example if anyone wants to access to: www.yourdomain.com/phpmyadmin, they won't be able to access.
     

    How to configure multiple applications with Varnish

    Varnish allows configuring multiple applications in the same server. To do that, you first have to get the VCL files for each of the applications (on our example, Ghost and WordPress):
     
    wget  -O ghost.vcl https://gist.github.com/lkarsten/6683179/raw/3d03b14b3a3ca1301a2397eaacf3158100b4c7bd/default.vcl
    sudo cp ghost.vcl /opt/bitnami/varnish/etc/varnish/ghost.vcl
    sudo sed -i 's/port = "[^"]*"/port = "80"/g' /opt/bitnami/varnish/etc/varnish/ghost.vcl
     
     
    wget -O wordpress.vcl http://wiki.bitnami.com/@api/deki/files/347/=wordpress.vcl
    sudo cp wordpress.vcl /opt/bitnami/varnish/etc/varnish/
    sudo sed -i 's/port\s*=\s*"[^"]*"/port = "80"/g' /opt/bitnami/varnish/etc/varnish/wordpress.vcl
     
    Now we have to append some configuration to the default.vcl file that will use the proper VCL file configuration depending on the application selected. This will depend on how you have your applications . For example, if your applications are configured to be accessed based on a URL prefix such as "example.com/foo" corresponding to the "foo" app and "example.com/bar" pointing to the app "bar", you could use:
     
     if ( req.url ~ "^/foo/")
     {
        # /foo configuration
     }
     elsif ( req.url ~ "^/bar/")
     {
      # /bar configuration
     }
     
     Or if you have your applications configured with different domains such as wordpress.example.com pointing to your WordPress app and ghost.example.com pointing to Ghost, you will need to setup:
     
     if (req.http.Host == "wordpress.example.com")
     {
        # Wordpress configuration
     }
     elsif (req.http.Host == "ghost.example.com")
     {
       # Ghost configuration
     }
     
    Or if you want to be able to access though different ports:
     
     if (req.http.Host ~ "^wordpress.example.com:[0-9]+")
            # Wordpress configuration
       } elsif (req.http.Host ~ "^ghost.example.com:[0-9]+")  {​
        # Ghost configuration
     }
     
     
     In our case, we are using the second approach.
     
    To create the new configuration, we need to merge the contents of the different sections (vcl_recv, vcl_fetch, vcl_deliver, vcl_*) inside wordpress.vcl and ghost.vcl.
     
    sub vcl_recv {
      if (req.http.Host ~ "^wordpress.example.com:[0-9]+")
         # Wordpress vcl_recv configuration
         ...
       } elsif (req.http.Host ~ "^ghost.example.com:[0-9]+")  {​
         # Ghost vcl_recv configuration
         ...
       }}
    sub vcl_fetch {
      if (req.http.Host ~ "^wordpress.example.com:[0-9]+")
       # Wordpress vcl_fetch configuration
         ...
       } elsif (req.http.Host ~ "^ghost.example.com:[0-9]+")  {​
         # Ghost vcl_fetch configuration
         ...
       }
    }
    sub vcl_deliver {
      if (req.http.Host ~ "^wordpress.example.com:[0-9]+")
         # Wordpress vcl_deliver configuration
         ...
       } elsif (req.http.Host ~ "^ghost.example.com:[0-9]+")  {​
         # Ghost vcl_deliver configuration
         ...
       }
    }
     
    If any of the sections is empty in any of the VCL files, you can ommit it. In addition, if you find addtional  vcl_ subst in addition to vcl_recv, vcl_fetch and vcl_deliver, you can just have to merge them in a similar way In our scenario, we would end up with:
     
    # Previous contents of default.vcl
    # ....
    # ....
    
    # Multi-apps configuration:
    # This is a merge of all the different sections of wordpres.vcl and ghost.vcl
     
    sub vcl_recv {
      if (req.http.Host ~ "^wordpress.example.com:[0-9]+") {
          # Contents of the section vcl_recv in wordpress.vcl      
          if (req.http.Accept-Encoding) {
            #revisit this list
            if (req.url ~ "\.(gif|jpg|jpeg|swf|flv|mp3|mp4|pdf|ico|png|gz|tgz|bz2)(\?.*|)$") {
              remove req.http.Accept-Encoding;
            } elsif (req.http.Accept-Encoding ~ "gzip") {
              set req.http.Accept-Encoding = "gzip";
            } elsif (req.http.Accept-Encoding ~ "deflate") {
              set req.http.Accept-Encoding = "deflate";
            } else {
              remove req.http.Accept-Encoding;
            }
          }
          if (req.url ~ "\.(gif|jpg|jpeg|swf|css|js|flv|mp3|mp4|pdf|ico|png)(\?.*|)$") {
            unset req.http.cookie;
            set req.url = regsub(req.url, "\?.*$", "");
          }
          if (req.http.cookie) {
            if (req.http.cookie ~ "(wordpress_|wp-settings-)") {
              return(pass);
            } else {
              unset req.http.cookie;
            }
          }
        } elsif (req.http.Host ~ "^ghost.example.com:[0-9]+") {
          # Contents of the section vcl_recv in ghost.vcl
              
              
          # If the client uses shift-F5, get (and cache) a fresh copy. Nice for
          # systems without content invalidation. Big sites will want to disable
          # this.
          if (req.http.cache-control ~ "no-cache") {
            set req.hash_always_miss = true;
          }
          set req.http.x-pass = "false";
          # TODO: I haven't seen any urls for logging access. When the
          # analytics parts of ghost are done, this needs to be added in the
          # exception list below.
          if (req.url ~ "^/(api|signout)") {
            set req.http.x-pass = "true";
          } elseif (req.url ~ "^/ghost" && (req.url !~ "^/ghost/(img|css|fonts)")) {
            set req.http.x-pass = "true";
          }
          
          if (req.http.x-pass == "true") {
            return(pass);
          }
          unset req.http.cookie;
        }
    }
    sub vcl_fetch {
      if (req.http.Host ~ "^wordpress.example.com:[0-9]+") {
        # Contents of the section vcl_fetch in wordpress.vcl
          if (req.url ~ "wp-(login|admin)" || req.url ~ "preview=true" || req.url ~ "xmlrpc.php") {
            return (hit_for_pass);
          }
          if ( (!(req.url ~ "(wp-(login|admin)|login)")) || (req.request == "GET") ) {
            unset beresp.http.set-cookie;
          }
          if (req.url ~ "\.(gif|jpg|jpeg|swf|css|js|flv|mp3|mp4|pdf|ico|png)(\?.*|)$") {
           set beresp.ttl = 365d;
          }
        } elsif (req.http.Host ~ "^ghost.example.com:[0-9]+") {
        # Contents of the section vcl_fetch in ghost.vcl
              
        # Only modify cookies/ttl outside of the management interface.
          if (req.http.x-pass != "true") {
            unset beresp.http.set-cookie;
            if (beresp.status < 500 && beresp.ttl == 0s) {
              set beresp.ttl = 2m;
            }
          }  
        }
     }
     sub vcl_deliver {
        if (req.http.Host ~ "^wordpress.example.com:[0-9]+") {
            # Contents of the section vcl_deliver in wordpress.vcl
               
            # multi-server webfarm? set a variable here so you can check
            # the headers to see which frontend served the request
            #   set resp.http.X-Server = "server-01";
            if (obj.hits > 0) {
                set resp.http.X-Cache = "HIT";
            } else {
                set resp.http.X-Cache = "MISS";
            }
        } elsif (req.http.Host ~ "^ghost.example.com:[0-9]+") {
            # Contents of the section vcl_deliver in ghost.vcl
            
        }   
     }
     
     
    Once you have added the new configuration to the existing default.vcl file and restarted varnish, you should have both configurations working simultaneosly depending on the hostname.
     
    Here you can donwload the different files used in the example as a reference:
     

    Comments

    You must login to post a comment.

    Attach file

    Attachments

    FileSizeDateAttached by 
     default.vcl
    Modfied default.vcl
    6.71 kB12:20, 21 May 2014JuanjoActions
     ghost.vcl
    Original ghost.vcl
    5.38 kB12:20, 21 May 2014JuanjoActions
     wordpress.vcl
    Original wordpress.vcl
    1665 bytes12:20, 21 May 2014JuanjoActions