A Varnish solution to your Flash Transfer-Encoding: chunked woes

The symptom

You embed a nice Flash object on your page.  This Flash object refers to other objects, incrementally downloading them.  You save the page, and then proceed to test it.

The first time you test it, it does not work!  Your Flash object loads alright, but the referred objects do not load.  You stare at the Network tab of Firebug, utterly baffled.  Or worse: you are testing on an Android phone, and it just won't work.

You reload the page.  This time, mysteriously, it works.  The referred objects do load.

Voodoo?  Magic?

The problem

Transfer-Encoding: chunked.

Chunked transfer encoding is a special way to transmit data to HTTP clients which relies on writing, not a long stream of bytes, but an integer number, followed by a newlline, followed by a stream of bytes equal in length to the integer number, followed by another newline, and repeating the process until the transfer completes.

This usually happens for various reasons:

• Your Flash output is constructed on the fly by a dynamic Web server or application such as a PHP script.  Your application does not buffer the output and write it in one fell swoop, but rather writes it "chunk by chunk".
  The Web platform, for efficiency reasons, obliges.
• Your Flash output got encoded as gzip somewhere in the line (say, the Web server served it compressed), and then decoded at a proxy cache such as Varnish, which leaves the proxy with a bunch of chunks to transmit, so the efficient choice is to transfer them using chunked encoding.
  This often happens if you are using Plone 4 with gzip encoding enabled in plone.app.caching, and fronting it with Varnish 3 and automatic gzip normalization.

So, when the browser downloads the first Flash object, all is peachy (because browsers usually have very good transfer encoding implementations).  This object loads.  Then Flash itself attempts to download the remaining objects... and that's where Flash's crappy implementation of HTTP bites you.

The next result is a half-loaded Flash movie.

The next time you load the page, the first object is already loaded, and the subsidiary objects are in the browser's cache already.   This time, Flash has no problem locating the subsidiary objects, so it loads them right away, and your Flash movie works fantastic.

But, erm, you can't just tell your Web site visitors to reload the page once, right?

The solution

This is the solution I implemented with Varnish 3.  All we do is this: we will set Varnish up to request only one type of encoding: identity.  We will do this in two circumstances:

1. when Varnish decides "hey, I don't have this object in my cache, so I need to fetch it" (vcl_miss),
2. when Varnish decides "hey, I'm going to pass on this request because it has cookies or whatnot".

Thsi is the VCL you need to implement such a solution:

sub donotgzipflash {
    if (bereq.http.host == "yourwebserverdomainname.com") {
       if (bereq.url ~ "\.swf($|?)") {
            set bereq.http.Accept-Encoding = "identity";
       }
    }
}

sub vcl_miss { call donotgzipflash; }
sub vcl_pass { call donotgzipflash; }

All we are doing here is detecting, based on the URL extension (Shockwave Flash), and then telling the backend to please do not gzip, and please do not encode it in any other way than a byte stream (hence, identity).