Net.WebClient and GZip

The Net.WebClient class doesn't support HTTP compression, eg, when you add the Accept-Encoding: gzip,deflate header to your request:

        Dim wc As New Net.WebClient
        '-- google will not gzip the content if the User-Agent header is missing!
        wc.Headers.Add("User-Agent", strHttpUserAgent)
        wc.Headers.Add("Accept-Encoding", "gzip,deflate")
        '-- download the target URL into a byte array
        Dim b() As Byte = wc.DownloadData(strUrl)

What you get is a gzipped array of bytes. It's pretty easy to add the missing gzip support, though. First, download the SharpZipLib and add a reference to ICSharpCode.SharpZipLib to your project. Then it's only a few more lines of code..

        Dim gz As New GZip.GZipInputStream(New MemoryStream(b))
        Dim intSizeRead As Integer
        Dim unzipBytes(intChunkSize) As Byte

        Dim OutputStream As New MemoryStream
        While True
            '-- this decompresses a chunk
            '-- remember the output will be larger than the input (one would hope)
            intSizeRead = gz.Read(unzipBytes, 0, intChunkSize)
            If intSizeRead > 0 Then
                OutputStream.Write(unzipBytes, 0, intSizeRead)
            Else
                Exit While
            End If
        End While

        '-- convert our decompressed bytestream into a UTF-8 string
        Return System.Text.Encoding.UTF8.GetString(OutputStream.ToArray)

And voila, the bandwidth, you have saved eet! How do I know this actually works? Using my network sniffer of course..

Posted by Jeff Atwood    View blog reactions