Why use data compression for your web service?

By Roger Willcocks

Speed.

Let me tell you a story about how this came into existance. It might sound a little familiar. :)

It all begins with a customer, who wants to provide a web service to report on the tracking of packages. They need this, and they need it now, so 'Just get it done as quickly as possible', and we do. Problem is, as quickly as possible means that the web service is sitting on a web server in the main network, while the database was sitting at the end of a 1MB pipeline in another country.

Soon enough, the service starts getting popular with the clients customers, and they start reporting lots of time out errors. Some quick work with a packet sniffer shows that the database connection is suffering not too surprising, and we said it was a security and performance risk before we started.

So, the solution is to move the web service onto the same network as the database, and the timeout errors go away. BUT... XML is much more verbose than native SQL Server data, so although the timeouts go away, the response time increases by around 10%. And that's a problem too, it makes for a slow web site on the other end.

So we do some hunting, some searching, and some thinking, and we decide that compression would be good. But IIS doesn't compress dynamic content by default, and we can't mess with the settings on the web server.

Eventually, we hit on some source code for carrying out compression in ASP.NET, and it works very well for web pages, reducing web pages by 40% to 65%. Then comes the next problem. The HttpClientProtocol classes [POST/SOAP/etc] don't support compression. No problem, we inherit from them and build compression handling into them, and there we are. And even better, our XML data is compressed even more than web pages, anywhere from 80% to 95% smaller.

Now we do some performance testing, and the compression adds less than 1 microsecond to our overhead, and the response time improves by 25%. It probably would have been more, but the database needed some work, and was handled by their internal IT staff.

Anyway, on with the rest the article.

In today's connected environment, the two greatest delays in the transfer of data are data retrieval, and network transfer speed. Data processing is not typically a major time factor.

By using data compression, you are doing yourself, your clients, and everyone else on the internet a favor. For yourself, you cut your bandwidth costs, and provide improved response times. For your clients, you provide a faster experience, and better performance of other applications using their network connection, and for everyone else? Well, for them you get to help ease the flood of data currently congesting the information super highway. Now if only someone could find a way to compress spam. :)

Why does compression work so well for web services? Web service data can be compressed between 7 and 20 fold in most cases. This is far in excess of the 2 to 4 fold most people expect. The reason for this is that compression relies on repeating elements, and web service data is made up of XML, which is both verbose, and repetitive.

So, how do you go about implementing support for compression with your web service? Well in fact, that is very simple. All you need is a mechanism for compressing dynamic data, and a way of knowing which clients can decompress the data.

For ASP.NET, the basis of Microsoft web services, the mechanism is simple. You need an HTTP Handler that can compress data. To know which clients can handle compressed data, you need to examine the request header for ACCEPT-ENCODING which specifies what sort of data compression the client understands.

How to install an HTTP Handler in three easy steps

1. Install Bandwidth Buddy
2. Add a reference to the Bandwidth Buddy dll to your web service
3. Add the example sections from the help file to your Web.config file

That's it. Pretty easy on the server side. The example section will work for localhost, other addresses require a license.

How to handle data compression in your webservice client in five slightly more complicated steps

1. Build your application with a reference to the web service as usual
2. Add a reference to the LSpaceDesign.WebServiceClient.dll [included with Bandwidth Buddy for free]
3. Open the reference.cs file for the web service.
   EG: Open the object browser and double click on one of the web service methods.
4. Goto the top of the file, and where it says:

   public class MyWebServiceName: System.Web.Services.Protocols.SoapHttpClientProtocol
   {
   

   replace it with:

   public class MyWebServiceName: LSpaceDesign.Web.Services.Protocols.SoapHttpClientProtocol
   {
   

5. Every time you rebuild your web service proxy class, you will need to repeat step 4

Points to remember when testing

• Data compression requires HTTP 1.1.
  HTTP 1.1 is not available via proxy servers by default
• Check data compression ratios, they should be between 1 in 7 and 1 in 20
• Check what happens if no compression is specified

The author Roger Willcocks is a Microsoft Certified Solutions Developer in .NET, with experience in developing Microsoft product based solutions since 1996. Recent experience includes ASP.NET websites and web services as well as windows services and data collection applications. He is trained in MS SQL Server as well as MS Access and the .NET Framework. Roger is based in Auckland, New Zealand.

Bandwidth Buddy