A Position Paper prepared for the NLANR Web Cache Workshop, June 1997, by Donald Neal, Information & Technology Services, The University of Waikato - d.neal@waikato.ac.nz
This paper describes the pricing models adopted for the New Zealand Internet Exchange (NZIX) cache between 1995 and 1997, and argues that the design of caching software needs to take into account the competitive nature of Internet bandwidth provision.
Until the end of 1995, almost all Internet traffic to and from New Zealand passed across international links operated by NZGate, a non-profit activity of The University of Waikato. NZGate operated a system of charging by volume of traffic.
The NZGate charging system sought to charge most heavily that international traffic which created the greatest costs for the link operator, so rather than charge a fixed rate per megabyte, a series of multipliers were applied to the volume of traffic logged.
| Traffic Type | Discount |
| Telnet and FTP Commands | Full Rate |
| Other Traffic (e.g. WWW) | 15% |
| Email, FTP Data | 30% |
| Outgoing Traffic | 50% |
| Off-Peak Traffic (8pm - 9am) | 80% |
Each month, an adjusted traffic figure was calculated for each customer by applying these discounts to their metered international traffic. A charge was then made based on this adjusted traffic volume. This charge was calculated according to the band of use in which the customer's traffic placed them. So for example one dollar amount was charged to customers using between two and three gigabytes per month, and higher rate (but cheaper per MB) to those using between three and four gigabytes per month.
If a customer's charging step changed from one month to the next, no change was made to the dollar amount charged until the new step had been occupied for two consecutive months. The effect of the step structure was to allow a customer time to adjust their bandwidth consumption before an increase affected their bill.
In May 1995 The University of Waikato began to operate a caching server, known as the NZGate cache, which served traffic only to other caches. The charge made for use of this service was a simple addition to the existing NZGate charging model. The volume of traffic (adjusted as described above) consumed by the NZGate cache was distributed among the customers of the cache in proportion to the amount of traffic they retrieved from the cache as recorded by the server's cache access log file. This then appeared as an addition to the traffic from which their NZGate charges were calculated.
So the cache was billed for in megabytes rather than directly in money.
By January 1996, around 6% of all traffic across NZGate's international links was coming from the cache, a figure which seemed set to grow.
As an interim arrangement, customers continued to be able to buy bandwidth under the NZGate pricing model, but gradually customers moved to purchasing their bandwidth from international link operators, who for the most were telephone companies.
Kawaihiko, a consortium acting for six of the country's seven universities, concluded an arrangement under which bandwidth was bought at a fixed price per megabyte during peak time (7am to 11pm) and a much lower fixed price per megabyte off-peak. This made it necessary for the NZGate cache, relabelled as the NZIX Cache, to move to a system of charging per megabyte in real money. Each month a multiplier is calculated by dividing total IP traffic consumed by the caching server by the total size of objects served to customers as shown in the server's access log. This is then multiplied by fixed peak and off-peak per megabyte rates to give the rates per megabyte charged for that month.
This minimises the financial risk to the university from operating the cache, and ensures that savings are passed on to customers. In addition, maximum peak and off-peak rates are set which reduce the risk born by customers, who do not find out their actual per megabyte charge for a given month until after the month's end. (These fixed maximum rates are charged at all times for URL's containing a question mark or the string "/cgi-bin/" or ending in ".cgi". This elaboration is aimed at preventing prices for other customers being elevated by the decision of a small number of customers to send requests for non-cachable objects to the NZIX cache.)
This pricing model also allows ready comparisons to be made by universities with non-cached traffic. But this is not true for other organisations. A model which has become common for the sale of Internet connections is that a fixed-capacity connection to the NZIX is sold at a fixed price. This provides customers with a predictable price for Internet traffic, though sometimes at the cost of performance. This option was not available under the NZGate pricing scheme.
An effect of this pricing arrangement is that traffic between any point outside New Zealand and the NZIX is free of charge, which puts the NZIX Cache outside New Zealand for charging purposes. The only possible appeal of the NZIX Cache in such an environment is as a vendor of high-speed bandwidth (perhaps to customers taking only objects already in cache) at a premium price.
It is possible to provide such a service and also to deliver predictability of cost. This could be done by selling access to the cache at a fixed price on top of the price of the pipe to the exchange where the cache is located. But because a subscription model does not relate the cache's income to the amount of bandwidth consumed by it, it is likely only to be feasible where the cache operator is also the company selling the fixed-size pipes into the cache.
The use of such fixed-size pipes is likely to be particularly awkward at an Internet exchange, where it would be desirable for multiple caches to communicate with each other.
In this environment, growth in traffic taken from the NZIX cache ceased. Also in 1996 complaints of poor performance from the cache were received from some customers. The poor performance could not be reproduced by the cache administrator. It is not clear what the relationship is among perception of poor performance, desire for a fixed price for Internet service, and the desire by some ISP's to purchase their own international bandwidth independent of New Zealand's telephone companies. But it seems reasonable to assume that the disappearance of cost saving as an incentive for cache use by many companies was a factor in the failure of the service to grow further in 1996.
With effect from 1 June 1997, the Kawaihiko consortium intends to replace the system of purchase at peak and off-peak per-megabyte rates with a more complex system aimed at reflecting the actual cost of provision of international bandwidth. The new structure uses the concept of a "virtual pipe", as distinct from the fixed-size pipe referred to above.
The intention is that traffic between each Kawaihiko member site and networks outside New Zealand will be measured in each direction at five-minute intervals. From this recorded data for each calendar month the third quartile is then to be calculated for each five-minute interval of the day. This gives a "third quartile day" [3], which is an approximation of the way the site's bandwidth demand varies through the day. The "virtual pipe" size needed by the site is then the highest level of throughput seen in this "third quartile day".
The purpose of this model is to drive down the price per unit bandwidth paid by the customer by transferring the risk inherent in uneven distribution of bandwidth consumption from the provider to the customer.
It would be possible to charge a fixed rate per capacity unit of a customer's virtual pipe. In fact, Kawaihiko's intention is to allocate actual costs incurred by the consortium in proportion to each site's virtual pipe.
Under this model, bandwidth at times which are clearly away from the peak use period for a given site is free of charge to that site. This means, as has been observed before [1], that there is greater potential for caches to produce benefit by loading material expected to be wanted at off-peak times. But since no customer saves money by using the NZIX cache outside the customer's own peak period, it appears necessary for the NZIX cache to move to a similar model for charging its own customers, or at least those of its own customers buying their non-cached bandwidth under this scheme. But that is not likely to win the cache customers among the many organisations, including one New Zealand university, for whom the purchase of bandwidth at a fixed, entirely predictable price is a priority in itself.
The ideal answer for customers wishing to use the fixed-capacity pipe model would be for the cache to be able to sell bandwidth in this way, whether or not that was how the cacvhe purchased its bandwidth. this would require the caching server to know the current transfer rate to a given client, and to queue incoming requests for that client once that client's set threshold is reached. (The implementation of a varaible pipe could be made easier by similar code being used to log the throughput to clients at particular times for later analysis, removing the need for external traffic monitoring or providing a check on external monitoring.)
Such control would presumably be based on throughput in terms of object and header size rather than true IP volume, but hopefully this would be an acceptable approximation.
An alternative approach for a large cache in such a pricing environment is to seek viability as a premium-cost service offering igher performance, for instance by the use of persistent TCP over high-latency links. But even if the performance improvement achieved is such that cache misses can be served not observably more slowly than if the customer had retrieved them directly, this may not attract customers whose priority is predictability of pricing.
There is a further problem with the variable pipe model. Consider the case of a cache which buys its bandwidth from two providers using this pricing structure. If the link to one of the providers fails, this will result in an unusually high load on the other. This is very likely to inflate the total cost of bandwidth to the cache for that charging period, since cost is based on peak load, and peak load on one line is unlikely to be compensated for by a similar reduction in the third quartile day seen by the supplier whose connection failed.
One way to reduce this risk is to ensure that bandwidth is bought from more than two suppliers. A preferable approach would be for the cache to be able to restrict its own bandwidth consumption, especially if that restriction could be imposed only to apply during peak times of day. This ability would be useful in a caching server receiving traffic in a high-bandwidth environment even if it were not on-charging its clients.
A cache is a source of bandwidth, and it must be assumed that customers will seek to compare the costs and benefits to them of using a cache with those of using other available sources of bandwidth. The fact that until 1995 New Zealand had a single pricing structure for almost all Internet use may have been an important factor in the rapid uptake seen of use of the NZGate cache. This structure ensured that cached traffic had a pricing structure easily comparable to that for non-cached traffic.
The issues considered in this paper do not affect the desirability of caching in a zero-settlement environment, for example within a medium-sized company. But the fact that no caching software allows bandwidth to be sold from the cache in a manner comparable to non-cached traffic pricing structures has the potential greatly to obstruct the development of distributed caching.
[2] Chankhunthod, Anawat et al., A Hierarchical Internet Object Cache, 1996, http://www.netapp.com/netcache/cache.ps
[3] Houlker, John, 19 April 1997, Private email to staff of The University of Waikato including the author