At my new employer, I put something together where the collection subsystem won’t refer to the previous total for deltas: it just takes whatever you give it and relays that to the metrics backend. Most applications provide you with a request/sec metric these days, so I figured I’d just use that. Instead of doing deltas on TotalAccesses, I started just relaying ReqPerSec instead. Simple enough, right? For those of you not familiar, here’s an example of server-status output:

Total Accesses: 400
Total kBytes: 800
CPULoad: 5
Uptime: 12345
ReqPerSec: 20
BytesPerSec: 20000
BytesPerReq: 1024
BusyWorkers: 15
IdleWorkers: 5

Well, amidst my metrics collection testing for the new system, I stumbled upon an interesting detail: I would be nailing the Apache instance with a flood of requests using my friend ab (aka: apachebench). However, I immediately noticed ReqPerSec wasn’t accurate at all. For example, I would be averaging a test of 300-400 req/s, but ReqPerSec would be displaying 21 req/s. It was as if I wasn’t sending it requests.

That being said, the TotalAccesses metric was reflecting the requests I was sending, so I knew the web server was receiving them. Interesting.

It got me thinking: “I wonder how Apache determines ReqPerSec?” Time for some source code action! Sure enough, here’s what I found:

ap_rprintf(r, "ReqPerSec: %g\n",
    (float) count / (float) up_time);

Interesting! All Apache does to calculate ReqPerSec is take the total number of requests throughout the life of the parent (count) and divide that by the total uptime in seconds (up_time). So, really, ReqPerSec is just an *average* requests per second over the parent’s lifetime. That’s NOT what we want. Maybe it’s useful in some scenarios, but when you need a live metric to let you know what your server is ACTUALLY doing, this isn’t the one.

If you want an accurate request rate, your best bet is going to be to collect Total Accesses and do deltas off of the previous collection. I’m glad that’s how we did it before!

To a point, this goes into the expense of metrics. For Apache, Total Accesses is just a simple counter. It returns an accurate value live, on-the-fly, and makes it very easy for the sys admin to collect and process. Very little work is done on it’s end. Even the current implementation of ReqPerSec has very little cost: it just does some simple division on-demand, using counters it already tracks anyway.

To keep an actual average request rate, there would be some more resource involved. It would need to set a timer (ie: every 60 seconds), have a callback for the timer (to actually handle the calculation), and store counters in memory to calculate deltas. However, in reality, this will really have a very minimal impact on server performance and system load (a few extra CPU hits every minute, storing two little 64-bit counters, etc).

So what did I end up doing? I hacked up my own mod_status using shared memory!

Using the existing mod_status code, I pulled some code from a shared memory module example and smashed them together to calculate an accurate ReqPerSec metric on the fly, returned when you request server-status.

How does it work?

When you request server-status, it will go into shared memory, pull out the total accesses and uptime from the previous server-status request, do deltas on the current total accesses and uptime, then return the rate. It will then put the current total accesses and uptime back into shared memory for the next poll (base is our shared memory baseline).

ap_rprintf(r, "ReqPerSec: %g\n", (
    ( (float)count - (float)base->prev_req_count ) /
    ( (float)up_time - (float)base->prev_uptime ) )
);

There is of course all of the shared memory code, but the end result is the calculation above. We poll every 60 seconds, so we incur a small cost every minute, but, like I mentioned before, it’s not noticeable, and it gives us the view we need into the environment.

Some of you may also wonder what happens to the first request, since there isn’t any data in our shared memory space yet. For now, it will be inaccurate, but only for the first request in the lifetime of the parent, which I can deal with until I make it better.

Well, that’s all for this one. It was an interesting discovery in the world of Apache metrics collection, and an interesting adventure to hack together the shared memory module. I’m sure there are other scenarios out there like this one; if you come across one, feel free to drop a comment!

I’m off to the Apple Store now- it’s iPad launch day