18
Jun 11

a little abstraction goes a long way (fun with SOAP APIs)

I’ve been working with SOAP services a lot lately. Is that moaning I hear? It is! But you know, as long as I’m provided an API that works, and isn’t *too* slow, I’ll deal! However, it’s nice everyone finally realized XML.. well.. isn’t the best for stuff like this, and are moving away from it. Took them long enough. :D

This should be a relatively quick post, but I think the point it makes is nice. My colleague Paul once told me my code looks like a 5 year old wrote it. I try to keep things simple, and make things simple. The goal is to take something that could have been complex (or sounds complex), and break it down into little logical chunks you can make sense of the first time through. I’m also a 5 year old, so that might have something to do with it.

In this case, I was working on interacting with a SOAP API for a load balancing product. Their API is actually really nice, but with the amount of settings you can change for certain objects, their free, open code samples definitely wouldn’t fly for an enterprise-class tool. Here it is:

Original Code

 
my $result = $soap->addservice(
    SOAP::Data->name( 'name' => 'test-svc'),
    SOAP::Data->name( 'IP' => '1.2.3.4'),
    SOAP::Data->name( 'servicetype' => 'http'),
    SOAP::Data->name( 'port' => '80'),
    SOAP::Data->name( 'maxreq' => '8'),
)->result();

If this is all you need to do, then it definitely works. But, what happens if you need to set more than 5 values? Maybe you have 10 you need to set? Maybe the number of arguments is going to be dynamic every time you run it (ie: you don’t know if maxreq is going to be set EVERY time). Well, that could be a pain in the bootay.

There are a couple things we’re going to end up doing here, but first thing first: this whole multi-argument thing isn’t going to fly if I want to make things dynamic (ie: loop through a data set to build your input key/val pairs). I also reeeaaally don’t want to have to say SOAP::Data->name for every key/val pair I set. I’d rather just be able to define my arguments as a hash (or dictionary, or whatever your $langauge calls it), and let something take care of the dirty work for me. Simple!

Since our SOAP call takes an array of SOAP::Data objects, all we really need to do is make something that takes a hash, converts it into an array of SOAP::Data objects, then pass to our SOAP call. We’re going to use an arrayref to make things easier. I think pointers are fun, so I tend to just use ref’s for everything. I hope that’s okay.

 
sub hash_to_soap_data {
 
    my($hashref)    = @_;
    my $soap_data = [];
 
    foreach my $key(sort keys %{$hashref}) {
        push(@{$soap_data},SOAP::Data->name( $key => $hashref->{$key} );
    }
 
    return $soap_data;
 
}

Ta da! That wasn’t so bad, was it? Now, we can do something like this:

 
# we could also just define the hash
# as the argument to hash_to_soap_data
# if you wanted to smash it together even
# more.
 
my $hashref = (
    name => "test-svc",
    servicetype => "http",
    port => 80,
    ipaddress => "1.2.3.4",
);
 
my $soap_data = hash_to_soap_data($hashref);
my $result = $soap->addservice(@{$soap_data});

Looking better! But you know, it’s still a little bit too annoying. I don’t want to have to define a hash, pass it to a function X, then take the result of function X and pass it to function Y. The ideal code would just to be to make an API call by passing a hash to a function, and let the underlying code handle making things happen. In short, we’re going to abstract things.

This is where we decide how we want to actually interface with our function. In our case, I think something like this is nice:

my $result = call_api($func_name,$hashref);

With that example, we can just make our API call in one easy line by passing a hashref. Yes, we’ll have a function call_api() to deal with, but that’s part of the abstraction.

Ultimately, this will eventually allow us to easily change the inner-workings of call_api() later on without ever having to change the rest of our code. For example, maybe the API changes from SOAP to a RESTful style, maybe using JSON. By abstracting, you can re-write call_api() to just make RESTful calls and return JSON results.

In any case, let’s write it:

 
sub call_api {
 
    # note: $soap is our SOAP or SOAP::Lite object
    # that we have already defined elsewhere.
 
    my($func,$hashref) = @_;
    my @soap_data;
 
    foreach my $key(sort keys %{$hashref}) {
        push( @soap_data, SOAP::Data->name( $key => $hashref->{$key} );
    }
 
    my $result = $soap->$func(@soap_data);
    return $result;
 
}

With that complete, now we just implement:

Abstracted Code

 
my $result = call_api( "addservice" , {
    name => "test-svc",
    ip => "1.2.3.4",
    servicetype => "http",
    port => 80,
    maxreq => 8,    
});

Your first thought probably is: it looks almost exactly like our original code! What’s the point? You suck? The point is you have now abstracted things just enough to make things a lot easier to manage in the long run. Based on inputs, you can dynamically assign the function (ie: maybe it’s addservice to start, but based on a conditional, it may change to setservice, and all you need to change is $func_name. Your single API line still works as expected, so no need for multiple API lines. Same with your data hash. You can build your input key/val pairs manually, dynamically based on IO, other conditionals, etc, and never have to change that one friendly line:

my $result = call_api($func_name,$hashref);

It also helps with troubleshooting, because you know that all API calls ultimately funnel in to call_api(). And, like I mentioned before, if the server-side ever changes from SOAP to whatever the latest and greatest is, you should have hopefully abstracted things enough to just be able to change your single function without affecting everything else.

In practice, I’ve found this to work pretty well. There will be one-offs now and then, and you’ll have to account for them. However, over time, you’ll start to be able to identify places for abstraction beforehand, which in the long run will just help you to make pretty sweet abstractions for doing anything. This doesn’t just apply to Perl, and it doesn’t just apply to SOAP. I try to apply it to as many places I can. It tends to work best in an object-oriented codebase (yes, even Perl objects), but even abstraction via functions can be good too!

Someone joked once about “hello world” for a new programmer is 1 line, while it’s 250 lines with 5 layers of abstraction and an additional 20 comments for an experienced programmer. While it’s not far off, there’s definitely a sane balance somewhere in the middle, and that’s part of the journey ;) Enjoy!


23
Nov 10

perl typecast beware! weak typing and the range operator.

Background: I’ve been working on a suite of tools for the past few months to help with some config automation, and one of the config files allows you to put in a range (ie: 1-100, or even 0200-0399). The tool will take the range, look for running jobs with an ID in that range, then do something with it. In some cases, we do have jobs whose ID starts with a zero (or multiple zeroes), so the leading zero can be important. This is where I ran into some interesting discoveries!

Okay, so, to aid in making it easy to iterate over the range, I was using the range operator in Perl, which is the little “..” thing you can do, (or dotdot, or dot-dot, or whatever you might call it). For example:

for(1 .. 100) { 
    printf("%s\n",$_);
}

.. or, being a little more realistic:

my $start = 1;
my $end   = 100;
 
for my $i( $start .. $end ) {
    printf("%s\n",$i);
}

Run those, and you’ll get all the numbers from 1 to 100. Fantastic. Want to get even more crazy? How about letters. Yeah, it does those too. Sweet.

In this example, we’ll get everything from a to z:

my $start = "a";
my $end   = "z";
 
for my $i( $start .. $end ) {
    printf("%s\n",$i);
}

Hrmm, what about if we want the range from a to zzzzz? Yup, it does that too:

my $start = "a";
my $end   = "zzzzz";
 
for my $i( $start .. $end ) {
    printf("%s\n",$i);
}

Of course, I’m not giving you the output, because that would be insane.. but it works. Nice!

There’s actually a bunch more you can do with it, too. There’s even a “…” (dot-dot-dot) that you can do something with (but I forget what). However, it is all documented here:

http://perldoc.perl.org/perlop.html#Range-Operators.

But that’s not what this post is about. It’s about something interesting I ran into. Still interested? Good. Keep reading.

So anyway, in the case of my config parser, I would parse out ranges as tokens (ie: 0200-0300), split, and do a search across the range. For example:

# $range = "0200-0300"
my($start,$end) = $range =~ /^(\d+)\-(\d+)$/;
 
for my $id( $start .. $end ) {
   if(found_job_id($id)) {
      do_something_with($id);
   }
}

This basically allows us to lazily put large ranges into the config, but requires us to be more specific when creating jobs with new IDs (to ensure it falls within our range). In any case, this works exactly as we’d expect, leading zeroes and all.

Now, as part of error checking, I decided I should make sure that the $start value was less than the $end value, since the range operator will not work if $start is greater than $end. It will just return an empty set. This is documented and not an error, so I wanted to throw an error.

To do so, I just modified my code to implement a simple comparison check:

# $range = "0200-0300"
my($start,$end) = $range =~ /^(\d+)\-(\d+)$/;
 
if($start > $end) {
   throw_error_and_exit("Start is greater than end! Fix your range, buddy.");
}
 
for my $id( $start .. $end ) {
   if(found_job_id($id)) {
      do_something_with($id);
   }
}

The change seemed harmless enough, but when I ran the config parser again to make sure things worked, I got nothing. Literally. Nothing. It appeared as if everything broke. Weird. No errors, the range is valid (I was using 0100-0200), but, um, what?

After scratching my head for about 20 seconds, I went, “aha!” Yes, just like that.

Sure enough, when I parse out my range from the config, the values I get back are strings (which makes sense), so in the case of 0200-0300, the range operator is actually performing the operation based on the string value of each, using what they consider the “magical auto-increment algorithm” (per the Perl documentation). I don’t know what type of analysis it does, but it’s probably just based on ASCII values.

In any case, AFTER I added my new comparison operation (to check if $start was greater than $end), Perl had to typecast my variables (which were strings) to integers to compare them, therefore dropping the leading zeroes! Here’s something a little more graphic:

# $range = "0200-0300"
my($start,$end) = $range =~ /^(\d+)\-(\d+)$/; # <--- STRINGS
 
if($start > $end) { # <--- OMG NOW THEY'RE INTEGERS
   throw_error_and_exit("Start is greater than end! Fix that.");
}
 
for my $id( $start .. $end ) { # <--- HEY WHERE'D THE LEADING ZERO GO?
   if(found_job_id($id)) {
      do_something_with($id);
   }
}

This is one of those things with Perl where a mix of dynamic and weak typing features lead to strings magically (and uncontrollably) being casted to integers where needed. In most cases, having a language with both dynamic and weak typing on variables can be nice, but it can also lead to laziness and unexpected results if you’re not careful. ;)

Well, that’s all for today. Just thought I’d share that one.


23
Jul 10

offload Mac OS X swap with dynamic_pager and an SDXC card

I have an idea. ;)

So, I’ve run into some issues with OS X where it will start swapping, even though there is more than enough physical RAM available. I’m assuming this has to do with the OS deciding certain processes have been idle for long enough; it will pre-emptively swap them out with the assumption of a need for new, soon-to-be “things” requiring memory. This would give the user the perception that their machine loads up things faster when said “things” need to be loaded. That could be total BS on my part, but until I have a full technical explanation of it, that’s the story I’m going to stick to.

Now, I’m used to Linux, where you can set /proc/sys/vm/swappiness=0 and effectively stop swapping altogether. As long as there’s enough physical memory available, it will keep using it. If you run out, then OOM kill (assuming you haven’t disabled it via /proc/sys/vm/oom-kill) will come and start playing whack-a-mole with your processes. Sweet. This is good.

Since OS X doesn’t have this, what *does* it have? Well, it has a process called dynamic_pager, which oversees the management of swap (and swap files). There are two files we’re interested in here:

/sbin/dynamic_pager
/System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

The binary is obvious, but the LaunchDaemon plist is what we’re really interested in here. This is what tells dynamic_pager to kick off at launch. Now, the plist is in binary format, so to really get a look (from Terminal, unless you use the plist Editor), you can convert it to XML:

plutil -convert xml1 com.apple.dynamic_pager.plist

From there, the contents of the file will be something like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>EnableTransactions</key>
<true/>
<key>HopefullyExitsLast</key>
<true/>
<key>Label</key>
<string>com.apple.dynamic_pager</string>
<key>OnDemand</key>
<false/>
<key>ProgramArguments</key>
<array>
<string>/sbin/dynamic_pager</string>
<string>-F</string>
<string>/private/var/vm/swapfile</string>
</array>
</dict>
</plist>

Fantastic! By the way, to convert back to binary when you’re done, you can do this:

plutil -convert binary com.apple.dynamic_pager.plist

Now that we’re looking at our plist, you’ll notice the config setting for -F /private/var/vm/swapfile. This is the base name used by dynamic_pager for when it needs to create a new swapfile. If you do a ps, you’ll notice it running:

[chet@cilantro ~]$ ps ax | grep dynamic_pager
39 ?? Ss 0:00.02 /sbin/dynamic_pager -F /private/var/vm/swapfile

And if you check out /private/var/vm:

[chet@cilantro ~]$ ls -lh /private/var/vm/
-rw------- 1 root wheel 64M Jul 19 22:02 swapfile0
-rw------- 1 root wheel 64M Jul 23 21:38 swapfile1
-rw------- 1 root wheel 128M Jul 23 21:38 swapfile2

As you can see, dynamic_pager is alive and well, managing a set of swap files on my local disk. Side note about the listing above: as the need for swap increases (or decreases) with activity, dynamic_pager will take the liberty of dynamically changing the swapfile size for you. You can also specify the -S flag in your plist to specify a static file size.

In any case, here’s where the fun starts: I don’t know about you, but the disk in my MacBook Pro isn’t anything super-amazing. Maybe it’s on a 3Gb/sec SATA channel, but it’s a 5400rpm drive, and random writes usually yield somewhere around 12MB/sec. Sequential writes will hit about 60MB/sec at times, but lets be honest, random is what we should be caring about; they’re nothing spectacular.

So, when I’m in the middle of working on something, have a few Flash widgets running, doing who knows what, and I start to experience the lovely effects of swapping, it’s not always the best experience. On top of slowing things down, it’s also adding a lot of extra physical load to the disk. Poopy. I wish there was a way to decouple swap from my physical root disk, and possibly put it somewhere that might help out swap performance a bit. Oh wait, I have a thought!

Thanks to the designers at Apple, who graciously added an SD card slot to the latest line of MacBook Pro’s, we now have an option for swap: offload it to an SD card! Now, a plain old SD card might not be the best (even a class 10 would only give you 10MB/sec max, and under real world conditions it’s obviously going to be worse.. about 2MB/sec), but with the latest SDXC technology, Kingston, Toshiba (and others) have developed cards hitting between 50-60MB/sec on writes, which is very similar to the sequential writes on our local disk. Again, under real-world, random usage, it’s going to be less, but should be relatively similar to the random writes of our local disk. To try it out, just slide in your SDXC card, and update your plist to something like the following:

<string>/sbin/dynamic_pager</string>
<string>-F</string>
<string>/Volumes/sdxc_swap/swapfile</string>

After that, restart your dynamic_pager process, and you should be good to go, enjoying an offloaded swap, and totally burn out your SDXC card ;)

That being said, I’d like to mention a couple of things. First, I don’t have a high performance SDXC card of my own for testing, so performance mileage may vary depending on the card. I know Toshiba is targeting their SDXC cards for use in Netbooks, which is great; it gives us an glimmer of hope that their cards are built for some relatively heavy duty lifting- more than your standard camera SD card; maybe you won’t burn it out as quickly after all!

Another thing to point out is that the high performance cards we’re discussing are all relatively larger.. 32GB or greater in size. Chances are you’ll [hopefully] never hit this in swap. However, dynamic_pager will also take care of dumping out a sleepimage file to the swap space, which will be up to however many GB of RAM you may have; take that into consideration when picking a card. You’ll probably still have leftover space, but just a word of advice.

That being said, putting your sleepimage on an SDXC card is actually nice in the sense of security. Put your machine to sleep, pop your SDXC card out, and you basically have locked your laptop down in terms of a thief being able to try and yank whatever you were working on. There are threads out there about people dumping sleep/hibernate images and stealing sensitive data, so this is a nice little side effect of offloading swap to a little card.

In any case, that’s all for now. It was a thought I had the other day, so I figured I would share the idea with others. Maybe some people will think it sucks, but I think it’s kind of fun. If you have a card and would like to dump back some performance data to me, that would be awesome (ie: test via Xbench or the like). And, if I ever get my own card worth testing, I’ll post an update on how it went! If you’d also like to tell me how dumb I am, that’s okay too; I’m used to it. Enjoy!


11
Apr 10

writing a custom Netscaler health check for NTP

There are lots of load balancing products on the market today. Some are good, some are bad. Opinions aside, one such product is the Citrix Netscaler.

Now, the Netscaler has built in health checking for many different types of backend service protocols: HTTP, FTP, DNS, and more. However, one protocol not included is NTP. If you happen to be load balancing a farm of NTP servers, and want to ensure you’re not routing to a hosed NTP daemon (even if it’s ping-able), it would be nice to be able to health check the service [and take it out of rotation if it's not responding].

Luckily, the Netscaler provides (with documentation) a very flexible way to write your own custom health checks. Using the system, you can test whatever you want, in any way you want, with just a little Perl. Even better, the custom health check subsystem provides a few useful things to simplify the whole process, including:

  • passing in the service IP, and port, as arguments to your script
  • passing in a custom argument string (key value pairs, user/pass info, whatever)
  • handling the timeout for you from a higher level; if your check doesn’t respond within -respTimeout seconds, it will be considered a failure.

Now, for just the basics, this monitor will actually be pretty easy to put together. We are given the IP/port to send an NTP request to, and we don’t need to worry about a timeout, because the underlying subsystem will handle that for us. All we really need to do is send a request and [hope] for a response!

That being said, we’ll need to do a couple things to get to that point. These things are:

  1. Load the health check subsystem module.
  2. Load the Socket module, and create a socket to use to send the request [using the parameters passed in as args].

From that point, we can build and send our request. Since we don’t need to worry about handling a timeout, we can just sit there waiting for a response. First up:

use IO::Socket;
use Netscaler::KAS;

We’ll need both of these modules – one will tap into the health check subsystem, and the other is so we can send our UDP-based request.

Next is our function, which we’ll just call ntp_probe. It assigns a few variables from $_[0] (the service IP) and $_[1] (the service port) as arguments, which get passed in from work within probe().

So, the magic-fu to this is that, later on, we will pass our ntp_probe function as a coderef to the KAS probe() command. That will ultimately result in our function being called with the arguments passed in; it is similar to this:

$custom_function->($host,$port,$args);

If you look at the KAS code (and since it’s perl, you can), you can see the exact line; this just gives you an idea of what’s going on.

In any case, after that, we’ll create a new UDP socket, send the NTP request message down it, and either:

  • return 1 if something wasn’t defined (ip, port, or sock) – failure!
  • return 0 if we got a response – success!

Again, we don’t need to worry about timeouts, since the subsystem will handle that for us. So, here’s the function in it’s entirety:

sub ntp_probe {

    my $host    = $_[0];
    my $port    = $_[1];
    my $req     = "\010"."\0"x47;

    if(!$host || !$port) {
        return(1,"Host or port not specified.");
    }

    my $sock = IO::Socket::INET->new(
        Proto     => "udp",
        PeerAddr  => $host,
        PeerPort  => $port,
    );

   if($sock) {
        $sock->send($req);
        $sock->recv($_,1);
        return 0;
    }

    return(1,"Could not create socket");

}

Not bad, was it? Now that we have loaded our modules, and defined a function, the next step is to call the probe() method, with a coderef to our new function, to tell KAS what function to use for probing:

probe(\&ntp_probe);

And you’re off to the races! Just don’t forget to start your script with #!/usr/bin/perl, and it should be good to go ;)

Now, you might be asking “how do I actually configure the Netscaler to use this?” That’s also just as easy. Let’s pretend our new script is called customNTP.pl. Just plop it into the /nsconfig/monitors/ directory, and do something like this from the CLI:

add lb monitor "custom-ntp-mon" USER -scriptName customNTP.pl
   -scriptArgs 1 -dispatcherIP 127.0.0.1 -dispatcherPort 3013
   -interval 20 -respTimeout 3

With that line, you will now have a working NTP healthcheck (20 second intervals, 3 second timeout). Just bind it to your service(s) that need it and enjoy!. Note that the type of monitor is USER; this is the type used for custom health checks.

Well, that’s all for this evening. There are bits and pieces of info like this around the web, on the Citrix AppExpert site, etc. And, even though it wasn’t too difficult to figure out, hopefully this will help someone out some day.

Take care!


03
Apr 10

apache mod_status: the misleading ReqPerSec metric

To be honest, I had never thought twice about it. At my past employer, we had implemented tracking request rate for the Apache hosts by parsing out Total Accesses (ie: Total Requests) from server-status (thanks to our friendly module mod_status), publish that every minute, and let our metrics collection subsystem do a delta between the current and previous to determine a request rate to publish.

At my new employer, I put something together where the collection subsystem won’t refer to the previous total for deltas: it just takes whatever you give it and relays that to the metrics backend. Most applications provide you with a request/sec metric these days, so I figured I’d just use that. Instead of doing deltas on TotalAccesses, I started just relaying ReqPerSec instead. Simple enough, right? For those of you not familiar, here’s an example of server-status output:

Total Accesses: 400
Total kBytes: 800
CPULoad: 5
Uptime: 12345
ReqPerSec: 20
BytesPerSec: 20000
BytesPerReq: 1024
BusyWorkers: 15
IdleWorkers: 5

Well, amidst my metrics collection testing for the new system, I stumbled upon an interesting detail: I would be nailing the Apache instance with a flood of requests using my friend ab (aka: apachebench). However, I immediately noticed ReqPerSec wasn’t accurate at all. For example, I would be averaging a test of 300-400 req/s, but ReqPerSec would be displaying 21 req/s. It was as if I wasn’t sending it requests.

That being said, the TotalAccesses metric was reflecting the requests I was sending, so I knew the web server was receiving them. Interesting.

It got me thinking: “I wonder how Apache determines ReqPerSec?” Time for some source code action! Sure enough, here’s what I found:

ap_rprintf(r, "ReqPerSec: %g\n",
    (float) count / (float) up_time);

Interesting! All Apache does to calculate ReqPerSec is take the total number of requests throughout the life of the parent (count) and divide that by the total uptime in seconds (up_time). So, really, ReqPerSec is just an *average* requests per second over the parent’s lifetime. That’s NOT what we want. Maybe it’s useful in some scenarios, but when you need a live metric to let you know what your server is ACTUALLY doing, this isn’t the one.

If you want an accurate request rate, your best bet is going to be to collect Total Accesses and do deltas off of the previous collection. I’m glad that’s how we did it before!

To a point, this goes into the expense of metrics. For Apache, Total Accesses is just a simple counter. It returns an accurate value live, on-the-fly, and makes it very easy for the sys admin to collect and process. Very little work is done on it’s end. Even the current implementation of ReqPerSec has very little cost: it just does some simple division on-demand, using counters it already tracks anyway.

To keep an actual average request rate, there would be some more resource involved. It would need to set a timer (ie: every 60 seconds), have a callback for the timer (to actually handle the calculation), and store counters in memory to calculate deltas. However, in reality, this will really have a very minimal impact on server performance and system load (a few extra CPU hits every minute, storing two little 64-bit counters, etc).

So what did I end up doing? I hacked up my own mod_status using shared memory!

Using the existing mod_status code, I pulled some code from a shared memory module example and smashed them together to calculate an accurate ReqPerSec metric on the fly, returned when you request server-status.

How does it work?

When you request server-status, it will go into shared memory, pull out the total accesses and uptime from the previous server-status request, do deltas on the current total accesses and uptime, then return the rate. It will then put the current total accesses and uptime back into shared memory for the next poll (base is our shared memory baseline).

ap_rprintf(r, "ReqPerSec: %g\n", (
    ( (float)count - (float)base->prev_req_count ) /
    ( (float)up_time - (float)base->prev_uptime ) )
);

There is of course all of the shared memory code, but the end result is the calculation above. We poll every 60 seconds, so we incur a small cost every minute, but, like I mentioned before, it’s not noticeable, and it gives us the view we need into the environment.

Some of you may also wonder what happens to the first request, since there isn’t any data in our shared memory space yet. For now, it will be inaccurate, but only for the first request in the lifetime of the parent, which I can deal with until I make it better.

Well, that’s all for this one. It was an interesting discovery in the world of Apache metrics collection, and an interesting adventure to hack together the shared memory module. I’m sure there are other scenarios out there like this one; if you come across one, feel free to drop a comment!

I’m off to the Apple Store now- it’s iPad launch day ;)