wildpointer*referencing whatever comes to mind, by chet nichols

Background: I’ve been working on a suite of tools for the past few months to help with some config automation, and one of the config files allows you to put in a range (ie: 1-100, or even 0200-0399). The tool will take the range, look for running jobs with an ID in that range, then do something with it. In some cases, we do have jobs whose ID starts with a zero (or multiple zeroes), so the leading zero can be important. This is where I ran into some interesting discoveries!

Okay, so, to aid in making it easy to iterate over the range, I was using the range operator in Perl, which is the little “..” thing you can do, (or dotdot, or dot-dot, or whatever you might call it). For example:

for(1 .. 100) { 
    printf("%s\n",$_);
}

.. or, being a little more realistic:

my $start = 1;
my $end   = 100;
 
for my $i( $start .. $end ) {
    printf("%s\n",$i);
}

Run those, and you’ll get all the numbers from 1 to 100. Fantastic. Want to get even more crazy? How about letters. Yeah, it does those too. Sweet.

In this example, we’ll get everything from a to z:

my $start = "a";
my $end   = "z";
 
for my $i( $start .. $end ) {
    printf("%s\n",$i);
}

Hrmm, what about if we want the range from a to zzzzz? Yup, it does that too:

my $start = "a";
my $end   = "zzzzz";
 
for my $i( $start .. $end ) {
    printf("%s\n",$i);
}

Of course, I’m not giving you the output, because that would be insane.. but it works. Nice!

There’s actually a bunch more you can do with it, too. There’s even a “…” (dot-dot-dot) that you can do something with (but I forget what). However, it is all documented here:

http://perldoc.perl.org/perlop.html#Range-Operators.

But that’s not what this post is about. It’s about something interesting I ran into. Still interested? Good. Keep reading.

So anyway, in the case of my config parser, I would parse out ranges as tokens (ie: 0200-0300), split, and do a search across the range. For example:

# $range = "0200-0300"
my($start,$end) = $range =~ /^(\d+)\-(\d+)$/;
 
for my $id( $start .. $end ) {
   if(found_job_id($id)) {
      do_something_with($id);
   }
}

This basically allows us to lazily put large ranges into the config, but requires us to be more specific when creating jobs with new IDs (to ensure it falls within our range). In any case, this works exactly as we’d expect, leading zeroes and all.

Now, as part of error checking, I decided I should make sure that the $start value was less than the $end value, since the range operator will not work if $start is greater than $end. It will just return an empty set. This is documented and not an error, so I wanted to throw an error.

To do so, I just modified my code to implement a simple comparison check:

# $range = "0200-0300"
my($start,$end) = $range =~ /^(\d+)\-(\d+)$/;
 
if($start > $end) {
   throw_error_and_exit("Start is greater than end! Fix your range, buddy.");
}
 
for my $id( $start .. $end ) {
   if(found_job_id($id)) {
      do_something_with($id);
   }
}

The change seemed harmless enough, but when I ran the config parser again to make sure things worked, I got nothing. Literally. Nothing. It appeared as if everything broke. Weird. No errors, the range is valid (I was using 0100-0200), but, um, what?

After scratching my head for about 20 seconds, I went, “aha!” Yes, just like that.

Sure enough, when I parse out my range from the config, the values I get back are strings (which makes sense), so in the case of 0200-0300, the range operator is actually performing the operation based on the string value of each, using what they consider the “magical auto-increment algorithm” (per the Perl documentation). I don’t know what type of analysis it does, but it’s probably just based on ASCII values.

In any case, AFTER I added my new comparison operation (to check if $start was greater than $end), Perl had to typecast my variables (which were strings) to integers to compare them, therefore dropping the leading zeroes! Here’s something a little more graphic:

# $range = "0200-0300"
my($start,$end) = $range =~ /^(\d+)\-(\d+)$/; # <--- STRINGS
 
if($start > $end) { # <--- OMG NOW THEY'RE INTEGERS
   throw_error_and_exit("Start is greater than end! Fix that.");
}
 
for my $id( $start .. $end ) { # <--- HEY WHERE'D THE LEADING ZERO GO?
   if(found_job_id($id)) {
      do_something_with($id);
   }
}

This is one of those things with Perl where a mix of dynamic and weak typing features lead to strings magically (and uncontrollably) being casted to integers where needed. In most cases, having a language with both dynamic and weak typing on variables can be nice, but it can also lead to laziness and unexpected results if you’re not careful.

Well, that’s all for today. Just thought I’d share that one.

So I like using Perl, if anyone hasn’t realized yet. Python seems awesome and all, but I haven’t had the time to start writing things in Python. I’m still at the stage when, if a tool needs to be put together, it needs to be done ASAP; I can’t delay trying to figure out Python particulars.

In any case, one of our metrics publishing systems is wrapped by a few different object classes I put together in Perl (follows a certain protocol created by another group, etc). Of the many data types supported by our metrics receivers, we can do things like 32-bit unsigned integers, single precision (32-bit) floating points, and a few others.

Now, when I would publish the 32-bit uint’s, it would work great- no issues. The receivers would expect a byte stream, so I would use pack() to send my values as binary. For example:

$val=29;
print $sock pack('N',$val);

Easy, whatever. However, when I would send values as a single precision float, it would totally be hosed on the other end. For example:

$val="30.0";
print $sock pack('f',$val);

On the other end, they would get some massive negative number. I don’t deal with this too much, things usually just work. Thankfully, my mind was jogged in regards to standards- there are lots of standards around byte order for integers (16 bit, 32 bit, 64 bit), but not as much for float; one architecture may use big-endian, and another architecture might use little-endian.

In my searches, I found that Intel stores floating points in little-endian byte order, and Sparc stores them in big-endian byte order. Sure enough, I was trying to publish from an Intel-based BSD host to a Sparc based Solaris host. Aha!

Luckily, pack() can *also* handle this for us! With a simple carat added to the templates section of pack(), I can easily pack the value in big-endian byte order and resolve our issue. I do that like so:

$val="30.0";
print $sock pack('f>',$val);

And, ta da! Worked like a charm. Was it difficult to get it working? No. But, it was fun to have to deal with something like this. On any given day, you don’t really worry about endian-ness; things just work. This was one of those days where I actually had to use the old noggin, and that was refreshing.

I’m not sure how I’ve never run into this issue before. In some work I was doing recently, I ran into what I thought was a nasty bug [in my code] but couldn’t explain it. Without thinking, I started trying to undo this, fix that, hack this, and ignore that.

After the meeting I was in finished, I was able to sit down and actually put some thought into what was happening. The one thing that stood out was interesting, but until then, I had no clue it was how Perl interpreted [in that scenario]. So, I wrote a short test script, sure enough proving the theory, and was able to fix my error.

Take a look at this:

1
2
3
4
5

my @animals = ("cat","dog","emu","frog");
 
foreach my $animal(@animals) {
    printf("do not eat the %s\n",$animal);
}

which dumps out this:

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog


In my head, the code above would work like this: for each element in the array @animals, copy the data from that element into a new scalar $animal, then print it out. Simple enough. Now, consider this sample:

1
2
3
4
5
6
7
8
9
10
11
12

my @animals = ("cat","dog","emu","lemur");
 
foreach my $animal(@animals) {
    printf("do not eat the %s\n",$animal);
    $animal = sprintf("rabid %s",$animal);
}
 
printf("\n");
 
foreach my $animal(@animals) {
    printf("do not eat the %s\n",$animal);
}

I had *EXPECTED* it to output this:

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog


For the first loop, again, my thinking was that we copy each array element’s data into the new scalar $animal, print it, modify it (by adding ‘rabid’ to it), but do nothing with the modification (we are just modifying $animal, which should be assigned by copy, of which would be lost when we iterate to the next element). Then, in our next loop, we iterate over @animals again, initializing $animal yet again for each element, so we just hit the un-modified @animals array and see the same thing.

[un]Surprisingly, I was totally wrong. This was the output I got:

do not eat the cat
do not eat the dog
do not eat the emu
do not eat the frog

do not eat the rabid cat
do not eat the rabid dog
do not eat the rabid emu
do not eat the rabid frog


Wow! What happened? Turns out, like the title suggests, when you use foreach in Perl, it actually assigns $animal to be a pointer (reference, whatever) to that array element, and not a copy. When you make any changes, the change applies directly to the element the array. As another example, it’s for-loop equivalent would be this:

1
2
3
4
5
6

my @animals = ("cat","dog","emu","lemur");
 
for( my $index=0 ; $index < @animals ; $index++ ) {
    printf("do not eat the %s\n", $animals[$index] );
    $animals[$index] = sprintf( "rabid %s", $animals[$index] );
}

Fun! Doing some research after the fact, it turns out there has been some mild discussion on the topic. Some consider it a bug, but in reality, it works by design. What does this mean for you? Well, if you happen to be iterating over an array using foreach, and want to modify each element, and plan on looping over the array multiple times, make sure you understand what’s going on behind the scenes, or else you’ll run into the same issue I did. You can create a temporary variable (which will assign by copy), ie:

my $new_animal = $animal;

or you can iterate using a for-loop and just do it like this:

my $animal = $animals[$index];

There of course may be some cases where you want to take advantage of the referencing feature, and if you do, all the more power to you.

Well, that’s all I have for now. Hopefully this helps someone out some day. Enjoy!