So, I’ve run into some issues with OS X where it will start swapping, even though there is more than enough physical RAM available. I’m assuming this has to do with the OS deciding certain processes have been idle for long enough; it will pre-emptively swap them out with the assumption of a need for new, soon-to-be “things” requiring memory. This would give the user the perception that their machine loads up things faster when said “things” need to be loaded. That could be total BS on my part, but until I have a full technical explanation of it, that’s the story I’m going to stick to.

Now, I’m used to Linux, where you can set /proc/sys/vm/swappiness=0 and effectively stop swapping altogether. As long as there’s enough physical memory available, it will keep using it. If you run out, then OOM kill (assuming you haven’t disabled it via /proc/sys/vm/oom-kill) will come and start playing whack-a-mole with your processes. Sweet. This is good.

Since OS X doesn’t have this, what *does* it have? Well, it has a process called dynamic_pager, which oversees the management of swap (and swap files). There are two files we’re interested in here:

/sbin/dynamic_pager
/System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

The binary is obvious, but the LaunchDaemon plist is what we’re really interested in here. This is what tells dynamic_pager to kick off at launch. Now, the plist is in binary format, so to really get a look (from Terminal, unless you use the plist Editor), you can convert it to XML:

plutil -convert xml1 com.apple.dynamic_pager.plist

From there, the contents of the file will be something like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>EnableTransactions</key>
<true/>
<key>HopefullyExitsLast</key>
<true/>
<key>Label</key>
<string>com.apple.dynamic_pager</string>
<key>OnDemand</key>
<false/>
<key>ProgramArguments</key>
<array>
<string>/sbin/dynamic_pager</string>
<string>-F</string>
<string>/private/var/vm/swapfile</string>
</array>
</dict>
</plist>

Fantastic! By the way, to convert back to binary when you’re done, you can do this:

plutil -convert binary com.apple.dynamic_pager.plist

Now that we’re looking at our plist, you’ll notice the config setting for -F /private/var/vm/swapfile. This is the base name used by dynamic_pager for when it needs to create a new swapfile. If you do a ps, you’ll notice it running:

[chet@cilantro ~]$ ps ax | grep dynamic_pager
39 ?? Ss 0:00.02 /sbin/dynamic_pager -F /private/var/vm/swapfile

And if you check out /private/var/vm:

[chet@cilantro ~]$ ls -lh /private/var/vm/
-rw------- 1 root wheel 64M Jul 19 22:02 swapfile0
-rw------- 1 root wheel 64M Jul 23 21:38 swapfile1
-rw------- 1 root wheel 128M Jul 23 21:38 swapfile2

As you can see, dynamic_pager is alive and well, managing a set of swap files on my local disk. Side note about the listing above: as the need for swap increases (or decreases) with activity, dynamic_pager will take the liberty of dynamically changing the swapfile size for you. You can also specify the -S flag in your plist to specify a static file size.

In any case, here’s where the fun starts: I don’t know about you, but the disk in my MacBook Pro isn’t anything super-amazing. Maybe it’s on a 3Gb/sec SATA channel, but it’s a 5400rpm drive, and random writes usually yield somewhere around 12MB/sec. Sequential writes will hit about 60MB/sec at times, but lets be honest, random is what we should be caring about; they’re nothing spectacular.

So, when I’m in the middle of working on something, have a few Flash widgets running, doing who knows what, and I start to experience the lovely effects of swapping, it’s not always the best experience. On top of slowing things down, it’s also adding a lot of extra physical load to the disk. Poopy. I wish there was a way to decouple swap from my physical root disk, and possibly put it somewhere that might help out swap performance a bit. Oh wait, I have a thought!

Thanks to the designers at Apple, who graciously added an SD card slot to the latest line of MacBook Pro’s, we now have an option for swap: offload it to an SD card! Now, a plain old SD card might not be the best (even a class 10 would only give you 10MB/sec max, and under real world conditions it’s obviously going to be worse.. about 2MB/sec), but with the latest SDXC technology, Kingston, Toshiba (and others) have developed cards hitting between 50-60MB/sec on writes, which is very similar to the sequential writes on our local disk. Again, under real-world, random usage, it’s going to be less, but should be relatively similar to the random writes of our local disk. To try it out, just slide in your SDXC card, and update your plist to something like the following:

<string>/sbin/dynamic_pager</string>
<string>-F</string>
<string>/Volumes/sdxc_swap/swapfile</string>

After that, restart your dynamic_pager process, and you should be good to go, enjoying an offloaded swap, and totally burn out your SDXC card

That being said, I’d like to mention a couple of things. First, I don’t have a high performance SDXC card of my own for testing, so performance mileage may vary depending on the card. I know Toshiba is targeting their SDXC cards for use in Netbooks, which is great; it gives us an glimmer of hope that their cards are built for some relatively heavy duty lifting- more than your standard camera SD card; maybe you won’t burn it out as quickly after all!

Another thing to point out is that the high performance cards we’re discussing are all relatively larger.. 32GB or greater in size. Chances are you’ll [hopefully] never hit this in swap. However, dynamic_pager will also take care of dumping out a sleepimage file to the swap space, which will be up to however many GB of RAM you may have; take that into consideration when picking a card. You’ll probably still have leftover space, but just a word of advice.

That being said, putting your sleepimage on an SDXC card is actually nice in the sense of security. Put your machine to sleep, pop your SDXC card out, and you basically have locked your laptop down in terms of a thief being able to try and yank whatever you were working on. There are threads out there about people dumping sleep/hibernate images and stealing sensitive data, so this is a nice little side effect of offloading swap to a little card.

In any case, that’s all for now. It was a thought I had the other day, so I figured I would share the idea with others. Maybe some people will think it sucks, but I think it’s kind of fun. If you have a card and would like to dump back some performance data to me, that would be awesome (ie: test via Xbench or the like). And, if I ever get my own card worth testing, I’ll post an update on how it went! If you’d also like to tell me how dumb I am, that’s okay too; I’m used to it. Enjoy!

Now, that got me thinking. If sshd NEVER hits MaxStartups via the OS X default launchd setup, then it should be relatively easy to DoS an OS X box, right? By default, OS X only allows 256 maximum user processes and 512 file descriptors, so with each new fork of an sshd parent, you can easily eat into that space and essentially block anyone from doing anything once you hit the limit. Even better (or worse), I’m guessing most OS X (and OS X Server) users don’t change any of these defaults: they keep sshd via launchd, and never run into any type of issue where they’d have to increase maxprocs or max file descriptors.

With this idea in my head, I chose to use my wife’s Macbook Pro as our test server; it could use a little exercise. So, I did what any OS X user would do: I went to System Preferences > Sharing, and then enabled Remote Login. Of course, if  I do a ‘ps’ to see if it’s running, I won’t see anything, because launchd is just hanging out waiting for someone to hit port 22.

Next thing, my Macbook Pro will have the same system defaults, so I have to increase those to be able to exceed them limit on the receiving end. That’s easy enough- let’s increase our maxprocs and max fds:

chets-laptop $ sudo ulimit -n 2048
chets-laptop $ sudo ulimit -u 1024

There we go. Now, with my wife’s laptop willing and ready, let’s take out our trusty for loop:

chets-laptop $ for i in `jot 512 1`; do ssh 192.168.1.110 & done

Now, with that running, I jump over to my wife’s laptop (Terminal already open), and I try to do an ‘ls’. Sure enough:

wifes-poor-laptop $ ls
-bash: fork: Resource temporarily unavailable

Whoops! At this point, with a flood of SSH connections incoming, the only real thing I can do to stop the DoS is unload sshd from launchd. However, since the machine has hit the maximum number of processes allowed, I can’t even run the command to unload sshd from launchd. I can’t kill the sshd processes either, because I’d only be killing each individual parent, and launchd would just keep forking them off with each new connection. Regardless, killing won’t work anyway, just like unloading sshd from launchd:

wifes-poor-laptop $ sudo launchctl unload /System/Library/LaunchDaemons/ssh.plist
-bash: fork: Resource temporarily unavailable

also tried:

wifes-poor-laptop $ sudo killall sshd
-bash: fork: Resource temporarily unavailable

Ouch. Of course, I also tried opening some apps (System Preferences, Address Book, etc), and those would just bounce and bounce forever; the system had hit the hard limit, so there was nothing that could be done.

At this point, the ONLY way I could stop the attack was to take the machine off of the network, resulting in the sshd forks timing out and exiting. My wife’s laptop was actually wired at the time; if it were on Airport, I’m not sure if the system would have required an extra process resource to actually disable Airport, meaning a machine on the net via Airport would be totally, 100% useless.

Regardless, I think this is something people should be aware of before starting up Remote Login. Of course, running any type of remote login daemon is always a security risk, but if you’re on an open network with a public facing IP, you’re pretty much exposing your machine to being easily DoS’d by someone who knows how to write a for loop. You could, of course, set up TCP wrappers to buffer some of that, run ipfw, and whatever else, but I’m guessing that most OS X users aren’t going to be doing much in the way of this.

My suggestion: set up your hosts.allow/deny, configure ipfw to look for connection flooding on port 22, take sshd out of launchd, and, of course, only run sshd if you really need to. If someone out there wants to lock up your machine, and you’re running the OS X defaults, they can do it in about 10 seconds, and you won’t have a clue what hit you until you suddenly can’t do anything with your machine anymore.

All things aside, I love OS X and think it’s a great OS. There are just a few design decisions here and there that still need to be tweaked. Running sshd via the equivalent of xinetd is one of them. Good luck!

We have a rather heavily loaded administrative server, an Xserve, running Mac OS X Server. Additionally, we run a modified sshd configuration, where sshd runs as a standalone service, instead of via launchd as is with the default system installation. Is it relevant? Yes.

So, Friday evening rolls around (of course), and I get an e-mail from a co-worker saying that when she tries to log in, she gets that oh-so-common error message:

ssh_exchange_identification: Connection closed by remote host

Usually, in all of my experiences, this has been related to TCP wrappers; you’ve got a hosts.allow or hosts.deny set up, and you’re blocking (purposely or by accident), and just need to fix the config. Sometimes you’re totally screwed, but whatever. In our case, however, we aren’t using TCP wrappers. We could, since the default sshd in OS X does support it:

[chet@myhost]$ strings /usr/sbin/sshd  | grep wrap
Connection refused by tcp wrapper
libwrap refuse returns

or, if you have Xcode installed, you can use otool (the OS X equivalent to ldd in Linux):

[chet@myhost]$ otool -L /usr/sbin/sshd | grep libwrap
/usr/lib/libwrap.7.dylib (compatibility version 7.0.0, current version 7.6.0)

However, we don’t have a hosts.(allow|deny), so that’s not the issue. Thinking further, I remembered someone mentioning that the box was under more load than usual, and may have hit some type of system limit (maxprocs, maxttys, etc). To test, I basically flooded the box with ssh connection requests using a super-basic loop that a 3rd grader would write:

$ for connection in `jot 500 1`; do
$   ssh myhost &
$ done

Sure enough, right when I started the loop, I immediately began getting ssh_exchange_identification errors thrown back at me, whereas a single login not amidst a loop would work just fine. Bingo. So, it wasn’t a system limit per-se, but it was some type of sshd limit that was being triggered within the daemon itself.

Looking through the man pages, I came across an sshd_config option called MaxStartups. Here’s what the man page had to say:

MaxStartups

Specifies the maximum number of concurrent unauthenticated con-
nections to the sshd daemon.  Additional connections will be
dropped until authentication succeeds or the LoginGraceTime
expires for a connection.  The default is 10.

This sounded like a totally plausible cause for the issue. When I would flood, it would make requests so fast that sshd couldn’t keep up with authentications, so sshd would reach the MaxStartups limit and start blocking (to protect against DoS attacks, etc). To test it out, I changed MaxStartups from 10 to 200, ran my flood loop again, and sure enough, was able to connect without any errors. Perfect.

However, how come we hadn’t run into it before? Did it have something to do with us running sshd as a standalone daemon instead of via launchd? The standalone configuration is new for us, so we’re still keeping our eye out for bugs. To test, I ran my same flooding loop on an OS X Server machine running the system default (via launchd) sshd configuration. Sure enough, the issue did NOT present itself for that machine. Interesting.

Now, why is this, you may ask? It has to do with the way the sshd daemon runs via launchd.

The launchd service is very similar to xinetd when it comes to running network services: launchd knows what network services are configured under it, and it knows the ports those network services should accept connections from. So, launchd itself will listen on those ports. When a new connection request comes in (example: ssh), launchd will see it’s for port 22/sshd, fork off an sshd parent, and your connection gets handed off to that parent. If another connection comes in, launchd will fork off yet another sshd parent, and that 2nd user will get the 2nd parent. See the issue?

With the MaxStartups option in sshd_config, it relies on the sshd parent being the process handling all incoming connections, and forking off children for each new connection. If it sees that >= MaxStartups children are in an un-authenticated state, it will block new connections from establishing.

In the case of launchd, however, each connection is a parent forked from launchd, so no parent will ever see more than 1 connection (since it will have no children). In that respect, it totally subverts the sshd_config option for MaxStartups, opening up your machine to DoS attacks via sshd floods. Awesome.

In any case, now that we understand the issue, how come it was presenting itself under a scenario where people weren’t trying to flood it with ssh connections? The answer lies somewhere between system load, DirectoryServices, and the pam_securityserver.so PAM module.

When an ssh request comes in, and PAM is enabled, sshd will use the options defined in the /etc/pam.d/sshd file to determine what needs to be done to authorize the user. In our case, we have this configuration (slightly modified from the default OS install, since we’re not running via launchd):

[chet@myhost] $ cat /etc/pam.d/sshd
# sshd: auth account password session
auth       required       pam_nologin.so
auth       optional       pam_afpmount.so
auth       sufficient     pam_securityserver.so
auth       sufficient     pam_unix.so
auth       required       pam_deny.so
account    required       pam_securityserver.so
password   required       pam_deny.so
session    required       pam_permit.so
session    optional       pam_afpmount.so

Looking at the file, we notice a couple of entries for pam_securityserver.so. This module is actually the bread and butter of the authentication process: it queries the local DirectoryServices agent running on the machine to do the user authentication, and will receive a success or failure from the agent. The DirectoryServices agent can be configured to authenticate for local users, remote users from an LDAP directory server, or a mix of both. It’s versatile, but it does a good amount of work. In the case of it being bound to a remote LDAP server, it can also be very slow (depending on if you’re caching account results or not).

Our issue was two-fold: at the time, the host was heavily loaded with some CPU and network intensive processes. The CPU intensive processes were slowing down the communication between pam_securityserver.so and DirectoryServices, and the network intensive processes were slowing down the communication between DirectoryServices and our remote LDAP server. Between the both of these, authentications became super slow, and incoming, un-authentication connections were slowly creeping up (especially due to the scripted cron-job logins that had no logic to drop a hung login – awesome).

However, at that time, things were actually still okay. Slow, but people could still get in. Unfortunately (or luckily), someone sent out a chat message saying the host was acting slow, which of course prompted everyone to try logging in at the same time. At that point, 10 MaxStartups was hit almost immediately (ie: 10 people tried logging in, and since authentications were taking forever, all were stuck as un-authenticated), resulting in the ssh_exchange_identification error being sent back to all subsequent logins immediately (this was a nice bit of detail to note: the sshd daemon itself was still very quick to respond, but as soon as it had to hand an authentication back off to PAM, it just sat around waiting).

Ultimately, the resolution for this was we increased MaxStartups for our heavily-loaded machines (which is still better than the un-regulated launchd sshd), and we will be setting up  separate server to handle all the jobs that were running.

Looking at other options, I noticed while writing this that pam_securityserver.so comes BEFORE pam_unix.so. As a test, I may try using pam_unix.so first, allowing root logins to bypass DirectoryServices, hitting passwd/shadow directly, and hopefully allowing us to jump in on what seems like a hung system due to DS being super slow. I’ll try to get back with an update if I end up testing this.

That’s all from me for now. Feel free to comment on your own experiences, knowledge, other ideas, or if you’ve run into the same thing. Enjoy!