The DF Lab

The Competition

On Saturday, myself and 7 of my classmates from GWU had a chance to head up to Lancaster, PA to the home of White Wolf Security for the 4th Annual Mid-Atlantic Collegiate Cyber Defense Competition Qualifying Round. At this round of the competition along with GWU were George Mason, Jameson Madison, and Millersville Universities. For those who aren’t familiar, the competition puts the students in the roles of system administrators who were recently hired to secure and maintain a company’s network. The whole affair is pretty exciting, and the pressure can get very intense. While attempting to prevent and root out attacks from an all-volunteer (but skilled) red team sitting in another room, a white team also throws business injects at us that have us do things like install wikis, set up PKI, and create office templates for our company. We get scored separately on attack prevention, injects, and service uptime, and at the end of the day the top two teams move on to the next round. The whole competition ran for about 7 hours, and we were getting pounded from minute 1.

The Plan

After experiencing the chaos last year, we put together a list of basic things to do as soon as everything started that would keep out the easiest attacks. After blocking all external traffic with our firewall (for a few minutes so we could have some “safe time”), we set out to do these things in the first 15 minutes or so. This was just changing all the passwords on the boxes, killing extraneous services, setting client firewalls, and backing up important data and configuration files. I only managed to get to changing passwords on the boxes I was handling. They gave us 4 Linux boxes, and those were the ones I was in charge of. They weren’t the newest versions of the OS’s, and for the life of me I can’t understand how our Nagios box (Fedora, I believe) didn’t come with lsof, but I did my best to get everything locked down.

The First Problem

Well, the best plans of mice and men blah blah blah, and in the few minutes it took our firewall guy to figure out where the admin console was, the red had team managed to get onto two of the Linux boxes and leave their mark before I had a chance to change the root passwords. After about an hour, I noticed the intrusion on one of the boxes as I attempted to set up iptables and noticed that there were a bunch of identical ACCEPT rules in there that I didn’t put there. It was go time.

The Source

I called over our team captain to let him know there was a problem, and I set out to figure out just what was going on. I flushed the tables, set the firewall policies to DROP, and hopped over to /sbin to take a look if anything seemed weird. After checking iptables again, I noticed some more ACCEPT rules were in there. I cleared them out and opened the crontab to see if anything was running; it wasn’t. Not sure what was going on, I took a moment to restart SSH to boot off any active connections, just in case. Upon examining the files in /sbin, there were some that were world-writable. I knew that wasn’t quite right. One of those files, however, was iptables. At cappy’s suggestion, I viewed the contents of the file, and sure enough it was a perl script instead of the iptables binary. Since I was under the gun I didn’t quite deduce what the script did, but it called the real iptables (which they renamed) with ACCEPT commands, instead of the ones I kept giving it. While the real iptables was mentioned in that perl script, I didn’t quite catch on right away, so I looked at the size of iptables on another computer and looked for a binary in /sbin with a similar size, and found it. After that, I chmod-ed all the files in /sbin to remove world-writability, to prevent any further problems in case of non-root intrusion.

The Solution

At this point, I had found intruder access, a malicious script, and a moved iptables. However, more and more ACCEPTs kept being added to my chains. Once again at cappy’s suggestion, I moved the real iptables to another name, and left their script in place as “evidence,” and in case they had any mechanism for replacing it. This seemed to finally stop the problem. At this point, I just needed to figure out where our attackers came from. The rules of the competition say we can’t completely block any IP addresses without approval from the white team, which will come if we have details proving that the IP is malicious. I believe the reasoning for this is so that we can’t just block any IP range, as well as the fact that the scoring bot shifts IPs, so we could also screw ourselves if we just blocked lots of them.

The Culprit

I needed to find out the intruder, but I didn’t know how. I took a look in /var/log and saw a bunch of files, more than I usually see, so apparently I don’t log enough on my own computer. My first look was at /var/log/messages, but that didn’t yield anything of value. Next, I stumbled across /var/log/secure, which seemed to be a log of SSH activity. I hit the jackpot, because I found logins about an hour and a half prior by two specific IP addresses. I was ecstatic. This was our culprit. I was surprised that they didn’t delete such logs, but perhaps they didn’t think of it, or were instructed not to by the white team, as not to make our jobs of tracking them impossible. In any case, I saved the log to a file, filled out an incident report, and sent it over to the white team. They looked over the report and checked on something (I honestly don’t know what) and then let us block the IPs. Mission accomplished.

The Wikis

Well, at least that mission was accomplished. I felt pretty good a little after 11am when this all was wrapped up, but that quickly faded as business injects got annoying. We had to install wiki software, which gave us infinite problems. We first tried MediaWiki, which was a bust because our database server was using MySQL 3.x. 3.x? What the hell? I’ve never seen that anywhere before. My impeccable sources tell me that it’s about 9 years old! Yeah, pretty egregious, but there wasn’t too much we could do about it under the circumstances. So we looked for other softwares, of which there were many. However, after failing to find Tigerwiki (apparently it’s discontinued) and having ridiculous troubles with MoinMoin and TikiWiki, we ended up running out of time and failing the inject. That wouldn’t have been so bad if it weren’t for having another inject which built off of that one later in the day. So that sucked. In the end, we found an older version of MediaWiki (why didn’t we think of it earlier?) and installed that for the second inject, but we ran out of time and failed. And in that last bit when I say “we,” I mean two of my teammates, because I was sick of wikis and had to step away before bashing the computer with a chair.

The Cable

The rest of the day was relatively less pressure for me, just keeping a check on my systems, handling another inject, and trying to get our damn Nagios box to actually work. For some reason, it wasn’t connected to anything. We couldn’t explain it, though we thought our routes were a bit screwy. After a lot of investigation, one of my teammates brilliantly found that there was no network cable in the computer. I know, I know, that’s normally the first thing to check, but we were given computers and were allowed to assume that there’d at least be cables in everything! And it’s not like it had come loose or fallen out or anything; there was just no cable for that box. So we went to the white team and they remedied the problem, but we all had a good laugh over that. In the afternoon there was also another intrusion that I helped get logs for, but it wasn’t nearly as exciting as the morning breach.

The Nagios Box

Amidst everything else I was doing, I took a good shot in the afternoon to configure our Nagios box. I remembered from the last competition that the IPs were wrong, so that seemed to be what I’d probably have to do again, just fix up the network portion to that which we were assigned for all the entries. Simple enough with sed. However, I found enormous difficulties getting into the web console, considering they didn’t give us the username and password, and they weren’t any kind of defaults. Well, in the end, it turns out they were. “nagiosadmin” is apparently a standard username, and the password was the standard one for the competition. It just took way too long to figure that out. Once I fixed the IPs and logged in, I realized that all most of the checks were failing. Not good. That generally meant that the scorebot also would be counting those tests as failures. I talked to our firewall guy, who had egress filtering on (blocking outgoing traffic), which he suggested would give such results. We argued, he got busy, and I never got to see the beautiful green colors of Nagios that come with a fully working network. Oh well. At around 4, the competition ended, we packed up our computers, and we headed over to another room for a debrief.

The End

All tired, hungry, and anxious to hear the results, we waited and were fed pizza while the event organizers talked to us for a while. Then they let the red team have a go and both tell us what they did to us over the day, as well as query us about our strategies and give us some tips for defending. I actually got to talk to the guy who put that perl script on the Linux boxes, and he asked “did you find the others?” We laughed, and then realized that we hadn’t even looked for others. It didn’t even occur to us for some reason. So he pointed out that if you find something malicious, there’s almost certainly something else there, and you should make some effort to find it. He suggested grepping all the files in /sbin for “perl”, while I probably would have used “find” to find any files modified in the last few hours. Either way, it’s something solid that I learned and will most certainly apply at the next competition. Which leads to the most awesome part: GWU got 2nd place, and we’ll be competing at the regionals in Baltimore in March! We’ve got a lot of work to do, myself included.

All in all, I found the whole thing very worthwhile for the third time, and recommend any college students in the US with an interest in computer security to look at creating a team and competing in a regional competition. The whole affair, while stressful, is not only fun but a great experience for anyone interested in information assurance. As a matter of fact, I’m not particularly enthralled with security and I found it a great experience too. In past events (but not this one, because of the inauguration), we had Secret Service agents there as well to talk to us a bit and have to consult regarding some of the legal issues with intrusions, to discuss our incident reports with, and have drinks with afterward. I can’t wait until March. Maybe we’ll make nationals!

External Links

• White Wolf Security
• Mid-Atlantic CCDC Information
• National CCDC Website
• Nagios Website

miscellaneous linux, security