I just got done triaging the results of an AppScan website scan all the way through. I’ve done it before but never on a production run. It has always been partial triages before on training runs. After seven years of secure code review, I have triaged a lot of static analysis results from a variety of tools before and I’ve gotten quite good at it. I know once you get to know the tool and know how it spits out it results, which rules are almost always reliable, which are sometimes reliable and sometimes wrong, which are often wrong, you can work pretty quickly. Work some hard issues, switch to some easy ones for a break, back to hard. Adjust that based on the schedule if you need to focus on the most important stuff and do not have time to do it all. For source code scanning, I have gotten pretty good at it.
This was quite a bit different. The results of a dynamic, black box scan of a website are quite a bit different than a static analysis scan of source code where everything is laid out before you. It’s automated pen testing followed by verification still against that black box. As I’ve said before my pen testing skills are rusty from disuse and it has been a few years since I have evaluated a web application outside of training sessions. Like the pen testing, it’s a bit rusty. No well established rhythm in my triaging muscle memory.
We have a technical oversight/sanity check person working with us which is really nice in general to provide fresh eyes on things like this but it is really nice when you are in a new job in a new company. You are good at the way you do things but not necessarily the way the new company does things so having somebody around that is good at the company way is nice. Since this first time through with the new guys I made sure to ask questions and made sure I wasn’t wandering off down the rabbit hole. I was shocked at how quickly these experienced folks said they could triage things on these web scans. As I looked at the results before me I thought there was no way I could get close to that speed.
And I didn’t.
But between the triaging cheat sheets with experienced here where the rules are good, here’s where they are prone to false positive guidance and diving in I started to find that rhythm. Figure out the tool’s display, what to look at to verify the finding or dismiss it. Find the pieces to replicate the results manually. Get to know the HTML in front of you and set the search feature right you can progress through false positive weeding out fast in some cases like when a custom error page is mistaken for a result. I didn’t come close to the speed the old pros can do but I can see how they do it as I find that rhythm and scrape the rust off the lesser used skills.
I love the Internet.
I have been putting together a pen testing virtual machine lab to work through Georgia Weidman’s Penetration Testing: A Hands-On Introduction to Hacking book that just came out. It’s been years since I’ve done much of this and there are all kinds of new tools to use doing it so I want to scrape the rust off my skills. Part of the lab setup is a Kali Linux virtual machine on VirtualBox. I am used to setting up fairly lean Linux VMs to play with so I started off with just a 15 GB virtual disk and installed Kali to that disk.
Georgia has you set up a number of additional tools into the Kali Linux system and as I was installing the Android SDK and components for the mobile pen testing I ran out of room on that virtual drive. Ugh!
I didn’t want to create a second drive and I didn’t want to start over so I hit Google. Turns out VirtualBox includes a way to expand the size of the virtual disks and Jonathan Mifsud has an excellent how to on his blog. He is working on a Linux host and I am on a Windows host but there wasn’t much difference. Now I had a larger virtual disk.
Unfortunately that new free space was not right next to the main partition so you cannot just expand it into the unused space. I had to move the swap partition to the end of the drive and open up the free space right next to the partition I wanted to expand. Luckily there’s a good blog post by Eugene at trivialproof.blogspot.com. Since I’d already increased the size of my virtual disk I skipped down to step 4. Booting with the Kali Linux live ISO image I used GParted just as described and restructured my virtual disk.
Now I’ve got plenty of space and can keep on setting up the system to follow along with the book.
I’m sure there are other ways to have skinned this cat but those were the ones I found fast. Hard not to solve problems with the Internet these days if you at least know something about what you want to do.
Ah the joys of waiting for tools to do their job. Set the scan up either of the source code of an application or a dynamic scan of a website, click go and wait and wait and wait and…
If you’re lucky, the progress indicators s l o w l y creeps along. And you wait and wait and wait and…
Of course you can go off and do other stuff while the computer chugs like attend a North Alabama ISSA lunch time meeting or write a blog post but you still end up coming back, looking at the progress, hoping it has moved since the last time you looked and you wait and wait and wait and…
As tedious as that is, it’s far better than the alternative. It is far more tedious to look at code line by line by line for thousands or hundreds of thousands lines of code. Far more tedious to try to hand jam parameter manipulation and send it all to a website over and over again. It’s far less tedious to periodically check that progress bar, fingers crossed, to see if it has advanced. As much as you might be eager to get to triaging the results, letting the tool compile those results for you to look at is far less tedious than doing it all yourself. The computer doesn’t get tired or bored doesn’t need coffee and it’s pretty good at grinding its way through finding things that would take us weeks or months to do. It doesn’t care that the work day is over and can happily chug along overnight. (If you’re lucky and it doesn’t hang!) It can tirelessly keep track of a data flow from source to sink across call after call after call across complicated call stacks. Try doing that manually for each and every input and not grow old while doing it!
And when the tool is done, you get to spend your time looking at interesting things and diving deep on something rather than spending your time and your customer’s money on tediously finding everything the hard way. We get to have fun, the computer gets to do the drudge work.
As fun as waiting can be, it beats the alternative.
Sigh, still scanning but at least the bar is moving. Back to waiting and waiting and…
You often hear that training is a key piece of secure software development and it is. On the other hand I like to flip things around a bit and point out that another key is overcoming our training.
It may seem counter intuitive but we have to overcome our training. Not our secure development training but a lot of our other development training.
Have you ever heard the old joke that half the doctors out there graduated in the bottom half of their class? We have the same problem in the development world but with an added burden. Not all developers studied computer science or any other related degree in college. They come to the profession from other fields where they learned to program for some reason and decided they liked it. But like doctors in that joke, half of us are below average when it comes to our training.
A lot of our training created bad habits. I like to quote a computer science professor of mine from back in the mid 80s. The lesson was not hard so some of us set off to add bells and whistles to the assignment. The instructor told us “forget that extra stuff and just focus on the lesson.” Translation, get it to work and move on. Lesson learned and it wasn’t good. There is a whole lot of get it to work and move on in the software world. When I gave a talk that touched on this at an ISSA conference a few years ago some members of the audience who were current computer science majors came up afterwards and said their instructors are still telling them to focus on the lesson and forget about the extra stuff. Bad habits still being taught. Continue reading “Overcoming Our Training”
The nature of my secure code review work over the last several years meant we rarely had a chance to do any pen testing to verify the issues we found could be reached by an attacker. It was just the nature of the beast. Because of that my pen testing skills have become very rusty.
I have been using WebGoat on Samurai WTF 2.0 to refresh my memory on hacking web applications. It has been a lot of fun but getting started was a might bit frustrating. Finding WebGoat on Samurai WTF was not straight forward. I knew WebGoat was part of the distribution and there is even a link to it on the start menu. The problem with that is it took me off to www.webgoat.com and one of those “this domain may be for sale pages.” Not exactly what I was looking for.
Googling for starting webgoat on Samurai WTF wasn’t a whole lot more help. Most references said to point my browser to http://localhost:8080/webgoat/attack which unfortunately did not work. There wasn’t a site there.
I ended up having to poke around on the file system to find the configuration for WebGoat. In /etc/apache2/sites-enabled/webgoat (and /etc/apache2/sites-available/webgoat) I found WebGoat was running on a virtual server. Instead of 127.0.0.1, it was set up to run on 127.42.84.3.
I pointed my browser to http://127.42.84.3:8080/webgoat/attack and was in business. Frustration over at least as far as getting started was concerned. I still had to dig how to do all of the exercises out of the cobwebbed over corners of my brain.
One of my favorite memories of living through Microsoft’s adoption of secure development was sitting in a hacking demo by Foundstone for a lot of the Windows development team. The network security team were invited over to watch as well. The Foundstone guys demonstrated hacking an unpatched Windows 2000 system. As they got in to one part of the system I watched a developer hang his head and say “I wrote that code.” I think that moment probably did more for software security with that developer than any other single thing. From that moment on he knew security features did not make code secure. He was won over.
During my SAIC days, all our customers were developing software for DoD. One of the fun parts of being a security guy in a DoD world is the reaction of people when you arrive. There are so many IA guys (DoDspeak for security folks, IA stands for Information Assurance) who think their job is to always say no that security guys who arrive to help are viewed with suspicion until they show they are different. Showing that could be a lot of fun.
Early on one of the teams we were called in to help were not too happy to see us. In the initial meeting, the dev lead sat in the back corner, leaning back with his arms crossed. He clearly didn’t want us to be there. The team happened to be wrestling with a memory leak they could not find and we were able to find it via the static analysis tool we were using at the time. Solving that problem for them made them more receptive but not welcoming yet. As we passed on the results, and there were a lot since there was a lot of legacy C/C++ code that had been around for a long time, things looked grim until we set their priorities and waved them off worrying about fixing everything right away. Since their schedule permitted it, we sent them off on some easy to fix low hanging fruit that allowed a very large, quick win. We then sung their praises to their boss. From then on out we were greeted pretty well as we looked at all their changes periodically every year. Continue reading “Winning Over Developers”