Some of the 2,200 databases that the IRS uses to manage and process
taxpayer data are not configured securely, are running out-of-date
software, and no longer receive security patches. Nor has the IRS fully
implemented its plans to complete vulnerability scans of its databases
-- although the IRS spent more than $1.1 million in software licenses
and support costs for a database vulnerability scanning and compliance
assessment tool, it did not fully implement it.
TIGTA used database vulnerability assessment software to conduct
remote scans of the primary databases for 13 applications supporting
critical tax administration business processes. Its review found high
and medium risk vulnerabilities, as classified by the scanning tool in
each of the 13 databases.
It's easy to stand on the sidelines and complain about what the IRS folks could/should have been doing all along. That's not at all productive. But I highly recommend reading the TIGTA's report front to back because it paints such a great picture at just how difficult it is to put basic security concepts in to practice and keep them going.
Those slow painful procurement protocols are actually useful.
Based on a previous audit, the IRS agreed to purchase a database scanning solution and conduct scans of its databases. That alone was difficult. A vendor solution was selected ad paid for, only later to find out that the vendor didn't support scanning the mainframe databases. Furthermore, of the systems that were supported, the IRS had difficulty deploying the solution across all their servers. As a result, only a subset of the 220 databases were actually scanned. The report notes that in 2010 the IRS switched to IBM's Guardium
, which had fuller coverage, especially for the mainframe environments. Lesson learned, it's worth it to do formal requirements gathering and formal product evaluations when selecting a vendor. At very least you'll go into it knowing what can't be done. Oh and don't forget to keep the records of the evaluation in case the auditors ask for it later.
Identity Management and Privilege Management Are Vulnerable By Default
The report calls out details of 13 applications and their corresponding databases that were scannable by the previous tool and the TIGTA noted that there were both basic account management problems and privilege management problems. In te account management arena, the TIGTA report noted the continued existence of product default accounts and passwords as well as "inappropriate password settings." In the privilege management arena, the issues are a little fuzzier, but the report noted that powerful administrative privileges were granted to some accounts that should be granted according to job function. Lesson learned: Part of planning for an deploying an application MUST inlude deploying operational processes to manage both end user and administrative access to the environment. Because these costs are not incurred by developers, they are easily overlooked by management responsible for overall deployment.
Software Accumulates Risk Over Time
The TIGTA reported noted that not only were the database servers not patched for major security issues, many of them were running on software versions that were so out of date that the vendor was not even issuing security patches for them anymore. The TIGTA report did it's homework and specifically identified security vulnerabilities in each of the database vendors products that were present in the existing levels of running software and for which no patch was available or planned on being available. The difficult part here is that no upgrade is done in isolation. If they could be done in isolation, people would do them. But the risk is that you are never quite sure if an upgrade to an infrastructure component is going to be transparent to the applications that depend on it. The upgrade is always risky and mitigating that risk by testing the upgrades in a sandbox environment with the applications and real (hopefully obfuscated) data is expensive. Lesson learned: The longer you put off the pain of upgrades, the higher the risk and the more intractable the problem becomes.
Patch Management is an Ongoing Drill
Patch deployment is a never ending daily grind. The TIGTA report calls out the patch history for the databases of 13 of the key applications. And it's an interesting ready. Just glancing over the tables, it appears that for a while the patches were getting deployed within 1-3 months of the patch being available. OK, maybe that's too big a window of opportunity, but at least there was evidence that the a program was in place. But the most recent update noted happened in January 2010 and some had not had security patches applied since 2007! Lesson learned: Patch management requires constant diligence. The TIGTA report (see Table 3) indicates to me that that patch management processes were probably in place at first for the apps and then somewhere along the way they fell off the priority list and were superceded by other pressing issues. And eventually they just weren't picked up. (Kinda reminds me of my flossing habits, but that's a story for a different day).
Again, my point is not to cast stones at the IRS. On the contrary, I applaud them for their transparency. The TIGTA report should be required reading for everyone in IT Security, not because there's anything new in it, but because it reminds us just how difficult it is to maintain the basics of IT security operations. It's no wonder that there is a huge focus on automating these types of activities with tools like Tivoli Endpoint Manager
and standards like SCAP.