Joint Oversight Hearing on Managing Costs and Mitigating Delays in the Building of Social Security's New National Computer Center
Good morning, Mr. Chairman, Mr. Chairman, Mr. Becerra, Mrs. Holmes Norton, and members of both Subcommittees. I would like to welcome the new members of the 112thCongress, along with new members of both Subcommittees. It is a pleasure to appear before you, and I thank you for the invitation to testify today. I have appeared before the Subcommittee on Social Security many times to discuss issues critical to the Social Security Administration (SSA) and the services the Agency provides to American citizens. Today, we are discussing SSA’s progress in constructing a replacement facility for its National Computer Center (NCC), SSA’s aging national computer processing and data storage facility.
I last spoke to both Subcommittees about the replacement of the NCC at a hearing in December 2009. At the time, there was much debate among the Subcommittees, SSA, and the General Services Administration (GSA) regarding the location of a new data center—on the grounds of SSA Headquarters in Woodlawn, Maryland, or at a location away from the Agency campus. After more than a year of consideration and discussion between GSA and SSA, the decision was made to locate the Agency’s new National Support Center (NSC) at the Urbana Research Center in Urbana, Maryland, just south of Frederick, Maryland. We understand that OMB has approved that site selection and that SSA finds it to be functionally acceptable.
While SSA’s Office of the Inspector General (OIG) is pleased with the site selection for the NSC, it is a matter of some concern that GSA and SSA to this point have taken longer than anticipated to follow their timeline for the project. During the December 2009 hearing, GSA said it anticipated selecting and acquiring a site for the NSC in March 2010, with a design-build contract awarded in March 2011. According to GSA, the project will not be completed by its original target completion date of October 2013.
The importance of the NCC to SSA’s operations cannot be understated. The NCC houses about 460 million records of Americans’ earnings, as well as current benefit data for about 57 million beneficiaries, supporting SSA programs provided to the public and other services provided to government agencies. Ensuring the continued operation of the NCC while properly planning for the transition to a new NSC is critical; were there an extended outage or another issue that caused the NCC to become unavailable, the Agency would be unable to process tens of thousands of retirement, survivor, and disability claims, as well as Social Security number verifications. This type of service interruption would severely affect the American public, delaying the delivery of benefits to citizens who depend on these funds in their day-to-day lives, and likely hindering people’s ability to obtain employment, driver’s licenses, and even loans and mortgages.
In recent years, we have called on SSA to develop a long-term overall information technology (IT) strategic plan, with a critical focus on the replacement of the NCC and the proper and timely transition to the NSC. In a September 2007 OIG report, The Social Security Administration’s Information Resources Management Strategic Plan, our office said, “Because SSA’s Information Resources Management (IRM) strategic planning does not go beyond two years, its IRM does not provide a clear strategic vision of what the Agency needs or plans to do over the next few years to address its critical challenges.”
The replacement of the NCC with the NSC is SSA’s most critical IT challenge over the next five years. The NCC, located at SSA Headquarters in Woodlawn, was constructed in 1979, and the building in which it is housed is nearing the end of its useful physical life. The chance of a potentially crippling outage at the NCC increases as time passes; an NCC feasibility study completed by Lockheed Martin in 2008 estimated the NCC would reach maximum capacity in three to five years. The effort to replace the NCC gained momentum when Congress passed and the President signed the American Recovery and Reinvestment Act of 2009 (Recovery Act), which provided $500 million for SSA to replace the NCC. The OIG was provided $2 million for the oversight of SSA’s use of Recovery Act funds. We have said and maintain that swift and efficient planning for the replacement of the NCC is necessary for the Agency to provide benefits without delay to those who need and are entitled to them.
While the planning for the construction of the NSC continues, there are risks involved in the ongoing reliance on the aging NCC to handle SSA’s daily operations. In its 2008 study, Lockheed Martin identified several structural and technical deficiencies with the NCC. Lockheed Martin identified, and SSA said it has taken action on, the following issues:
- Lockheed Martin identified the NCC electrical feeder cables as the only point of imminent failure at the data center, as degraded insulation could potentially result in loss of power, leaving SSA with only batteries as a power source to potentially accomplish a controlled shutdown. In February 2009, SSA spent $1.9 million to replace the deteriorating feeder cables between the Utility Building and the NCC.
- In 2008, the NCC operated with an outdated electrical panel breaker design that violated safety codes and could have caused overheating or fire damage and resulted in a data center outage. On Lockheed Martin’s recommendation, SSA spent $2.7 million to replace the electrical panels over three NCC shutdowns between 2009 and 2010. SSA also increased the number of circuits of per panel, allowing for additional IT equipment growth and technology changes.
- Lockheed Martin recommended SSA begin exploring options to address the overall risk of continuing to rely on a 30-year-old Uninterruptible Power Source (UPS), including extending its NCC maintenance contract through 2014. SSA said current terms of its maintenance contract will expire in September 2012, and the Agency plans to establish a new contract by that time that extends the maintenance service through 2018.
However, as of January 2011, SSA’s maintenance contractor stated its intent to provide NCC service through 2015, with a “Best Efforts” extension based on equipment available through 2018. Because of the vagueness of the contractor’s “Best Efforts,” we believe SSA should not depend on the same level of NCC maintenance from the contractor past 2015.
- Lockheed Martin said SSA had the ability to add electrical distribution panels at the NCC. SSA in January 2011 spent $2 million to install two new electrical risers for the NCC to provide capacity for additional cooling equipment on the data center floor, if needed.
- The feasibility study in 2008 recommended roof repair for the NCC and the Utility Building. SSA spent $1.2 million and replaced the Utility Building roof in 2007 and the NCC warehouse roof in 2009. The Agency said it has also increased routine maintenance inspections on both building roofs.
- SSA said it is also monitoring NCC plumbing conditions, performing foundation inspections, and monitoring HVAC ductwork, on recommendations from Lockheed Martin.
Lockheed Martin said it found NCC maintenance practices were managed and executed properly, but for the facility to be sustained through 2014, the same level of facility management and oversight needed to be continued. SSA said its staff performs regular tours of the NCC and facility equipment, with an “annual building walk-around” with technical experts to determine repairs or future replacement projects on the building, grounds, and equipment. The cost for preventative maintenance at the NCC is projected to be $17 million in Fiscal Year (FY) 2011, and anticipated costs in FY 2014 are $18 million with a 10 percent annual increase through FY 2020, according to SSA.
We acknowledge that SSA has taken necessary steps to address structural and technical issues at the NCC identified by Lockheed Martin. Nevertheless, the longer the construction and transition to the NSC is delayed, the risks to the systems at the aging NCC will increase, as many of the building’s systems reach and surpass their lifecycle age. Proper planning and efficient plan execution is critical for the Agency so that the NSC is delivered before significant problems arise at the NCC and affect SSA operations.
Should NCC outages occur at any time during the NSC building process, SSA will rely on its Second Support Center (SSC) as its backup data center and recovery site. As GSA and SSA continue the process in building the NSC, we are well aware of the challenges and delays that SSA faced in creating the SSC. The SSC was initiated in response to Agency vulnerabilities first identified in a 2002 Lockheed Martin assessment of SSA’s disaster recovery plan. The assessment concluded that no backup facility existed that could meet the Agency’s data processing needs in the event of a disaster that rendered the NCC unavailable. It was not until three years later, in 2005, that SSA worked with GSA to acquire a second data center.
SSA encountered a number of delays during the acquisition and construction of the SSC. We determined it took six years, starting in December 2002, for SSA to plan, construct, and occupy the co-processing center. The Agency spent the first 26 months analyzing disaster recovery solutions, then 14 months selecting a site, then 32 months obtaining permits and constructing the new data center. In May 2006, the SSC lease was awarded, with an anticipated completion date of August 2007. Delays in construction pushed the SSC occupancy date to January 2009.
SSA has said the SSC will not be “fully functional” until 2012, due to the time needed for efficiency testing and additional equipment and data connections. When we say “fully functional,” we mean that SSA will be able to meet its disaster recovery objectives by restoring critical functions within 24 hours of a disaster with less than one hour of data loss. Previously, SSA’s backup and recovery strategy relied on a vendor hot site, an alternate facility equipped with the technological capacity and personnel required to recover critical business functions or information systems, but SSA’s contract for its commercial hot site expired in September 2010. Rather than renew the contract, the Agency decided to purchase IT equipment in FY 2011 to enable the recovery of the disability systems at the SSC.
Given the importance of the Agency’s current efforts to build the NSC, we believe SSA should learn from its experience with the SSC and take the necessary steps to ensure proper planning to mitigate project delays and cost increases. In our September 2009 report, Processing Capacity of the Social Security Administration’s (Second) Support Center, we made several recommendations regarding the NSC planning process. Specifically, we recommended that SSA:
- Accelerate the use of the SSC as a fully functioning data center, with particular emphasis on using the SSC as the disaster recovery site for the NCC.
- Develop a comprehensive, long-range IT strategic plan that includes possible constraints and challenges on all aspects of IT projects.
- Formally document the Agency’s plan to accelerate the use of the SSC as part of SSA’s overall disaster recovery plan, and continually update the disaster recovery plan as the SSC and NCC replacement become fully functional.
At the time of the OIG report, SSA agreed with all of our recommendations. Recent disaster recovery tests at the SSC show that SSA has improved the time it takes to recover operations from 10 days in 2009 to a little less than five days in 2010. Over the next 12 months, SSA has indicated plans to reduce the five days down to about one day. Once SSA has demonstrated a process for recovering NCC workloads at the SSC, the Agency said it would update its disaster recovery documentation accordingly.
The NSC will be built on a 63.5-acre site located between a data center operated by another Federal agency and Interstate 270 in Urbana, about 40 miles from both Baltimore and Washington, D.C., allowing SSA to maximize data-sharing speed and to limit the commute for relocated NCC staff. Both Baltimore Gas & Electric Co. (BGE) and Allegheny Power have indicated they can provide electrical infrastructure to the site. An OIG contractor, Strategic e-Business Solutions (SeBS), visited the site with SSA and GSA officials in November 2010 and did not discover any major obstacles to data center construction. We deem the NSC site selection acceptable because the infrastructure is in place for a new data center, as it would be located next to an existing Federal data center and it would be easily accessible via highways.
However, according to GSA, the timeline for completing the build-out of the NSC will go beyond the original anticipated completion date of October 2013. We understand that the new schedule sets a September 2014 date for building completion, and a January 2015 date for the commissioning of the building. SSA estimates that data migration could take an additional 18 months. Because there are many risks involved if this project were to fall further behind schedule, it is critical that GSA and SSA identify and develop plans for foreseeable construction delays, including excavation challenges and weather-related delays; and negotiate contracts with suppliers and builders to ensure materials are delivered and the work is completed on time.
As previously stated, SSA has already allocated significant funds for repairs and maintenance at the aging NCC, where the possibility of an outage increases as time passes. Also, a number of delays occurred in selecting a site for the SSC and building the SSC, and data recovery testing at the facility is still ongoing; GSA and SSA have now taken longer than anticipated to select a site for the NSC. And while the SSC has the capability to function indefinitely as a backup facility in the case of an NCC outage, SSA should not rely on the SSC indefinitely if there are delays in the construction of the NSC, because the Agency does not have a backup facility should the SSC become unavailable.
Should the NCC’s UPS sustain an outage before the NSC is operational, SSA has said power can be fed directly to the data center from either its generators or BGE to maintain facility operations, though the IT equipment would then be susceptible to a shutdown because of any electrical surges or spikes. SSA said it is exploring other backup options, including:
- Limiting UPS usage by relocating office functions on non-data center floors, and/or limiting non-data center floors to using local power;
- Systematically shutting down data center operations to install new UPS systems for the data center only;
- Purchasing a new UPS, estimated at $17 million, to service the entire building;
- Renting a mobile UPS; and
- Contracting for space at a host data center or renewing its disaster recovery hot site contract.
We have done, and continue to do, significant and wide-ranging oversight of SSA’s planning of the transition to the NSC. Our most recent review, SSA’s New Data Center Site Alternatives, evaluated the appropriateness of the short list of potential sites selected by GSA and SSA for the new data center.
We also have additional reviews planned. One review will determine whether SSA followed best practices in developing its overall program of requirements for the new data center. Another review will evaluate the building plans for SSA’s new data center and determine whether the Agency followed building design best practices to provide a data center that meets SSA’s requirements. Going forward, we also plan to have our contractors evaluate how well GSA and SSA plan for contingencies and mitigate the risk of any further delays in the project.
_SSA’s Response to Congressional Inquiry Concerning New Data Center Site Selection,_released in August 2010, was an independent assessment of whether building the NSC on or off SSA’s Woodlawn campus was the most cost-effective and best solution for SSA’s requirements. OIG’s contractor, SeBS, concluded that a GSA contractor did not address all NSC construction costs or the costs of transition to the NSC, and it did not provide an analysis of alternatives available in the event of potential schedule delays of the NSC. SeBS recommended SSA and GSA move forward with an off-campus site selection and assess the impact of NSC construction delays.
In April 2010, we released two reviews, SSA’s Use of Site Selection Industry Best Practices for its New Data Center, and SSA’s Data Center Alternatives. The best practices review evaluated the appropriateness of the potential sites selected for the new data center and determined whether best practices were followed in the development of the project plan. The SeBS evaluation found that in general, SSA developed a highly sophisticated set of selection criteria to evaluate geographic areas of consideration and prospective properties.
The Social Security Subcommittee requested our review of SSA’s data center alternatives in December 2009. The SeBS evaluation indicated that in general, SSA had conducted a sophisticated evaluation of its data center requirements, timeframes, and options in planning to replace its NCC facility. However, at that time, SeBS said additional due diligence efforts might be warranted. In particular, there were questions about the costs and the risks that were assigned to the different options for building a new data center.
In conclusion, the sustainability and expansion of SSA’s IT systems are critical to the Agency’s ability to meet its goals and fulfill its mission; that mission affects nearly all Americans every day. The NSC project status shows it is imperative that the Agency have a clear IT vision that anticipates current and future needs. SSA’s current IT plans are short-term and do not provide a detailed description of how the Agency intends to address its IT processing needs 10 to 20 years into the future. The construction of the NSC in a timely fashion is SSA’s most critical IT investment over the next five years, to mitigate risks associated with relying on an aging NCC. With long-term planning and proactive management, we should be able to avoid a repeat of the current situation with the NCC.
My office is dedicated to working with SSA to ensure that the construction of and transition to the NSC follows best practices and is carried out based on sound planning and management. We look forward to continuing to assist in this vitally important undertaking. I thank you again for the invitation to speak with you today, and I would be happy to answer any questions.
Date: February 11, 2011
OIG Official: Patrick P. O'Carroll, Jr., Inspector General
Committee/Subcommittee: U.S. House of Representatives Committee on Ways and Means, Subcommittee on Social Security & Committee on Transportation and Infrastructure, Subcommittee on Economic Development, Public Buildings, and Emergency Management
Link: /assets/uploads/2011-02-11_joint-oversight-hearing-on-managing-costs-and-mitigating-delays-in-the-building-of-social-securitys-new-national-computer-center_patrick-p-ocarroll-jr.pdf