SOCIAL SECURITY ADMINISTRATION
RELIABILITY OF THE DATA USED TO
MEASURE THE NUMBER OF
SUPPLEMENTAL SECURITY INCOME
We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.
The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:
Conduct and supervise independent and objective audits and investigations
relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.
To ensure objectivity, the IG Act empowers the IG with:
Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.
By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.
Date: August 5, 2002
To: The Commissioner
From: Inspector General
Subject: Performance Measure Review: Reliability of the Data Used to Measure the Number of Supplemental Security Income Non-Disability Redeterminations Completed (A-02-99-11003)
The Government Performance and Results Act (GPRA) of 1993, requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also requires disclosure of the processes used to verify and validate the measured values used to report on program performance. SSA is committed to ensuring the verification and validation of performance measures, and the Office of the Inspector General (OIG) audits of the performance measures are a means of achieving this. The objectives of this audit were to assess the reasonableness of the methodology SSA used to measure the following performance measure, determine the reliability of the data SSA used in its measurement process, and evaluate the appropriateness of the performance measure with respect to GPRA compliance and SSA's Annual Performance Plan.
Indicator FY 2000 Goal FY 2000 Performance
Number of SSI Non-disability 2,025,000 2,182,027
SSA oversees two long-term disability programs. The Disability Insurance program was established in 1954 under title II of the Social Security Act (Act). The program is designed to provide benefits to wage earners and their families in the event the wage earner becomes disabled. The Supplemental Security Income (SSI) program was created as a result of the Social Security Amendments of 1972. SSI (title XVI of the Act) provides a program of income to financially needy individuals who are aged, blind and/or disabled. Redeterminations are post-eligibility reviews of the non-medical factors, such as income, resources and living arrangements, that determine whether a recipient is still eligible for SSI payments and receiving the correct payment amount. There are two types of redeterminations: scheduled and unscheduled. Scheduled redeterminations are selected depending on an assigned error profile or likelihood of changes in circumstance that may affect either an individual/couple's eligibility to receive SSI payments or the payment amount. These redeterminations are ranked as high-, medium- and low-level profile error types according to the probability of error. Unscheduled redeterminations are completed on an "as needed" basis and are triggered when recipients report or SSA learns of certain changes in circumstances that could affect the continuing SSI payment amount.
Once selected, redetermination cases are tracked by the Central Office Redetermination Control System (CORC). Based on the error profiles, the redeterminations are assigned either to a field office (FO) or the Wilkes-Barre Data Operations Center (WBDOC).
The methods and priority for performing redeterminations and recording and updating case data vary depending on the error profile assigned to a particular case. Claims representatives at FOs typically handle high-error profile redeterminations through face-to-face or telephone interviews. Claimants are sent letters informing them of any special documents they may be required to bring to the interview, such as rent receipts or bank records. The updated information is input via on-line entry to the Modernized Supplemental Security Income Claims System (MSSICS).
WBDOC conducts redeterminations that have low-error profiles using computer-generated mail-out forms to be completed and returned by the claimants. The low-error profile is less detailed in the information requested than the high-error profile. Second requests are mailed out if recipients do not respond to the first mailing within 90 days. Forms are manually reviewed for completeness. If "no change" is indicated, a completion indicator is posted to the Supplemental Security Record (SSR). Data entry clerks using Customer Information Control System (CICS) screens key in claimants' responses, and the SSR is updated by overnight batch processing. If WBDOC does not receive the form within 180 days of the mailing, or if a complication develops in the case, the case is transferred to the servicing FO.
Redeterminations that have medium-error profiles are generally assigned to FOs, with a small portion assigned to the WBDOC. These cases are handled by either telephone or mail.
Computer programs check the redetermination fields on the SSR on a daily basis to determine whether any changes have occurred. If a case has been completed, a clearance is assigned to the case. CORC removes the case from its data base of pending cases and sends a file of all such completed cases to the Redetermination Information System (RIS).
RIS counts the records of receipts, clearances, and pending cases received from CORC. An output file of these workload counts is sent to the Integrated Workload Management System (IWMS) on a weekly basis. IWMS is a data base comprised of FO-level or higher summary workload counts. IWMS redetermination data are updated automatically once a week. Once a month, the Division of Cost Analysis extracts the redetermination data on clearances from the IWMS and posts it in the Annual Performance Plan Status Report (Status Report) and, at year-end, in SSA's Accountability Report. The Status Report provides a cumulative summary of the statistic during the fiscal year (FY).
RESULTS OF REVIEW
We could not reach a conclusion on the reliability of the number of completed redeterminations reported by SSA because of the Agency's inability to provide us with clear documentation of the processes and methods used to calculate the statistic. SSA is not in compliance with Office of Management and Budget (OMB) Circular 123, Management Accountability and Control, section II, which calls for the maintenance of clear and readily available documentation of an agency's transactions, management controls and other significant events. Moreover, the performance measure emphasizes only the quantity of redeterminations completed, not the outcome or results of the reviews.
THE RELIABILITY OF THE PERFORMANCE MEASURE FOR THE NUMBER OF REDETERMINATIONS COMPLETED CANNOT BE DETERMINED
We could not reach a conclusion on the reliability of the number of completed redeterminations because SSA does not maintain sufficient documentation of the methods and processes it uses to calculate the measurement. SSA was unable to provide either clear documented criteria or a comprehensive guide documenting the flow of performance measure data. Furthermore, SSA was unable to provide a consistent explanation of the process. As a result of these deficiencies, we could not replicate the number of completed redeterminations reported by SSA.
Furthermore, SSA staff advised us that the data files used to produce the performance measure were not available for all of FY 2000. We were told that these records were retained and available only on a current and prior month basis. Because of the timing of our request, September was the only month within FY 2000 that was available for our review. Our ability to determine the causes of certain data anomalies that we identified through testing was considerably limited because we could only test 1 month within FY 2000.
We were also hampered in our ability to test or verify the data posted in the
Status Report. The problem here was that the reported number of redeterminations
completed was only a summary total without an audit trail of supporting individual
case-level data. Consequently, we could not perform a Social Security number
case-level analysis of the Status Report data to resolve the anomalies that
we found in the data.
OMB Circular 123 states, " documentation for transactions, management controls, and other significant events must be clear and readily available for examination." Furthermore, GPRA requires agencies to " describe the means to be used to verify and validate measured values." Since SSA lacks this documentation, there is no audit trail that could be used to verify the performance measure.
DATA ANOMALIES WERE IDENTIFIED
As already indicated, SSA was unable to provide us a documented listing of the criteria it used to define a completed redetermination for the calculation of the performance measurement. Nevertheless, we attempted to test the validity of SSA's reported number of completed redeterminations through tests of two sets of data extracts from the SSR and CORC. We selected these data bases because both should have contained the total universe of completed redeterminations for a given month. Our tests were based on September 2000 data. We applied what we believed to be reasonable criteria to the extracts to determine the number of completed redeterminations.
In both tests, the total number of completed redeterminations that we found
was less than the number of completed redeterminations reported by SSA in its
September Status Report. Lacking any documented or definitive criteria with
which to replicate the performance measurement, we cannot state with certainty
the actual number of completed redeterminations that should have been reported
or SSA's degree of error, if any. The results of our two tests are shown in
Figure 1. Details of the specific steps performed in our tests of the SSR and
CORC data are shown in Figures A-1 and A-2 in the Scope and Methodology section
(Appendix A) of this report.
Comparison of the Number of Completed Redetermination Reported by SSA and the Number of Completed Redeterminations Calculated by OIG Tests of SSR and CORC Data Extracts
Number Reported in SSA's September 2000
Status Report Number Calculated by OIG Test of SSR Data
Difference Between SSA and OIG (SSR)
Number Calculated by OIG Test of CORC Data
Difference Between SSA and OIG (CORC)
We identified another questionable condition within the data SSA used to calculate the number of completed redeterminations statistic. There are significant differences between the two data files we tested, when both should contain similar data. Both the SSR and CORC extracts distinguish the various categories and types of completed redeterminations, face-to-face interviews as opposed to telephone interviews. While the number of different types of redeterminations should agree in the two data bases since they both should contain the universe of completed redeterminations for the month we tested, the totals for each type of completed redeterminations differed between the two files. This could possibly indicate a problem with the accuracy of the data SSA used to calculate the number of completed redeterminations it reported.
We found still other inconsistencies that we could not resolve. We found that the number of completed redeterminations produced by CORC was less than that reported by SSA in its monthly Status Report. As the key workload management system for the title XVI redetermination program, the CORC system should represent the most authoritative and final source of input data on workload counts of redetermination clearances. Nevertheless, the 161,299 completions for September 2000 contained in CORC did not match the 163,006 completions reported in the Status Report. We found it inconsistent and illogical that the reported number of redeterminations should be greater than the number the CORC system reported for the same period. In both tests of the SSR and CORC, we could not identify the causes of the differences between OIG's and SSA's data and therefore could not resolve the discrepancies. The lack of clear documentation of the process did not allow us to draw definitive conclusions on the cause(s) of these discrepancies.
PERFORMANCE MEASURE EMPHASIZES QUANTITY RATHER THAN OUTCOME OR RESULTS
GPRA requires Federal agencies to " establish performance indicators to be used in measuring or assessing the relevant outputs, service levels, and outcomes of each program activity." OMB Circular A-11 states, "The set of performance goals in an annual plan should be periodically modified as necessary to reflect changes in programs, agency capability to collect and report information, the importance and usefulness of any goal, and any other circumstances." Accordingly, SSA's performance indicator and goal for redeterminations do not currently measure the most relevant, important or useful information related to the redetermination process.
SSA has performed studies concerning the cost/benefit of performing redeterminations, but it does not provide any such outcome-based data as part of its Annual Performance Report on the results of the redetermination program. Outcome-based measurements would better convey whether the real objectives of the SSI program are being accomplished. For example, SSA does not currently report on the dollar amount of incorrect payments or the number of SSI recipients that were found ineligible to receive SSI and dropped from the rolls as a result of the redeterminations. Furthermore, SSA lumps together all of its completed redeterminations as a single number in its performance measurement. SSA does not distinguish between the number of high-, medium-, and low-level error profile type redeterminations completed and the amount of incorrect payments identified in each category.
The Office of Quality Assurance and Performance Assessment (OQA) collects such data on the outcomes-and dollar amount benefits-of the redeterminations processed as part of its annual Change Rate Study. OQA determines whether any over- or underpayment was discovered as a result of the redetermination and, if so, the dollar amount involved. According to OQA's Change Rate Study for FY 2000, 41 percent of the 1,494,400 redeterminations performed by FOs resulted in a finding of overpayments totaling $1.6 billion. The OQA also found that 28 percent of the redeterminations resulted in a finding of underpayments totaling $883.6 million.
The use of outcome-based performance measures provides a means of assessing the significance of completed redeterminations and places more emphasis on the quality of the reviews rather than just the quantity performed.
CONCLUSIONS AND RECOMMENDATIONS
We could not reach a conclusion on the reliability of the performance measure for the number of completed redeterminations reported by SSA because of the lack of sufficient documentation of the process and data used to count and calculate the redeterminations performance measure. Moreover, the performance measure emphasizes only the quantity of redeterminations completed, not the outcome or results of the reviews. Accordingly, we recommend SSA take the following corrective measures:
1. Formally document the processes used to collect, analyze, and report the redetermination workload performance measure;
2. Maintain an audit trail of the data from which the redetermination workload data are produced;
3. Review and verify the accuracy of the number of completed redeterminations reported in the Annual Performance Report; and
4. Provide a more outcome-related redetermination performance measure.
SSA agreed with our recommendations. Specifically, SSA has developed the Post-Eligibility Operational Data Store (PEODS) system, which is a modernized system that will address many of our concerns. The PEODS system will ultimately replace the CORC system in October 2002. Also, SSA is considering adding a more outcome-related redetermination performance measure using payment data in FY 2003. (See Appendix C for SSA comments.)
We are pleased that SSA agrees with our recommendations. The development and implementation of the PEODS system to replace CORC appears to strengthen the weaknesses identified during our review. We also believe SSA's consideration to add a more outcome-related redetermination performance measure using payment data will provide a more meaningful indicator on the quality of the redeterminations performed. However, the FY 2003 Annual Performance Plan does not contain such an outcome-based performance measure. We continue to recommend that an outcome-based performance measure be used to measure the impact of the redetermination process.
James G. Huse, Jr.
APPENDIX A - Scope and Methodology
APPENDIX B - Acronyms
APPENDIX C - Agency Comments
APPENDIX D - OIG Contacts and Staff Acknowledgments
Scope and Methodology
The objectives of this audit were to assess the reasonableness of the Social Security Administration's (SSA) methodology used to measure the number of non-disability redeterminations reported, determine the reliability of the data SSA used in its measurement process, and evaluate the appropriateness of the performance measure with respect to Government Performance and Results Act (GPRA) compliance and SSA's Annual Performance Plan.
To meet our objectives, we discussed how a redetermination is defined by the Office of Policy, the SSA component responsible for setting forth the program requirements for Supplemental Security Income (SSI). We also met with systems personnel in the Offices of Information Management (OIM), Systems Design and Development (OSDD), and Systems Requirements to obtain an understanding of the data flow and process that produces the redetermination workload counts. We obtained and reviewed the systems descriptions of the key data bases involved in the production of the workload measure: the Central Office Redetermination Control System (CORC), Redetermination Information System, and Integrated Workload Measurement System (IWMS).
To identify the key internal controls in the data collection and reporting process, we flow-charted the process that collects, analyzes, and reports the redetermination workload measure. This process includes from the point of data entry of the completed redetermination and its posting to the Supplemental Security Record (SSR) to the final reporting of the workload measurement in the monthly Status Report. At year-end, the cumulative figure of redeterminations completed is posted in SSA's Annual Accountability and Performance Report. We also conducted a walk-through at an SSA District Office to observe redeterminations with SSI claimants and the quality review procedures once the cases are completed.
We reviewed the program language used by SSA to compile the redetermination counts that are passed on to the IWMS, the source for the performance measure calculation. Our ability to test the fiscal year (FY) data was considerably limited. We were advised by SSA personnel that the data files used to produce the performance measure were not available for all of FY 2000. Instead, the records were retained only on a current and prior month basis. As a result, when we requested the data files, we were only able to test for 1 month during the audit period.
To test the accuracy and reliability of the performance measure, we obtained
- a data extract of redetermination transactions from the SSR for the September 2000 period and
- the data files of redetermination transactions generated by CORC for the September 2000 period.
For the first test, we obtained the SSR data extract and excluded all cases that did not appear, in our judgment, to constitute a completed redetermination. We used the SSR because all redetermination activity is posted to this Record. To identify these cases, we applied criteria taken from explanations of the redetermination fields contained in the Program Operations Manual System (POMS), SM 01501.002, with particular emphasis on the disposition indicator field, to ensure that only completed redeterminations were counted. We used the POMS for guidance because of the lack of written documentation detailing the criteria SSA uses to conclude when a redetermination is considered completed. The criteria we applied, the sub-sets of cases we excluded, and the final total of completed redeterminations we derived are shown in Figure A-1.
OIG Test of SSR Data Extract
Category of Case Record in SSR Data Extract Number of Records Deleted Total
Total records provided in SSR extract 257,815
Limited issue cases
Pending cases 21,203 190,934
Pre-effectuation review cases 34,922 156,012
Duplicate cases 2,737 153,275
Total records 104,540 153,275
We provided SSA personnel with a copy of the SSR extract to validate. In an
attempt to replicate the number of redeterminations reported as completed, SSA
personnel utilized different criteria from the Office of the Inspector General
(OIG). For instance, SSA personnel considered some limited issue cases as completed
redeterminations at the
option of the servicing field office (FO). Also, "couple cases" were counted twice. SSA personnel calculated 162,777 completed redeterminations from the SSR extract file compared to 163,006 reported by SSA. We were advised the 162,777 redeterminations contained 4,800 redeterminations that were no longer considered limited issue cases. Also, there were 4,280 "couple cases" based on a 2.7-percent estimate from the November 2001 composition of CORC. SSA staff informed us that there was no documentation to support this approach.
For the second test, we obtained the CORC file consisting of 946,564 records representing all redetermination transactions for September 2000, including all receipts, clearances, pending cases, etc. We selected the CORC file because it was the central data source from which the final workload count was ultimately produced. The file contained status codes assigned by CORC, which clearly identified 161,299 completed redeterminations. Included in this total, however, were transactions worked on, but not completed, by the Wilkes-Barre Data Operations Center (WBDOC) and then transferred to an FO for completion. SSA personnel advised us that these WBDOC transfer cases were eliminated from the total number of completed redeterminations reported by SSA in the Status Report. We eliminated these transfers from the CORC extract. The results of our test are shown below in Figure A-2.
OIG Test of CORC Data
Type of Case Record in CORC Data Total Records
Total records redeterminations identified as clearances 161,299
Total records of clearances containing invalid dates ("0000" in the completion date field) and entries of "P" (pending) in the "Redetermination Type" field (7,701)
Duplicate cases (6)
Total Records 153,592
Our field work was conducted at the OIG New York FO and SSA Headquarters in
Baltimore, Maryland, from July 1999 through December 6, 2001. The entities audited
were OIM's Title XVI Systems Branch and OSDD. Our audit was performed in accordance
with generally accepted government auditing standards, as applicable to a GPRA
Act Social Security Act
CICS Customer Information Control System
CORC Central Office Redetermination Control System
FO Field Office
FY Fiscal Year
GPRA Government Performance and Results Act
IWMS Integrated Workload Management System
MSSICS Modernized Supplemental Security Income Claims System
OIG Office of the Inspector General
OIM Office of Information Management
OMB Office of Management and Budget
OQA Office of Quality Assurance and Performance Assessment
OSDD Office of Systems Design and Development
POMS Program Operations Manual System
RIS Redetermination Information System
SSA Social Security Administration
SSI Supplemental Security Income
SSR Supplemental Security Record
Status Report Annual Performance Plan Status Report
WBDOC Wilkes-Barre Data Operations Center
Date: May 21, 2002
To: James G. Huse, Jr.
From: Larry Dye
Chief of Staff
Subject: Office of the Inspector General (OIG) Draft Report, "Performance Measure Review: Reliability of the Data Used to Measure the Number of Supplemental Security Income Non-Disability Redeterminations Completed"(A-02-99-11003)-INFORMATION
Our comments on the report are attached. Staff questions can be referred to Odessa J. Woods at extension 50378.
COMMENTS OF THE SOCIAL SECURITY ADMINISTRATION (SSA) ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT, "PERFORMANCE MEASURE REVIEW: RELIABILITY OF THE DATA USED TO MEASURE THE NUMBER OF SUPPLEMENTAL SECURITY INCOME NON-DISABILITY REDETERMINATIONS COMPLETED" (A-02-99-11003)
Formally document the processes used to collect, analyze, and report the redetermination workload performance measure.
We agree. The Post-Eligibility Operational Data Store (PEODS) system for redeterminations is a new modernized system that will address many of OIG's concerns. PEODS is currently being provided to all field offices to control their redetermination workload. Installation of this system should be accomplished by the end of May 2002. In developing PEODS, users worked to identify, document and validate requirements for redetermination data, all calculations, report content and format. The existing Central Office Redetermination Control (CORC) system, the Agency's current performance measurement system, will not be terminated until all components agree that the performance measures information produced by PEODS is acceptable. Therefore, PEODS is not expected to replace CORC as the Agency's source for redetermination performance measures until October 1, 2002.
PEODS data will be retained (at least offline) so that validation by higher monitoring authorities will be possible. Users are working to evaluate how much data should be immediately retrievable. At this point, it appears SSA will retain 13 weeks of data online and 2-years, current and prior, detailed data offline.
Maintain an audit trail of the data from which the redetermination workload data is produced.
We agree. See response to recommendation 1.
Review and verify the accuracy of the number of completed redeterminations reported in the Annual Performance Report.
We agree that confirmation of the accuracy of Fiscal Year (FY) 2000 data is not easily obtainable. As noted in our comments to Recommendation 1, the future redetermination control system--PEODS--will provide the necessary systems structure so that we can affirm the outcome of the data for future Annual Performance Reports.
Provide a more outcome-related redetermination performance measure.
We agree. The Agency currently has available from its annual Change Rate Study, payment change data on a national level. The change data are classified by type of redetermination, percent of redeterminations resulting in a suspension or termination of SSI payments or the discovery of a retroactive and/or recurring overpayment/underpayment. The average amount of these payment changes is also available. SSA will consider adding a more outcome-related redetermination performance measure using payment data in FY 2003.
Page 2 Paragraph 2 -- The report does not accurately define the acronym for CORC. CORC stands for Central Office Redetermination Control system.
OIG Contacts and Staff Acknowledgments
Frederick C. Nordhoff, Director Financial Management and Performance Monitoring Audit Division, (410) 966-6676
Timothy Nee, Deputy Director, (212) 264-5295
In addition to those named above:
Robert Blake, Senior Auditor
Jeffrey Shapiro, Senior Analyst
Charles Zaepfel, Computer Specialist
Annette DeRito, Writer-Editor
For additional copies of this report, please visit our web site at www.ssa.gov/oig or contact the Office of the Inspector General's Public Affairs Specialist at (410) 966-1375. Refer to Common Identification Number A-02-99-11003.
Overview of the Office of the Inspector General
Office of Audit
The Office of Audit (OA) conducts comprehensive financial and performance audits of the Social Security Administration's (SSA) programs and makes recommendations to ensure that program objectives are achieved effectively and efficiently. Financial audits, required by the Chief Financial Officers Act of 1990, assess whether SSA's financial statements fairly present the Agency's financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA's programs. OA also conducts short-term management and program evaluations focused on issues of concern to SSA, Congress, and the general public. Evaluations often focus on identifying and recommending ways to prevent and minimize program fraud and inefficiency.
Office of Executive Operations
The Office of Executive Operations (OEO) supports the Office of the Inspector
General (OIG) by providing information resource management; systems security;
and the coordination of budget, procurement, telecommunications, facilities
and equipment, and human resources. In addition, this office is the focal point
for the OIG's strategic planning function and the development and implementation
of performance measures required by the Government Performance and Results Act.
OEO is also responsible for performing internal reviews to ensure that OIG offices
nationwide hold themselves to the same rigorous standards that we expect from
the Agency, as well as conducting employee investigations within OIG. Finally,
OEO administers OIG's public affairs, media, and interagency activities and
also communicates OIG's planned and current activities and their results to
the Commissioner and Congress.
Office of Investigations
The Office of Investigations (OI) conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement of SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, physicians, interpreters, representative payees, third parties, and by SSA employees in the performance of their duties. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.
Counsel to the Inspector General
The Counsel to the Inspector General provides legal advice and counsel to the Inspector General on various matters, including: 1) statutes, regulations, legislation, and policy directives governing the administration of SSA's programs; 2) investigative procedures and techniques; and 3) legal implications and conclusions to be drawn from audit and investigative material produced by the OIG. The Counsel's office also administers the civil monetary penalty program.