Release of 2002 unemployment benefit accuracy data

Official letter (https://www.dol.gov/sites/dolgov/files/ETA/advisories/UIPL/2003/UIPL19-03.html):

EMPLOYMENT AND TRAINING ADMINISTRATION ADVISORY SYSTEM
U.S. Department of Labor
Washington, D.C. 20210
CLASSIFICATION: UI
CORRESPONDENCE SYMBOL: OWS/DPM
DATE: March 4, 2003

UNEMPLOYMENT INSURANCE PROGRAM LETTER NO. 19-03
TO: ALL STATE WORKFORCE AGENCIES
FROM: CHERYL ATKINSON /s/, Administrator, Office of Workforce Security
SUBJECT: Procedures for Release of Unemployment Insurance (UI) Benefit Accuracy Measurement (BAM) Data for Calendar Year (CY) 2002
RESCISSIONS: None
EXPIRATION DATE: March 31, 2004

1. Purpose. To provide State Workforce Agencies with guidelines and key dates for the release of UI BAM paid and denied claims data for CY 2002 and to inform the states that the Department of Labor (DOL) will publish BAM program data in the CY 2002 UI PERFORMS Annual Report.

2. References. 20 CFR 602; Unemployment Insurance Program Letter (UIPL) No. 19-96; UIPL No. 18-00; Federal Register Notice (FRN) 67 FR 55879; and Benefit Accuracy Measurement State Operations Handbook, ET Handbook No. 395, 3rd ed.

3. Background. BAM data are published as part of the UI PERFORMS Annual Report. UI PERFORMS is DOL's management system for promoting continuous improvement in the performance of the UI system. UIPL No. 19-96 announced the elimination of the requirement in 20 CFR 602.21(g) that states publicly release BAM program data. DOL will publish a compendium of all states' CY 2002 BAM paid and denied claims data and a list of individuals designated by the states as points of contact for public inquiries about BAM data. States may release their BAM data to the public independent of the federal release. The CY 2002 UI PERFORMS Annual Report will include background information on the BAM program, a brief discussion of how the accuracy rate estimates are computed, national summary data, state paid claims Annual Reports, and state Denied Claims Accuracy (DCA) rates. During CY 2003 DOL will publish additional integrity measures for UI paid and denied claims from CY 2002 BAM program results.

4. Federal Release of UI PERFORMS Data. The CY 2002 UI PERFORMS Annual Report will include background information on the BAM program, a brief discussion of how the accuracy rate estimates are computed, national summary data, state paid claims Annual Reports, and state Denied Claims Accuracy (DCA) rates. During CY 2003 DOL will publish additional integrity measures for UI paid and denied claims from CY 2002 BAM program results.

The CY 2002 UI PERFORMS Annual Report and supplemental BAM analyses will be posted on the Employment and Training Administration (ETA) Office of Workforce Security (OWS) public Web site: http://workforcesecurity.doleta.gov. Effective with the publication of the CY 2002 UI PERFORMS Annual Report, DOL will announce the availability of BAM program data in a notice on the OWS public Web site rather than through a Federal Register Notice as in the past. If a state fails to follow the BAM methodology regarding sample selection, data collection, or case completion timeliness prescribed in ET Handbook No. 395 to such an extent that the data are not reliable, DOL will publish an explanation of the major deficiencies in lieu of BAM program data for that state. States must address these deficiencies through corrective action plans as part of the State Quality Service Plan (SQSP) process. UIPL No. 18-00 informed states that effective with the publication of the CY 2000 UI PERFORMS Annual Report, DOL no longer publishes BAM program narratives submitted by the state agencies. States may provide explanatory information related to any UI program data published in the UI PERFORMS Annual Report through the SQSP process.

5. Publication of DCA Data. Effective July 29, 2001 (BAM sampling batch 200131), states began to sample and investigate UI claims that were denied because of monetary, separation, and nonseparation issues. FRN 67 FR 55879 announced that because DCA data were available for less than half of CY 2001, they were not included in the CY 2001 UI PERFORMS Annual Report, but that DOL planned to publish CY 2002 DCA rate data based on a complete year of sampling and investigation. The following CY 2002 DCA data will be published for each state in the CY 2002 UI PERFORMS Annual Report and posted on the OWS public Web site:

• separate accuracy rates for monetary, separation, and nonseparation denial determinations;

• the number of sample cases on which the estimates are based;

• the population sizes from which the samples were selected;

• the sampling errors of the estimated accuracy rates (reported as 95 percent confidence intervals); and

• any applicable footnotes related to the state's performance in meeting the sampling, case completion, timeliness, and case investigation methodological requirements.

Because the DCA data have not yet been reviewed by DOL, the data will be noted as preliminary data that are subject to change.

6. BAM Date of Record. Chapter VI of ET Handbook No. 395 requires that "a minimum of 98 percent of cases for the year must be completed within 120 days of the ending date of the Calendar Year." Close of business (c.o.b.) April 30, the 120th day of CY 2003, is the date of record for state databases for computation of the CY 2002 BAM paid claims Annual Report and DCA rates. All BAM paid and denied claims accuracy rates published in the UI PERFORMS Annual Report and any supplemental analyses will be based on CY 2002 BAM paid and denied claims cases that have been signed off by the BAM program supervisor by c.o.b. April 30, 2003. Changes to the state's database subsequent to the date of record can result in inconsistencies with DOL's database that must be reconciled before the BAM program data can be published. States must not reopen any CY 2002 BAM cases after April 30, 2003, until they have reviewed the BAM program data transmitted to them by DOL and have confirmed that these data agree with the BAM paid and denied claims accuracy data produced by the state's database. Procedures for resolving differences between DOL and state BAM program data are discussed in section 9.

7. BAM Annual Report Software. States may run the BAM paid claims Annual Report software as soon as they complete all CY 2002 BAM paid claims cases. The National Office will produce a paid claims Annual Report for each state, based on the CY 2002 BAM cases as of c.o.b. April 30, 2003. The BAM Annual Report software excludes cases that do not meet the definition of the BAM paid claims population (for example, supplemental payments or payments made for Temporary Extended Unemployment Compensation (TEUC) claims) and adjusts the amount of UI benefits paid to the population to exclude the amounts paid that are represented by these excluded cases. Prior to the date of record, DOL will transmit a footnote lookup table for the CY 2002 BAM paid claims Annual Report. The BAM DCA software on the state Sun computer systems does not currently include programs that allow states to produce DCA rates reports. Prior to the date of record, the National Office will distribute to all BAM supervisors and state Sun system administrators an Informix ACE report program that produces DCA rates. States are encouraged to use this program to produce DCA rates and compare them with the rates DOL transmits to them for comment prior to publication. States can verify DCA case completion and timeliness with the DCA Case Completion and Timelapse Report, which was installed on State Sun systems in BAM DCA software release 3.2 on January 31, 2003. BAM DCA rates exclude sample cases that do not meet the definition of the DCA population (for example, monetary redeterminations, nonmonetary determinations that are not reportable on the ETA 207 and 9052 reports, and denied TEUC claims). DCA rates also exclude two types of cases for which there is no denial issue to investigate: 1) cases in which monetary eligibility was established upon receipt of CWC, UCFE, and/or UCX wage credits within the two-week sampling lag (Action Code Flag "0"); and 2) claims withdrawn by the claimant (Action Code Flag "8"). The software that the National Office uses to produce the DCA rates adjusts the DCA populations to exclude the determinations represented by these excluded cases.

8. BAM Report Formats, Definitions, and Supplemental Data. The BAM paid claims Annual Report format is displayed and the components of the Annual Report are defined in Attachment 1. In order for states to interpret the data in the Annual Report, the software also generates a report of the percentages of the estimated UI benefits that were overpaid by responsibility and cause. The format for this report is in Attachment 2. No sampling errors are computed for the responsibility and cause data. States should be aware that the percentages for some of the responsibility and cause categories are based on small numbers of BAM sample cases and may have relatively large sampling errors. A prototype DCA rates report, definitions, and footnotes are provided in Attachment 3.

9. Comment Period. DOL will mail BAM paid and denied claims accuracy rates to each state by May 9, 2003. Each state should carefully review the paid claims Annual Report produced by DOL to verify that it agrees with the Annual Report produced from the state's BAM database. States can check the DCA data by running the Informix ACE DCA rates reports. States should report any discrepancies or submit questions concerning the BAM paid and denied claims data or footnotes to the National Office, with a copy to the appropriate Regional Administrator, by May 30, 2003, to Andrew Spisak by e-mail ([email protected]), fax (202-693-3975), or mail:

U. S. Department of Labor
ETA / Office of Workforce Security
200 Constitution Avenue, NW
Room S-4522
Washington, DC 20210

National and Regional Office staff will address and reconcile issues raised by the states by June 6, 2003.

10. Key Dates. The following key dates for the public release of CY 2002 BAM data are the latest that the task or action must be completed.

• 4/18/03: States submit requests for waivers of BAM paid and denied claims time lapse requirements, along with supporting analysis, to the appropriate ETA Regional Administrator.

• 4/25/03: Regional Offices respond to states on the disposition of state requests for waivers of BAM time lapse requirements.

• 4/30/03: The National Office electronically transmits the footnote lookup table for the BAM paid claims Annual Report software to each state's Sun computer.

• 4/30/03: Each state checks its UI database to: 1) ensure that the BAM supervisor has signed off all BAM paid and denied claims cases which the investigators have completed, and 2) reopen any case in order to update information or correct any data errors. Cases closed by supervisors after April 30, 2003, will not be included in the paid claims Annual Report or DCA rates. States must not reopen any CY 2002 cases after April 30, 2003, until they have reviewed the BAM program data transmitted by the National Office and verified the data against their state database or reconciled any differences with DOL.

• 5/9/03: The National Office sends the CY 2002 BAM program data to the state administrators.

• 5/30/03: States review BAM paid and denied claims accuracy rate data and footnotes, and report any discrepancies or submit questions concerning the BAM data or footnotes to the National Office (to the attention of Andrew Spisak, as discussed in section 9), with copies to ETA Regional Administrators.

• 6/6/03: National and Regional Office staff address issues raised by the states and reconcile any differences in state and DOL BAM program data.

• 6/6/03: States provide the National Office the name, address, telephone number, and e-mail address of the contact person (or persons) for public inquiries concerning BAM paid and denied claims data. This information should be sent to the attention of Andrew Spisak at the address, e-mail address, or fax provided in section 9, with copies to the appropriate ETA Regional Administrator.

• 8/29/03: DOL publishes the CY 2002 UI PERFORMS Annual Report on the OWS public Web site, along with names of state contacts for BAM paid and denied claims data.

11. Action Required. State Administrators are requested to:

a. provide copies of these guidelines to the appropriate staff;

b. send any request for a waiver of time lapse requirements to the appropriate ETA Regional Administrator by c.o.b. April 18, 2003;

c. ensure that any issues related to CY 2002 BAM paid or denied claims data or footnotes that require reconciliation of differences between the state's and DOL's BAM program data are sent to DOL's National Office and that copies are sent to the appropriate ETA Regional Administrator by May 30, 2003; and

d. ensure that the name, address, telephone number, and e-mail address of the contact person(s) to whom interested parties may write or call with questions or inquiries about the state's BAM program data are sent to DOL's National Office and copies are sent to the appropriate ETA Regional Administrator by June 6, 2003.

12. Inquiries. Questions should be directed to the appropriate Regional Office.

13. Attachment. (1) BAM Annual Report format, footnotes, and definitions; (2) BAM Annual Report Supplemental Data format; (3) DCA rate prototype report, definitions, and DCA footnotes.

Attachment 1: BAM Annual Report format, footnotes, and definitions

Official attachment: https://www.dol.gov/sites/dolgov/files/ETA/advisories/UIPL/2003/UIPL19-03_Attach1.html

QC ANNUAL REPORT (New Version), CY 2002
Run as of 03/13/2003; State: XX
Unemployment Insurance Quality Control Annual Report for Calendar Year 2002

Footnotes

[Footnote definitions are provided below.]

BAM Annual Report Definitions

Footnote 1 Claimants failing to conduct required work search were given formal warnings and no overpayment was established. The proper payment rate would be lower and the overpayment rate would be higher if these cases were counted as erroneous payments.

Condition: Any CY 2002 BAM case with key week action code '14' in field ei2 of the b_errisu table of the UI database.

Footnote 2 Percentages apply to less than a complete year of UI payments due to the state not pulling a sample for x weeks.

Condition: There is no record in the b_comparison table of the UI database for one or more BAM batches 200201 through 200252.

Note: If a state requested and received permission from DOL to temporarily suspend BAM sampling due to workload contingencies the footnote will reflect that DOL concurred with the suspension.

Footnote 3 The state selected samples that were below the minimum prescribed levels for x weeks.

Condition: State selected one or more BAM weekly samples below the minimum level prescribed in Benefit Accuracy Measurement State Operations Handbook, ET Handbook No. 395, chapter III, p. 27. Sampled cases that fail to meet the BAM population definition are counted toward meeting the minimum weekly sample.

Note: If a state requested and received permission from DOL to temporarily reduce BAM sample sizes due to workload contingencies, the footnote will reflect that DOL concurred with the reduction.

Footnote 4 Percentages based on data collection procedures that were not completely in accordance with the program methodology prescribed in ET Handbook No. 395.

Condition: Based on program monitoring conducted by DOL Regional and National Offices.

Footnote 5 The state completed x percent of the cases within 90 days. The program standard is 95 percent completed within 90 days.

Condition: State failed to meet case completion objectives established in Benefit Accuracy Measurement State Operations Handbook, ET Handbook No. 395, chapter VI, p. 7. Cases not meeting the BAM population definition are not counted in calculating state time lapse rates.

Footnote 6 x percent of the sample cases were not completed when this report was prepared. This exceeded the program requirement that no more than 2 percent of the cases for the year remain incomplete.

Condition: The percentage is based on the number of BAM cases that were not completed (no supervisor sign-off) by c.o.b. April 30, 2003, divided by the number of valid cases selected for BAM weekly samples during CY 2002 (BAM batches 200201 through 200252). Cases that do not meet the BAM population definition are not counted in calculating state case completion rates.

Footnote 7 The annual sample for [state] is x cases below the allocated annual sample for the state. The precision of the data might be reduced due to the failure to sample at the prescribed level.

Condition: The state's annual sample selection is 60 or more cases (one-half staff year) below its allocated annual sample (for BAM paid claims, one staff year equals 120 cases). Cases that do not meet the BAM population definition are counted toward meeting the annual sample allocation.

Note: If a state requested and received permission from DOL to temporarily reduce BAM sample sizes due to workload contingencies, the footnote will reflect that DOL concurred with the reduction.

Footnote 8 The population from which the BAM sample was selected did not include all of the UI benefits paid. This limits the degree to which inferences about the population can be made from BAM data.

Condition: The dollars paid in the BAM population for CY 2002 vary from the dollars paid reported in the state's ETA 5159 Claims and Payment Activities reports by more than the statistical control limit.

Attachment 2: BAM Annual Report supplemental data format

Official attachment: https://www.dol.gov/sites/dolgov/files/ETA/advisories/UIPL/2003/UIPL19-03_Attach2.html

QC ANNUAL REPORT FOR XX, Supplemental Data, CY 2002

Responsibility for Overpayments (Percent of Dollars Overpaid)

Cause for Overpayments (Percent of Dollars Overpaid)

NOTE: The percentages for the responsibilities or cause do not sum to 100 percent. You should check field (h5) to verify that all cases have been coded correctly within your specified batch range.

If cause and responsibility percentages do not add up to 100 percent, states should check for cases in which the key week amount overpaid coded in data element h5 in the b_master table of the UI database does not equal the sum of dollars overpaid coded in data element ei1 for key week actions 10, 11, 12, 13, and 15 in the b_errisu table. The amount overpaid cannot exceed the amount paid coded in data element f13 in the b_master table.

Attachment 3: DCA report prototype, definitions, and footnotes

Official attachment: https://www.dol.gov/sites/dolgov/files/ETA/advisories/UIPL/2003/UIPL19-03_Attach3.html

Denied Claims Accuracy Rates Report
State: XX; Beginning Batch: 200201; Ending Batch: 200252

* Excludes cases not meeting DCA definition for inclusion in population, withdrawn claims, and claims for which monetary eligibility was established upon receipt of CWC, UCFE, and/or UCX wage credits.

# Adjusted rate excludes erroneous denials that were corrected by agency or reversed on appeal prior to DCA case completion.

Note: 95% C.I. is the 95 percent confidence interval for the estimated rate. The interval is the range between the rate minus the value in the 95% C. I. column and the rate plus the value in the 95% C. I. column. For example, the interval for 10.0% +/- 2.5 is 7.5% to 12.5%. The true rate is expected to lie within 95 percent of the intervals constructed from repeated samples of the same size and selected in the same manner as the BAM DCA sample.

Run Date: 02/01/2003

BAM Denied Claims Accuracy Rates Report Definitions

Population. The number of denial determinations that constitute the sampling frames for all weeks in CY 2002 for which the state pulled a BAM DCA sample, adjusted to exclude agency actions that do not meet the definition of the BAM DCA population. These excluded cases are coded "8" (TEUC) or "9" (all other excluded records) in the Program code data element in the b_dca_master table of the UI database. In addition, DCA rates exclude cases for which monetary eligibility was established upon receipt of CWC, UCFE, and/or UCX wage credits within the two-week sampling lag (Action Code = "0") or withdrawn claims (Action Code = "8").

Sample Size. Total DCA cases selected during CY 2002 (BAM batches 200201 through 200252) and completed (supervisor sign-off) by c.o.b. April 30, 2003, excluding cases that do not meet the BAM DCA population definition. This is the number of BAM DCA sample cases from which the accuracy rates are estimated.

Improper Denial Rate. The weighted proportion of denied claims for which the DCA investigation concluded that the claimant should have been eligible for UI benefits or was underpaid UI benefits (Error Issue Action Code = "20", "21", "22", or "23"), expressed as a percentage.

Adjusted Improper Denial Rate. Improper Denial Rate excluding cases in which erroneous denials were corrected by the agency (Prior Agency Action = "20" - "29" or eligibility was established through appeal (Results of Appeal of Initial Determination = "1" or "3") prior to DCA case completion.

Weighting Procedures. Accuracy rates are computed separately for each batch with at least two completed DCA cases and weighted by the batch population, adjusted to exclude cases that do not meet the BAM DCA definition (Program code "8" or "9") or cases with Action Codes "0" (monetary eligibility established upon receipt of CWC, UCFE, and/or UCX wage credits) or "8" (withdrawn claims). The accuracy rate equals the sum of the products of the batch rates and weights. Batches with only one completed case are merged to insure that each batch has a minimum of two completed cases.

95 Percent Confidence Interval. A confidence interval, expressed as +/- x percentage points, is constructed for each of the estimated accuracy rates. The actual rate is expected to lie within 95 percent of the intervals constructed from repeated samples of the same size and selected in the same manner as the BAM sample.

BAM Denied Claims Accuracy Footnotes

Footnote 1 Not applicable to DCA.

Footnote 2 Percentages apply to less than the total population of denials due to the state not pulling a sample for x weeks.

Condition: There is no record in the b_dca_comparison table of the UI database for one or more BAM batches 200201 through 200252.

Note: If a state requested and received permission from DOL to temporarily suspend BAM sampling due to workload contingencies, the footnote will reflect that DOL concurred with the suspension.

Footnote 3 The state selected samples that were below the minimum prescribed levels for x weeks.

Condition: State selected one or more BAM DCA weekly samples below the minimum level prescribed in Benefit Accuracy Measurement State Operations Handbook, ET Handbook No. 395, chapter III, p. 27. Sampled cases that fail to meet the BAM DCA population definition are counted toward meeting the minimum weekly sample.

Note: If a state requested and received permission from DOL to temporarily reduce BAM sample sizes due to workload contingencies, the footnote will reflect that DOL concurred with the reduction.

Footnote 4 Percentages based on data collection procedures that were not completely in accordance with the program methodology prescribed in ET Handbook No. 395.

Condition: Based on program monitoring conducted by DOL Regional and National Offices.

Footnote 5 The state completed x percent of the cases within 90 days. The program standard is 85 percent completed within 90 days.

Condition: State failed to meet case completion objectives established in Benefit Accuracy Measurement State Operations Handbook, ET Handbook No. 395, chapter VIII, p. 3. Cases not meeting the BAM DCA population definition are not counted in calculating state time lapse rates.

Footnote 6 x percent of the sample cases were not completed when this report was prepared. This exceeded the program requirement that no more than 2 percent of the cases for the year remain incomplete.

Condition: The percentage is based on the number of BAM DCA cases that were not completed (no supervisor sign-off) by c.o.b. April 30, 2003, divided by the number of valid DCA cases selected for BAM weekly samples during CY 2002 (BAM batches 200201 through 200252). Cases that do not meet the BAM DCA population definition are not counted in calculating state case completion rates.

Footnote 7 The annual sample for [state] is x cases below the allocated annual sample for the state. The precision of the data might be reduced due to the failure to sample at the prescribed level.

Condition: The state's annual monetary, separation, or nonseparation sample selection is 38 or more cases (one-half staff year) below its allocated annual sample (for DCA, one staff year equals 75 cases for each DCA sample type). Cases that do not meet the BAM population definition are counted toward meeting the annual sample allocation.

Footnote 8 The population from which the BAM DCA samples were selected may not include all of the determinations that meet the definition for inclusion in the DCA population or may include records that do not meet the DCA population definition. This limits the degree to which inferences about the population can be made from BAM DCA data.

Condition: State was notified in November 4, 2002, summary transmitted to Regional Offices that a DCA population issue was identified, and State has not resolved issue as of April 30, 2003.