WIA annual report narrative guidance

Accessible letter PDF: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2010/TEGL09-10acc.pdf

                                                         CLASSIFICATION
   EMPLOYMENT AND TRAINING ADMINISTRATION                WIA/Performance Reporting System
                                                         CORRESPONDENCE SYMBOL
               ADVISORY SYSTEM
                                                         OPDR
          U.S. DEPARTMENT OF LABOR                       DATE
             Washington, D.C. 20210
                                                         September 23, 2010


ADVISORY:         TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 9-10

TO:               STATE WORKFORCE AGENCIES
                  STATE WORKFORCE ADMINISTRATORS
                  STATE WORKFORCE LIAISONS

FROM:             JANE OATES /s/
                  Assistant Secretary

SUBJECT:          Workforce Investment Act (WIA) Annual Report Narrative

1. Purpose. To provide guidance to the states about the content of the WIA Annual
Report narrative and the procedures for submission of the report to the Employment and
Training Administration (ETA).

2. References.
   • The Workforce Investment Act of 1998 (WIA), sections 134, 136 and 185;
   • 20 CFR Section 667.300
   • Training and Employment Guidance Letter (TEGL)-14-00, TEGL 14-00, Change
       1, Guidance on the Workforce Investment Act (WIA) Management Information
       and Reporting System
   • TEGL 14-03 and TEGL 14-03, Change 1, Performance Reporting Submission
       Procedures for the Workforce Investment Act Standardized Record Data
       (WIASRD), the Annual Report, and the Quarterly Reports under Title IB of the
       Workforce Investment Act (WIA) for Program Year (PY) 2003 and 2004
   • TEGL 17-05 Common Measures Policy for the Employment and Training
       Administration’s (ETA) Performance Accountability System and Related
       Performance Issues
   • TEGL 14-08, Guidance for Implementation of the Workforce Investment Act and
       Wagner-Peyser Act Funding in the American Recovery and Reinvestment Act of
       2009 and State Planning Requirements for Program Year 2009
   • TEGL 26-09, Workforce Investment Act (WIA) Waiver Policy and Waiver
       Decisions for PY 2009 and 2010
   • TEGL 31-09, Program Year 2009/Fiscal Year 2010 Performance Reporting and
       Data Validation Timelines
   • Office of Management and Budget (OMB) Circular No. A-94, Revised Guidelines
       and Discount Rates for Benefit-Cost Analysis of Federal Programs.



RESCISSIONS                                            EXPIRATION DATE
TEGL 14-00, Change 2 and TEGL 14-00, Change 3          Continuing
3. Background. Each state that receives an allotment under WIA Section 127 (youth
activities) or Section 132 (Adult and Dislocated Worker activities) of the Workforce
Investment Act of 1998 must prepare and submit an Annual Report of performance
progress to the Secretary of Labor in accordance with WIA Sections 136 and 185.

There are two components to the WIA Annual Report: (1) the required performance
results, as specified in Form ETA 9091—the WIA Title 1B Annual Report (OMB No.
1205-0420) found at: http://www.doleta.gov/performance/reporting/, and (2) a narrative
report. This guidance focuses on the narrative report and what states should address in
the narrative.

4. WIA Annual Report Narrative. The required portions of the annual report narrative
include the information required by WIA Sections 136(d) (1) and (2) and 185(d). This
includes:
    • Performance data on the core and customer satisfaction measures,1 including
       progress of local areas in the state in achieving local performance measures;
    • Information on the status of state evaluation activities;
    • Information on the cost of workforce investment activities relative to the effect of
       the activities on the performance of participants;
    • Assurance that all required elements are reported uniformly so that a state-by-
       state comparison can be made; and,
    • Information on participants in the workforce investment system (this information
       is typically addressed in the data portion of the Annual Report).

In addition to the required components of the WIA Annual Report narrative, ETA
encourages states to include the following information in its narrative:

    A. A discussion of the state’s unique programs and recent accomplishments. Such a
       report can describe these accomplishments in the most advantageous manner to
       all stakeholders and partners, including Congress, governors, state legislators and
       workforce investment boards. States may want to highlight “success stories” that
       focus attention on successful programs for participants, employers and
       communities. Messages from the governor or information about state workforce
       investment board members, market analysis, strategies for improvement, and
       effects on major industries may also be included.

    B. A discussion of the activities funded by the state’s discretionary (“15 percent”)
       funds. In this section of the narrative report, states may describe activities
       undertaken in whole or in part with their discretionary funds, and how those
       activities directly or indirectly affect performance.

    C. A discussion of the initiatives and activities outlined in the WIA and Wagner-
       Peyser Act State Strategic Plan to improve performance.

        TEGL 14-08 provided instructions to states for modifying their existing state
        plans to describe the implementation of the American Recovery and Reinvestment

1
  States that are approved to report only the common performance measure outcomes for WIA programs
do not have to report outcomes for the customer satisfaction measures, the credential measure for Adults
and Dislocated Workers, or the Older and Younger Youth measures on the WIA Annual Report.
                                                    2
     Act of 2009 (Recovery Act) and the response to the economic downturn, and
     updated ETA’s “National Strategic Direction.”

     In their strategic state plans, many states described initiatives that supported the
     Department’s strategic direction, some of which were in the preliminary stages.
     States may have progressed considerably in implementing such initiatives after
     submitting their state plan. This section of the narrative may be used to describe
     state activities in the time periods between state plans. States may provide an
     updated status of these initiatives and activities, including a discussion of how
     these initiatives have directly or indirectly affected performance. States may also
     use this portion of the narrative to describe activities or initiatives that began after
     the submission of the strategic state plan.

     States may want to also include information from their strategic plans that
     highlight innovative service delivery strategies, including program activities that
     support dislocated workers, low-skilled/low-income adults and disadvantaged
     youth, the outcomes expected, as well as the actual outcomes for their major
     customer segments. States may indicate actual Federal outlays for selected
     activities, if the information is available.

     Additionally, there are three areas that a state should focus on in the narrative: the
     waivers the state has; the status of state evaluation activities; and the costs of
     workforce investment activities. See below for additional information to be
     addressed in each area.

D. Waivers:

     States should include the waivers for which the state has received approval, how
     the waivers have changed the activities of the state and local areas, and how
     activities carried out under the waivers have directly or indirectly affected state
     and local area performance outcomes. TEGL 26-09 describes the waiver policy
     for PY 2009 and 2010.

E.   Status of State Evaluation Activities:

     As WIA Section 134(a)(2)(B)(ii) notes, conducting evaluations of workforce
     investment activities under WIA Section 136(e) is a required statewide activity.
     States should include information about all evaluation studies that were started
     and/or completed during the program year for which the WIA Annual Report is
     being submitted. For each evaluation, the narrative should include:
     • The timeline for starting and completing the evaluation;
     • The questions the evaluation will/did address;
     • A description of the evaluation’s methodology, including description of any
         control or comparison group and description of the analysis techniques
         employed;
     • The timeline for the final report and other deliverables; and
     • Summary of evaluation findings, including a summary of best practices, for
         those evaluations completed during the program year for which the WIA
         Annual report is being submitted.

                                            3
       If no evaluations were started and/or completed during the program year for
       which the WIA Annual Report is being submitted, then states should submit the
       following information for each planned evaluation:

       •   Expected timeline for starting and completing the evaluation; and
       •   The questions the evaluation is expected to address.

       These evaluation studies, conducted under WIA Title 1B, are to promote,
       establish, and implement methods for continuous improvement in the efficiency
       and effectiveness of the statewide workforce investment system in improving
       employability for job seekers and competitiveness for employers. These
       evaluation studies may also identify best practices or replicable models and tools,
       as well as challenges and potential workforce solutions. This information would
       be helpful to the public workforce system and may inform ETA’s national
       evaluation and research agenda. As such, states are encouraged to share a copy of
       each final evaluation report with ETA via their Regional representative. ETA will
       share relevant best practices, lessons learned, and other resources with the public
       workforce system through its online technical assistance (TA) platform
       (www.Workforce3One.org) and other appropriate TA events.

   F. Costs of Workforce Investment Activities:

       States should explain how the mix of services for adults, dislocated workers, and
       youth activities affected the outcomes. For adults and dislocated workers, the
       activities that states may wish to address are core, intensive and training services.
       For youth activities, states may wish to include information about front-end costs
       (e.g., intake, assessment and case management) and aggregated direct service
       costs for the ten youth program elements described in WIA Section 129(c)(2).

       ETA has a long-term interest in improving program efficiency so that both
       taxpayers and customers can be better served. A variety of methods for
       calculating and presenting cost-related activities and measures are described in
       prior WIA State Annual Reports. Among the most common cost calculations
       reported by the states is the “cost per participant” indicator. “Cost per
       participant” is of limited use in measuring the effectiveness of a program, because
       it measures appropriations or allotments to participants served.

       To facilitate further discussion about costs to outcomes, OMB charged ETA with
       the responsibility of developing and implementing an outcome-based measure for
       programs using the common performance measures. A study is underway to
       determine how costs and other measures of efficiency might be utilized to assess
       overall program effectiveness. Results will be available on ETA’s Web site when
       the study is completed. The attachment to this TEGL offers examples of potential
       efficiency measures.

5. Due Date. The Annual Report is due no later than October 1 following each program
year. For example, the report for PY 2009 will be due October 1, 2010. The Annual
Report, which includes the required performance reports and the narrative, will reflect
accurate performance outcome information that is available by the time the Annual

                                             4
Report for the program year is due. Failure to submit the performance progress reports
by the deadline may lead to incentive grant ineligibility for a state.

6. Submission. An electronic copy of the WIA State Annual Report narrative is due to
[email protected] by October 1. States should also submit an electronic copy to their
respective ETA Regional Administrator. ETA will publish each state’s report on the
Internet at www.doleta.gov/performance. Acceptable formats include WordPerfect,
Microsoft Word, Acrobat, and other commonly used formats. In an effort to be more
environmentally friendly hard copies of the report may be submitted, but are no longer
required.

7. Action Requested. Distribute this TEGL to those personnel responsible for
developing the WIA Annual Report narrative, including personnel responsible for
performance reporting, and to all local areas responsible for administering the WIA
programs.

8. Inquiries. Questions concerning this TEGL should be directed to your appropriate
Regional Office.

9. Attachment: Overview of Potential Alternative Efficiency Measures for
Consideration.




                                            5

Official attachment PDF: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2010/TEGL09-10a1.pdf

                                                                            ATTACHMENT

             Overview of Potential Alternative Efficiency Measures for
                                  Consideration

Examples of costs in relation to effect of participant services and outcomes

(1) Unit Costs = total cost by service / total participation by service.

        Expenditures            Participation                               Unit Costs
   Core Intensive Training Core Intensive Training               Core      Intensive Training
    $       $        $      #       #         #                   $            $        $

Pros:
   • It is applicable to most programs.
   • It makes it easier to understand how costs apply to participant services.
Cons:
   • Program services and costs must be closely tracked by year.

(2) Cost per Participant = This measure is calculated by taking the total program costs
in terms of expenditures and dividing by the number of participants served during the
year by the particular program.

Pros:
   • It is applicable to most programs.
   • Data is readily available.
   • Easy to understand.
   • Can be immediately generated each year.
   • Not costly or burdensome.
Cons:
   • It is of limited use in assessing program effectiveness, because it is not an
      outcome-based measure.

(3) Cost per Exiter = It is calculated by taking total program costs in terms of
expenditures and dividing by the number of exiters terminating the program during the
year by the particular program.

Pros:
   • It is applicable to most programs.
   • Data is readily available.
   • Easy to understand.
   • Can be immediately generated each year.
   • Not costly or burdensome.
Cons:
   • It is of limited use in assessing program effectiveness, because it is not an
      outcome-based measure.


                                              1
(4) Cost per Entered Employment = This measure is calculated by taking total program
    costs in terms of expenditures and dividing by the number of exiters entering
    employment in the first quarter following exit from the particular program.

Pros:
   •    It is applicable to most programs.
   •    Data is readily available.
   •    Easy to understand.
   •    Can be generated about two quarters after the end of each program year.
   •    Not costly or burdensome.
   •    Measure is an outcome-based efficiency measure. Therefore, it is of substantial
        use in understanding program effectiveness.
Cons:
   • Does not capture those who entered employment in the same quarter of exit.

(5) Cost per Retained Employment = This efficiency measure is calculated by taking
    total program costs in terms of expenditures and dividing by the number of exiters
    who are employed in both the second and third quarters after the exit quarter.

Pros:
   •    Potentially applicable to most programs.
   •    Data is readily available.
   •    Relatively easy to understand.
   •    Relatively low cost and low burden to produce.
   •    It is an outcome-based efficiency measure. Therefore, it is of substantial use to
        understanding program effectiveness and costs.
Cons:
   • Lengthier lags in data. Must wait for several quarters after the end of the program
      year.

(6) Cost per $1,000 Increase in Earnings = Total program cost divided by total earnings
    change from 2nd and 3rd pre-program quarters to 2nd and 3rd post-program quarters for
    participants or exiters.

Pros:
   •    Potentially applicable to most programs.
   •    Data is readily available.
   •    Relatively low cost and low burden to produce.
   •    It is an outcome-based efficiency measure. Therefore, it is of substantial use to
        understanding program effectiveness and costs.
Cons:
   • Somewhat difficult to understand.
   • Lengthier lags in data. Must wait several quarters after the end of the program
      year.



                                             2
(7) Cost per $1,000 in Post-Program Earnings = Total program cost divided by total
    earnings in 2nd and 3rd post-program quarters for participants of exiters * $1,000.

Pros:
   • Potentially applicable to most programs.
   • Data is readily available.
   • Relatively low cost and low burden to produce.
   • It is outcome-based.
Cons:
   • Lengthier lags in data. Must wait several quarters after the end of Program Year.
   • Somewhat difficult to understand.

(8) Cost per Exiter or Participant Receiving a Particular Service = Total program
    cost of a particular service divided by the number of exiters or participants receiving a
    particular service.

Pros:
   • Easy to understand.
   • No lags in data. Data can be immediately generated at the end of each year.
Cons:
   • Only applicable to programs that distinguish types of service.
   • Data is readily available for some programs, but not all.
   • Is not an outcome-based efficiency measure.
   • May be burdensome to generate.

(9) Cost per Placement in Employment or Education = Total program cost divided by
    the number of participants or exiters in employment or enrolled in post secondary
    education and/or advanced training or advanced training occupational skills in the 1st
    quarter after exit.

Pros:
   • The data is relatively easy to understand.
   • Relatively low cost and low burden to produce.
   • The measure is outcome-based so it is of substantial use to understanding program
      effectiveness.
Cons:
   • Limited to primarily the WIA Youth program.

(10) Cost per Individual Recognized Degree or Certificate (Credentials include but are
   not limited to, a high school diploma, GED, or other recognized equivalents, post-
   secondary degrees/certificates, recognized skill standards, and licensure or industry-
   recognized certificates.) = Total program cost divided by the number of participants
   or exiters receiving a training service attaining a recognized credential during
   participation or by the end of the 3rd quarter after exit.

Pros:
   • The measure is an outcome-based measure, so it is of substantial use in
      understanding program effectiveness.

                                             3
Cons:
   • Only applicable to programs that provide services and identify individuals as
      receiving training and types of credentialing.
   • Data is readily available for some programs, but not all.
   • The measure is somewhat difficult to understand.
   • Potentially lengthy lags in data.

(11) Return on Investment = Return on Investment (ROI) is a way of summarizing how
large the gain on an investment, such as workforce development, actually is. In its
simplest form, ROI is calculated by dividing the gain by the size of the investment. This
equation can be written as B/C, where B is the sum of all benefits that result from the
investment over the period considered and C represents the costs. For a workforce
program, one would divide the increase in earnings due to the program by the cost of the
program. In more sophisticated analyses, ROI calculations take into account the timing
of the gains due to the program. Economists typically compute a variation called the
internal rate of return (IRR), which is based on the costs and benefits over the life of the
investment. The IRR can be calculated, using a financial calculator or a spreadsheet, by
solving the following equation for i: 0 = -C0 + (B1 –C1)/(1+i) + (B2 –C2)/(1+i)2 + (B3 –
C3)/(1+i)3 + …. + (BN –CN)/(1+i)N where Bt is the benefit received in year I, Ct is the
cost incurred in year I, and N is the last year that benefits or costs occur. (The four dots
mean that the formula includes the same type of term for all years between year 3 and
year N.) The IRR is preferred to the simpler versions of ROI because it takes into
account the timing of the costs and benefits.

Pros:
   • Potentially applicable to most programs.
   • Measure is an impact-based efficiency measure, which controls for factors that
      could potentially influence/bias results. Therefore, it is of the greatest utility in
      understanding program cost-effectiveness.
   • This measure controls for difficulty or cost of serving different populations (e.g.,
      hard-to-serve, service mix, and economic conditions).
Cons:
   • Data is very costly to produce.
   • The measure is difficult to understand.
   • Lengthy lags in data.




                                             4