Program year 2011 WIA annual report narrative

Accessible letter PDF: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2012/TEGL_29-11-Acc.pdf

CLASSIFICATION
   EMPLOYMENT AND TRAINING ADMINISTRATION                   WIA/Performance Reporting System
               ADVISORY SYSTEM                              CORRESPONDENCE SYMBOL
          U.S. DEPARTMENT OF LABOR                          OPDR
             Washington, D.C. 20210                         DATE
                                                            May 16, 2012

ADVISORY:           TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 29-11

TO:                STATE WORKFORCE AGENCIES
                   STATE WORKFORCE ADMINISTRATORS
                   STATE WORKFORCE LIAISONS

FROM:               JANE OATES /s/
                    Assistant Secretary

SUBJECT:            Workforce Investment Act (WIA) Program Year (PY) 2011 Annual Report
                    Narrative

1. Purpose. To provide guidance to the states regarding the content of the WIA Annual Report
narrative; including strategies for serving veterans (as referenced in section 4, part E), and the
procedures for submission of the report to the Employment and Training Administration
(ETA). The Annual Report is due on Monday, October 1, 2012.

2. References.

       • Workforce Investment Act, Sections 134, 136 and 185;
       • 20 Code of Federal Regulations Section 667.300;
       • Training and Employment Guidance Letter (TEGL) 09-11, Guidance on Revised
          Fiscal Year 2012 “Advance Funding Levels Available October 2011 for Workforce
          Investment Act Programs and Clarification on the Level of Governor’s Reserve;
       • TEGL 21-11, Instructions for Submitting Workforce Investment Act and Wagner-
          Peyser Act State Plans for Program Year 2012;
       • TEGL 21-10, Change 1, Program Year 2010/Fiscal Year 2011 Performance Reporting
          and Data Validation Timeline;
       • TEGL 09-10, Guidance on the Workforce Investment Act Annual Report Narrative;
       • TEGL 17-05 and TEGL 17-05, Change 2, Common Measures Policy for the
          Employment and Training Administration’s Performance Accountability System and
          Related Performance Issues; and
        • Office of Management and Budget (OMB) Circular No. A-94, Revised Guidelines
          and Discount Rates for Benefit-Cost Analysis of Federal Programs.

3. Background. Each state that receives an allotment under WIA Section 127 (Youth activities)
or Section 132 (Adult and Dislocated Worker activities) must prepare and submit an Annual
Report of performance progress to the Secretary of Labor in accordance with WIA Sections 136
and 185.

 RESCISSIONS                                               EXPIRATION DATE
 TEGL 17-10                                                Continuing
 TEGL 7-11                                                 Continuing

As provided in TEGL 09-10, There are two components to the WIA Annual Report: (1) the
required performance results, as specified in form ETA 9091—the WIA Title 1B Annual Report
(OMB No. 1205-0420) found at:
http://www.doleta.gov/performance/guidance/WIA/WIAAnnualReportSpecifications.pdf, and
(2) a narrative report. This guidance focuses on the narrative report and what states should
address in this report.

4. WIA Annual Report Narrative. The required portions of the Annual Report
narrative include the information required by WIA sections 136(d) (1) and (2) and
185(d). This includes:

   •   Performance data on the core and customer satisfaction measures, including progress of
       local areas in the state in achieving local performance measures;
   •   Information on the status of state evaluation activities;
   •   Information on the cost of workforce investment activities relative to the effect of the
       activities on the performance of participants;
   •   Assurance that all required elements are reported uniformly so that a state-by-state
       comparison can be made;
   •   Information on participants in the workforce investment system (this information is also
       included in the performance results portion of the Annual Report); and
   •   A listing of the waivers for which the state has received approval, information on
       how the waivers have changed the activities of the state and local areas, and how
       activities carried out under the waivers have directly or indirectly affected state and
       local area performance outcomes. Further guidance and information on the current
       ETA waiver policy can be accessed at www.doleta.gov/waivers.

Additional information regarding the content of the required components is provided below.

Status of State Evaluation Activities

As WIA section 134(a)(2)(B)(ii) notes, conducting evaluations of workforce investment
activities under WIA section 136(e) is a required statewide activity. States should include
information about all evaluation studies that were started and/or completed during the program
year for which the WIA Annual Report is being submitted. For each evaluation, the narrative
should include:

   • The timeline for starting and completing the evaluation;
   • The questions for evaluation did/will address;
   • A description of the evaluation’s methodology, including description of any control or
       comparison group and description of the analysis technique employed;
   •   The timeline for the final report and other deliverables; and,
   •   Summary of evaluation findings, including summary of best practices, for those
       evaluations completed during the program year for which the WIA Annual report is being
       submitted.

If no evaluations were started and/or completed during the PY for which the WIA Annual
Report is being submitted, then states should submit the following information for each


planned evaluation:

   • Expected timeline for starting and completing the evaluation; and
   • The questions the evaluation is expected to address.

These evaluation studies, conducted under WIA Title 1B, are to promote, establish, and
implement methods for continuous improvement in the efficiency and effectiveness of the
statewide workforce investment system in improving employability for job seekers and
competitiveness for employers. These evaluation studies identify best practices or replicable
models and tools, as well as challenges and potential workforce solutions. This information
would be helpful to the public workforce system and may inform ETA’s national evaluation and
research agenda. As such, states are encouraged to share a copy of each final evaluation report
with ETA via their regional representative. ETA will share relevant best practices, lessons
learned, and other resources with the public workforce system through its online technical
assistance (TA) platform (www.Workforce3One.org) and other appropriate TA events.

Costs of Workforce Investment Activities

States should explain how the mix of services for adults, dislocated workers, and youth
activities affected the outcomes. For adults and dislocated workers, the activities that states
may wish to address are core, intensive, and training services. For youth activities, states may
wish to include information about front-end costs (e.g., intake, assessment and case
management) and aggregated direct service costs for the 10 youth program elements described
in WIA section 129(c)(2).

ETA has a long-term interest in improving program productivity so that both taxpayers and
customers can be better served. A variety of methods for calculating and presenting cost-
related activities and measures are described in prior WIA State Annual Reports. Among the
most common cost calculations reported by the states is the “cost per participant” indicator.

To facilitate further discussion about alternative cost measures (e.g., costs per outcome),
OMB charged ETA with the responsibility of developing and implementing an outcome-
based measure for programs using the common performance measures. The attachment to
this TEGL offers examples of potential, alternative efficiency measures.

In addition to the required components of the WIA Annual Report narrative, ETA encourages
states to include the following information in its narrative:

   A. States also may want to include information from their strategic plans that highlights
      innovative service delivery strategies, including program activities that support
      dislocated workers, low-skilled/low-income adults and disadvantaged youth, the
      outcomes expected, as well as, the actual outcomes for their major customer
      populations. States may indicate actual federal outlays for selected activities, if such
      information is available.

   B. A discussion of the state’s unique programs and recent accomplishments. Such a
      report can describe these accomplishments in the most advantageous manner to all
      stakeholders and partners, including Congress, governors, state legislators and
      workforce investment boards. States may want to highlight “success stories” that


       focus attention on successful programs for participants, employers, and communities.

   C. Messages from the governor or other contextual information about state workforce
      investment board members, market analysis, strategies for improvement, and effects
      on major industries may also be included.

   D. A discussion of the activities funded by the state’s discretionary (“5 percent”) funds.
      In this section of the narrative report, states may describe activities undertaken in
      whole or in part with their discretionary funds, and how those activities directly or
      indirectly affect performance.

   E. A discussion of programs and strategies for serving employers at the state and local
      level, including the performance metrics used by states or local areas to measure the
      effectiveness of such services and current available performance data. Effects on major
      industries may also be included.

   F. A discussion of the initiatives and activities outlined in the WIA and Wagner-Peyser Act
      State Strategic Plan to improve performance.

   G. A discussion of the programs, initiatives, and strategies for serving Veterans at the state
      and local level; including, the performance metrics used by states or local areas to
      measure the effectiveness of such services and current available performance data.
      Include a description of how veterans’ priority of service is being implemented for all
      Department of Labor training programs. Please describe how the state has implemented
      the Gold Card Initiative, to provide intensive services to post 9/11 Veterans, including
      how the six month follow-up has been implemented.

5. Due Date. The Annual Report is due no later than October 1 following each program year.
For example, the report for PY 2011 will be due October 1, 2012. The Annual Report, which
includes the required performance reports and the narrative, will reflect accurate performance
outcome information that is available by the time the Annual Report for the program year is due.
Failure to submit the performance progress reports by the deadline may lead to incentive grant
ineligibility for a state.

6. Submission. An electronic copy of the WIA State Annual Report narrative should be e-mailed to [email protected] by October 1, 2012. States should also submit an electronic copy
to their respective ETA Regional Administrator. Hard copies of the report may be submitted but
are no longer required, in an effort to be more environmentally friendly. ETA will publish each
state’s report on the Internet at www.doleta.gov/performance. Acceptable formats include:
WordPerfect, Microsoft Word, PDF, and other commonly used formats.

7. Action Requested. Distribute this TEGL to those personnel responsible for developing the
WIA Annual Report narrative, including personnel responsible for performance reporting, and
to all local areas responsible for administering the WIA programs.

8. Inquiries. Questions concerning this TEGL should be directed to your appropriate Regional
Office.

9. Attachment. Overview of Potential Alternative Efficiency Measures for Consideration.

Attachment: Overview of Potential Alternative Efficiency Measures for Consideration (https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2012/TEGL_29-11-Atta.pdf).

ATTACHMENT

         Overview of Potential Alternative Efficiency Measures for Consideration

Examples of costs in relation to participant services and outcomes

(1) Unit Costs = total cost by service / total participation by service.

            Expenditures            Participation                        Unit Costs
       Core Intensive Training Core Intensive Training             Core Intensive Training
        $       $        $      #       #         #                 $       $        $

       Pros:
          • It is applicable to most programs.
          • It makes it easier to understand how costs apply to participant services.
       Cons:
          • Tracking program services and costs by year requires great effort and attention to
             detail; it therefore would be more susceptible to human error.
          • It is of limited use in assessing program effectiveness, because it is not an
             outcome-based measure.

(2) Cost per Participant (CP) = This measure is calculated by taking the total program costs in
     terms of expenditures and dividing by the number of participants served during the year by
     the particular program.

                               CP = All Program Expenditures
                               All Program Participants

     Pros:
        • It is applicable to most programs.
        • Data is readily available.
        • Easy to understand.
        • Can be immediately generated each year.
        • Not costly or burdensome.
     Cons:
        • It is of limited use in assessing program effectiveness, because it is not an outcome-
           based measure.

(3) Cost per Exiter (CE) = It is calculated by taking total program costs in terms of
     expenditures and dividing by the number of exiters terminating the program during the year
     by the particular program.

                               CE = Total Program Expenditures
                               Total Exiters Terminating Program


     Pros:
        • It is applicable to most programs.
        • Data is readily available.
        • Easy to understand.
        • Can be immediately generated each year.
        • Not costly or burdensome.
     Cons:
        • It is of limited use in assessing program effectiveness, because it is not an outcome-
           based measure.

(4) Cost per Entered Employment (CEE) = This measure is calculated by taking total program
     costs in terms of expenditures and dividing by the number of exiters entering employment
     in the first quarter following exit from the particular program.

                              CEE = Total Program Costs
                              First Quarter Exiters Entering Employment
     Pros:
        •    It is applicable to most programs.
        •    Data is readily available.
        •    Easy to understand.
        •    Can be generated about two quarters after the end of each program year.
        •    Not costly or burdensome.
        •    Measure is an outcome-based efficiency measure. Therefore, it is of substantial use
             in understanding program effectiveness.
     Cons:
        • Does not capture those who entered employment in the same quarter of exit.
        • Puts a premium on quick labor exchange at a time we are trying to improve skills.

(5) Cost per Retained Employment (CRE) = This efficiency measure is calculated by taking
     total program costs in terms of expenditures and dividing by the number of exiters who are
     employed in both the second and third quarters after the exit quarter.

                              CRE = Total Program Costs
                              Exiters Employed in Q2 & Q3 after Exit

     Pros:
        •    Potentially applicable to most programs.
        •    Data is readily available.
        •    Relatively easy to understand.
        •    Relatively low cost and low burden to produce.
        •    It is an outcome-based efficiency measure. Therefore, it is of substantial use to
             understanding program effectiveness and costs.
     Cons:
        • Lengthier lags in data (must wait for several quarters after the end of the program
           year).


(6) Cost per $1,000 Increase in Earnings (CIE) = Total program cost divided by total earnings
     change from 2nd and 3rd pre-program quarters to 2nd and 3rd post-program quarters for
     participants or exiters.

                              CIE = Total Program Costs
                              Total Participant or Exiters Earnings Change from 2nd and
                              3rd pre-program quarters

     Pros:
        •    Potentially applicable to most programs.
        •    Data is readily available.
        •    Relatively low cost and low burden to produce.
        •    It is an outcome-based efficiency measure. Therefore, it is of substantial use to
             understanding program effectiveness and costs.
     Cons:
        • Somewhat difficult to understand.
        • Lengthier lags in data (must wait several quarters after the end of the program year).

(7) Cost per $1,000 in Post-Program Earnings (CPPE) = Total program cost divided by total
     earnings in 2nd and 3rd post-program quarters for participants of exiters multiplied by
     $1,000.

                              CPPE = Total Program Costs
                              Total Participant or Exiters Earnings in 2nd and 3rd post-
                              Program quarters multiplied by $1,000

     Pros:
        •    Potentially applicable to most programs.
        •    Data is readily available.
        •    Relatively low cost and low burden to produce.
        •    It is outcome-based.
        •    Unlike the prior measure, does not weight prior employment earnings against post
             program earnings.
     Cons:
        • Lengthier lags in data (must wait several quarters after the end of program year).
        • Somewhat difficult to understand.

(8) Cost per Exiter or Participant Receiving a Particular Service (CPS) = Total program cost
     of a particular service divided by the number of exiters or participants receiving a particular
     service.

                              CPS = Total Cost of Particular Program
                              Participants or Exiters Who Received Particular Service

     Pros:
        • Easy to understand.
        • No lags in data. Data can be immediately generated at the end of each year.


     Cons:
        • Only applicable to programs that distinguish types of service.
        • Data is readily available for some programs, but not all.
        • Is not an outcome-based efficiency measure.
        • May be burdensome to generate.

(9) Cost per Placement in Employment or Education (CPEE) = Total program cost divided
     by the number of participants or exiters in employment or enrolled in post secondary
     education and/or advanced training or advanced training occupational skills in the 1st
     quarter after exit.

                              CPEE = Total Program Costs
                              Number of Exiters or Participants Employed or in Post
                              Secondary Education Programs after 1st Quarter Exit

     Pros:
        • The data is relatively easy to understand.
        • Relatively low cost and low burden to produce.
        • The measure is outcome-based so it is of substantial use to understanding program
           effectiveness.
     Cons:
        • Limited to primarily the Workforce Investment Act Youth program.

(10) Cost per Individual Attaining a Recognized Degree or Certificate (CID) (Credentials
     include but are not limited to, a high school diploma, GED, or other recognized
     equivalents, post-secondary degrees/certificates, recognized skill standards, and licensure
     or industry-recognized certificates.) = Total training program cost divided by the number of
     participants or exiters receiving a training service attaining a recognized credential during
     participation or by the end of the 3rd quarter after exit.

                              CID = Total Training Program Costs
                              Number of Participants or Exiters who Attained
                              Certification or Degree by the end of 3rd Quarter after exit

     Pros:
        • The measure is an outcome-based measure, so it is of substantial use in
           understanding program effectiveness.
     Cons:
        • Only applicable to programs that provide services and identify individuals as
           receiving training and types of credentialing.
        • Data is readily available for some programs, but not all.
        • The measure is somewhat difficult to understand.
        • Potentially lengthy lags in data.

(11) Return on Investment (ROI). ROI is a way of summarizing how large the gain on an
     investment, such as workforce development, actually is. In its simplest form, ROI is
     calculated by dividing the gain by the size of the investment. This equation can be written
     as B/C, where B is the sum of all benefits that result from the investment over the period


considered and C represents the costs. For a workforce program, one would divide the
increase in earnings due to the program by the cost of the program. In more sophisticated
analyses, ROI calculations take into account the timing of the gains due to the program.
Economists typically compute a variation called the internal rate of return (IRR), which is
based on the costs and benefits over the life of the investment. The IRR can be calculated,
using a financial calculator or a spreadsheet, by solving the following equation for i:

0 = -C0 + (B1 –C1)/(1+i) + (B2 –C2)/(1+i)2 + (B3 –C3)/(1+i)3 + …. + (BN –CN)/(1+i)N

Where Bt is the benefit received in year i, Ct is the cost incurred in year i, and N is the last
year that benefits or costs occur. (The four dots mean that the formula includes the same
type of term for all years between year 3 and year N.) The IRR is preferred to the simpler
versions of ROI because it takes into account the timing of the costs and benefits.

Pros:
   • Potentially applicable to most programs.
   • Measure is an impact-based efficiency measure, which controls for factors that
      could potentially influence/bias results. Therefore, it is of the greatest utility in
      understanding program cost-effectiveness.
   • This measure controls for difficulty or cost of serving different populations (e.g.,
      hard-to-serve, service mix, and economic conditions).
Cons:
   • Data is very costly to produce.
   • The measure is difficult to understand.
   • Lengthy lags in data.