2013 WIA annual report narrative instructions

The signed original contains the letter and attachment. View the signed original.

                                                          CLASSIFICATION
   EMPLOYMENT AND TRAINING ADMINISTRATION                 WIA/Performance Reporting System
               ADVISORY SYSTEM                            CORRESPONDENCE SYMBOL
          U.S. DEPARTMENT OF LABOR                        OPDR
             Washington, D.C. 20210                       DATE
                                                          October 8, 2014

ADVISORY:         TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 9-14

TO:               STATE WORKFORCE AGENCIES
                  STATE WORKFORCE ADMINISTRATORS
                  STATE WORKFORCE LIAISONS

FROM:             PORTIA WU /s/
                  Assistant Secretary

SUBJECT:          Workforce Investment Act (WIA) Program Year (PY) 2013 Annual Report
                  Narrative

1. Purpose. To provide guidance to the states regarding the content of the WIA Annual
   Report Narrative, including: strategies for serving veterans, acceptable approaches to
   fulfilling the customer satisfaction requirements and the procedures for submission of the
   report to the Employment and Training Administration (ETA). The Annual Report
   Narrative is due on Friday, November 14, 2014.

2. References.

   •   WIA, Sections 134, 136, and 185;
   •   20 CFR 667.300;
   •   Training and Employment Guidance Letter (TEGL) No. 17-05 and TEGL No. 17-05,
       Change 2, Common Measures Policy for the Employment and Training
       Administration’s Performance Accountability System and Related Performance Issues;
   •   TEGL No. 09-07, Revised Incentive and Sanction Policy for Workforce Investment Act
       Title IB Programs;
   •   TEGL No. 09-11, Guidance on Revised Fiscal Year 2012 “Advance Funding” Levels
       Available October 2011 for Workforce Investment Act Programs and Clarification on
       the Level of Governor’s Reserve;
   •   TEGL No. 21-10, Change 1, Program Year 2010/Fiscal Year 2011 Performance
       Reporting and Data Validation Timeline;
   •   TEGL No. 21-11, Instructions for Submitting Workforce Investment Act and Wagner-
       Peyser Act State Plans for Program Year 2012;
   •   TEGL No. 6-13, Guidance on the Workforce Investment Act Annual Report Narrative;
   •   TEGL No. 3-14, Implementing a Job Driven Workforce System;
   •   Office of Management and Budget (OMB) Circular No. A-94, Revised Guidelines and
       Discount Rates for Benefit-Cost Analysis of Federal Programs.


 RESCISSIONS                                             EXPIRATION DATE
 None                                                    Continuing

3. Background. Each state that receives an allotment under WIA Section 127 (Youth
   activities) or Section 132 (Adult and Dislocated Worker activities) must prepare and submit a
   WIA Annual Report of performance progress to the Secretary of Labor in accordance with
   the requirements found in WIA Sections 136 and 185.

   There are two components to the WIA Annual Report: (1) the required performance results,
   as specified in form ETA Form 9091—WIA Title 1B Annual Report Form (OMB No. 1205-
   0420) found at:
   http://www.doleta.gov/performance/guidance/WIA/WIAAnnualReportSpecifications.pdf,
   and (2) a narrative report. This guidance focuses on the narrative report and what states
   should address in this report.

4. WIA Annual Report Narrative. The required portions of the WIA Annual Report
   Narrative include the information required by WIA sections 136(d)(1) and (2) and
   185(d). This includes:

       •   Performance data on the core and customer satisfaction measures, including progress
           that local areas in the state made toward achieving local performance measures;
       •   Information on the status of state evaluation activities;
       •   Information on the cost of workforce investment activities relative to the effect of the
           activities on the performance of participants;
       •   Assurance that all required elements are reported uniformly so that a state-by-state
           comparison can be made;
       •   Information on participants in the workforce investment system (this information is
           also included in the performance results portion of the WIA Annual Report); and
       •   A listing of the waivers for which the state has received approval, information on
           how the waivers have changed the activities of the state and local areas, and how
           activities carried out under the waivers directly or indirectly affected state and
           local area performance outcomes. To the extent possible, states should discuss
           whether waiver implementation and outcomes advance the President’s Job-
           Driven elements (For more information on the Job Driven Elements see TEGL 3-
           14). In future waiver requests, ETA will expect states to demonstrate how a
           given waiver will advance Job-Driven elements in the state.

   We provided additional information regarding the content of the required components below.

   Customer Satisfaction Measures

   The public workforce system established under WIA serves a dual-customer base consisting
   of job seekers and employers. Recognizing the importance of measuring service quality,
   WIA Section 136(b)(2)(B) requires states to measure customer satisfaction for employers and
   participants of state and local agencies that provide employment and training services.

   In PY 2005, ETA began approving a waiver to support adoption of the Common Measures,
   commonly referred to as the Common Measure waiver. States with an approved Common
   Measure waiver report only and are held accountable for the common performance measure
   outcomes; however, states must continue to collect customer satisfaction data. States with an

approved Common Measure waiver have provided information about their customer service
results in the narrative portion of their WIA Annual Report to the Department of Labor
(Department). Currently all but four states (Vermont, Rhode Island, Michigan, and
Minnesota) and Puerto Rico have this waiver for PY 2013, resulting in significantly limited
access to customer satisfaction data at the national level.

In TEGL No. 6-13, Workforce Investment Act (WIA) Program Year (PY) 2012 Annual Report
Narrative, ETA was interested in what approaches and methodologies states were using to
collect customer satisfaction information. In the PY 2012 WIA Annual Reports, some states
identified the approaches and methodologies they were using, including descriptions of how
the information was being used to improve customer service. Other states mentioned they
were in the process of developing new customer satisfaction measures. Many states and
local areas have begun piloting new and potentially less costly methods to collect customer
satisfaction data other than the phone surveys required previously when using the American
Customer Satisfaction Index (ACSI), which was the common methodology used to capture
and report customer satisfaction information prior to PY 2012.

For the reasons above, each state may select its own approach to capturing customer
satisfaction information. States that have already established customer satisfaction measures
should continue with those measures, and states that were in the process of identifying new
measures in PY 2012 should present those new measures for the PY 2013 report. ETA is no
longer contracting with ACSI for the procurement of national weights used in performing the
ACSI calculations. (ACSI remains an acceptable methodology that states and locals may
elect to use at their own cost.) We encourage states to use robust methodologies that use
multiple strategies for data collection (telephone, e-mail, paper surveys, or other technology
methods). ETA requires all states to describe their customer service methodology as part of
their annual narrative. ETA requires quality customer satisfaction narratives to include, at a
minimum:

    1. The approach used (if possible include a sample of the survey);
    2. The number of individuals/employers that were provided customer satisfaction
      outreach;
    3. The response rate;
    4. A summary of the results; and
    5. Any processes for incorporating the customer satisfaction feedback.

This approach provides states with the requested flexibility to use new technologies, which
may lead to higher quality services, while also providing the Department with better insight
into each state’s customer satisfaction initiatives. Through this increased flexibility and by
requiring all states to follow, at a minimum, the above parameters for structuring their
description of customer satisfaction activities, ETA hopes to enhance customer service.
Additionally, this strategy for capturing customer satisfaction provides a more robust national
understanding of state level activities.

For the PY 2013 Annual Report, similar to PY 2012 guidance, ETA is directing states with
an approved Common Measure waiver to provide performance data on the core and customer
satisfaction measures as a part of their narrative, including progress that local areas in the

state have made in achieving local performance measures. ETA made PY 2012 a transition
year to allow states to develop the customer satisfaction measures that were most useful for
them. States that do not have an approved Common Measure waiver must provide a brief
overview of how they transitioned to the PY 2013 reporting requirements described above;
they are exempt from entering information in the customer satisfaction field on ETA Form
9091. As described in TEGL No. 09-07, the omission of customer satisfaction from ETA
Form 9091 will have no impact on states’ eligibility for incentive grants.

Status of State Evaluation Activities

As WIA Section 134(a)(2)(B)(ii) notes, conducting evaluations of workforce investment
activities under WIA section 136(e) is a required statewide activity. States should include
information about all evaluation studies started and/or completed during the program year
for which the WIA Annual Report is submitted. For each evaluation, the narrative should
include:

   • The timeline for starting and completing the evaluation;
   • The questions for evaluation did/will address;
   • A description of the evaluation’s methodology, including description of any control
       or comparison group and description of the analysis technique employed;
   •   The timeline for the final report and other deliverables; and,
   •   A summary of evaluation findings, including summary of best practices, for those
       evaluations completed during the program year for which the WIA Annual report is
       submitted.

If no evaluations were started and/or completed during the program year for which the
WIA Annual Report is submitted, then states should either cite the waiver that exempted
them from the requirement to conduct these evaluations or submit the following
information for each planned evaluation:

   • An expected timeline for starting and completing the evaluation; and
   • Questions the evaluation is expected to address.

These evaluation studies, conducted under WIA Title 1B, are to promote, establish, and
implement methods for continuous improvement in the efficiency and effectiveness of the
statewide workforce investment system in improving employability for job seekers and
competitiveness for employers. These evaluation studies identify best practices or replicable
models and tools, as well as challenges and potential workforce solutions. This information
would be helpful to the public workforce system and may inform ETA’s national evaluation
and research agenda. As such, states are encouraged to share a copy of each final evaluation
report with ETA via their regional representative. ETA will share relevant best practices,
lessons learned, and other resources with the public workforce system through its online
technical assistance (TA) platform (www.Workforce3One.org) and other appropriate TA
events.

Costs of Workforce Investment Activities

States should explain how the mix of services for adults, dislocated workers, and youth
affected the outcomes. For adults and dislocated workers, the activities that states may
wish to address are core, intensive, and training services. For youth activities, states may
wish to include information about front-end costs (e.g., intake, assessment and case
management) and aggregated direct service costs for the 10 youth program elements
described in WIA Section 129(c)(2).

ETA has a long-term interest in improving program productivity to better serve both
customers and taxpayers. Prior WIA State Annual Reports describe a variety of methods for
calculating and presenting cost-related activities and measures. Among the most common
cost calculations reported by the states is the “cost per participant” indicator.

To facilitate further discussion about alternative cost measures (e.g., costs per outcome),
ETA developed examples of potential alternative efficiency measures (see Attachment).
The attachment provides states with optional measures to assist in describing and
analyzing the overall health of the workforce program using cost data.

In addition to the required components of the WIA Annual Report Narrative, ETA
encourages each state to include the following information in its narrative:

   A. Information from their strategic plans that highlights innovative service delivery
      strategies, including program activities that support dislocated workers, low-
      skilled/low-income adults and disadvantaged youth, the outcomes expected, as well
      as, the actual outcomes for their major customer populations. States may indicate
      actual federal outlays for selected activities, if such information is available.

   B. A discussion of the state’s unique programs and recent accomplishments. States
      can use the report to describe these accomplishments in the most advantageous
      manner to all stakeholders and partners, including Congress, governors, state
      legislators and workforce investment boards. States may want to highlight
      “success stories” that focus attention on how programs have been successful for
      participants, employers, and communities alike.

   C. Messages from the governor or other contextual information about state workforce
      investment board members, market analysis, strategies for improvement, and
      effects on major industries may also be included.

   D. A discussion of the activities funded by the state’s discretionary (“5 percent”)
      funds reserved subject to section 128(a) of WIA. In this section of the narrative
      report, states may describe activities undertaken in whole or in part with their
      discretionary funds, and how those activities directly or indirectly affect
      performance.

   E. A discussion of programs and strategies for serving employers at the state and local
      level, including the performance metrics used by states or local areas to measure

           the effectiveness of such services and current available performance data. States
           also may wish to talk about effects on major industries.

       F. A discussion of the initiatives and activities outlined in the WIA and Wagner-Peyser
          Act State Strategic Plan to improve performance.

       G. A discussion of the programs, initiatives, and strategies for serving veterans at the
          state and local level, including the performance metrics used by states or local areas
          to measure the effectiveness of such services and current available performance data.
          Include a description of the implementation of veterans’ priority of service for all
          training programs funded by the Department. Please describe how the state has
          implemented the Gold Card Initiative, to provide intensive services to post 9/11
          veterans, including how the 6 month follow-up has been implemented.

5. Due Date. The WIA Annual Report Narrative is usually due by October 1 following each
   program year. However, to give states sufficient time to develop its narrative, the report for
   PY 2013 will be due November 14, 2014. The complete WIA Annual Report, which
   includes the required performance reports and the narrative, will reflect accurate performance
   outcome information that is available by the time the WIA Annual Report for the program
   year is due. Please note that the WIA annual performance data outcome portion of the report,
   ETA Form 9091, was due on September 15, 2014. Failure to submit the performance
   progress reports by the deadline may lead to incentive grant ineligibility for a state.

6. Submission. An electronic copy of the WIA Annual Report Narrative should be e-mailed to
   [email protected] with a copy to the state’s respective ETA Regional Administrator by
   November 14, 2014. In an effort to be environmentally conscious, hard copies are no longer
   required. ETA will publish each state’s report on the Internet at
   www.doleta.gov/performance. Unlike previous years’ submissions, ETA will now only
   accept 508 compliant PDF formats. Since we will post each state’s report on the
   Performance Web site, all states must submit their WIA Annual Report electronically in a
   machine-readable format to comply with requirements set forth in Section 508 of the
   Rehabilitation Act.

7. Action Requested. States and grantees should distribute this TEGL to those personnel
   responsible for developing the WIA Annual Report Narrative, including personnel
   responsible for performance reporting, and to all local areas responsible for administering the
   WIA programs.

8. Inquiries. Please direct questions concerning this TEGL to your appropriate regional office.

9. Attachment. Overview of Potential Alternative Efficiency Measures for Consideration.

Attachment I: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2014/TEGL-9-14_Attachment-I_Acc.pdf

                                         ATTACHMENT

         Overview of Potential Alternative Efficiency Measures for Consideration

Examples of costs in relation to participant services and outcomes

(1) Unit Costs = total cost by service / total participation by service.

            Expenditures            Participation                        Unit Costs
       Core Intensive Training Core Intensive Training             Core Intensive Training
        $       $        $      #       #         #                 $       $        $

     Pros:
        • Applicable to most programs.
        • Easier to understand how costs apply to participant services.
     Cons:
        • Tracking program services and costs by year requires great effort and attention to
           detail; it therefore would be more susceptible to human error.
        • It is of limited use in assessing program effectiveness, because it is not an outcome-
           based measure.

(2) Cost per Participant (CP) = This measure is calculated by taking the total program costs in
     terms of expenditures and dividing by the number of participants served during the year by
     the particular program.

               CP = All Program Expenditures / All Program Participants

     Pros:
        • Applicable to most programs.
        • Data is readily available.
        • Easy to understand.
        • Can be immediately generated each year.
        • Not costly or burdensome.
     Cons:
        • It is of limited use in assessing program effectiveness, because it is not an outcome-
           based measure.

(3) Cost per Exiter (CE) = It is calculated by taking total program costs in terms of
     expenditures and dividing by the number of exiters terminating the program during the year
     by the particular program.

               CE = Total Program Expenditures / Total Exiters Terminating Program
     Pros:
        • Applicable to most programs.
        • Data is readily available.

        • Easy to understand.
        • Can be immediately generated each year.
        • Not costly or burdensome.
     Cons:
        • It is of limited use in assessing program effectiveness, because it is not an outcome-
           based measure.

(4) Cost per Entered Employment (CEE) = This measure is calculated by taking total program
     costs in terms of expenditures and dividing by the number of exiters entering employment
     in the first quarter following exit from the particular program.

              CEE = Total Program Costs / First Quarter Exiters Entering Employment
     Pros:
        •    Applicable to most programs.
        •    Data is readily available.
        •    Easy to understand.
        •    Can be generated about two quarters after the end of each program year.
        •    Not costly or burdensome.
        •    An outcome-based efficiency measure. Therefore, it is of substantial use in
             understanding program effectiveness.
     Cons:
        • Does not capture those who entered employment in the same quarter of exit.
        • Puts a premium on quick labor exchange at a time we are trying to improve skills.

(5) Cost per Retained Employment (CRE) = This efficiency measure is calculated by taking
     total program costs in terms of expenditures and dividing by the number of exiters who are
     employed in both the second and third quarters after the exit quarter.

              CRE = Total Program Costs / Exiters Employed in Q2 & Q3 after Exit

     Pros:
        •    Potentially applicable to most programs.
        •    Data is readily available.
        •    Relatively easy to understand.
        •    Relatively low cost and low burden to produce.
        •    An outcome-based efficiency measure. Therefore, it is of substantial use to
             understanding program effectiveness and costs.
     Cons:
        • Lengthier lags in data (must wait for several quarters after the end of the program
           year).

(6) Cost per $1,000 Increase in Earnings (CIE) = Total program cost divided by total earnings
     change from 2nd and 3rd pre-program quarters to 2nd and 3rd post-program quarters for
     participants or exiters.

               CIE = Total Program Costs / Total Participant or Exiters Earnings Change from
               2nd and 3rd pre-program quarters

     Pros:
        •    Potentially applicable to most programs.
        •    Data is readily available.
        •    Relatively low cost and low burden to produce.
        •    An outcome-based efficiency measure. Therefore, it is of substantial use to
             understanding program effectiveness and costs.
     Cons:
        • Somewhat difficult to understand.
        • Lengthier lags in data (must wait several quarters after the end of the program year).

(7) Cost per $1,000 in Post-Program Earnings (CPPE) = Total program cost divided by total
     earnings in 2nd and 3rd post-program quarters for participants of exiters multiplied by
     $1,000.

               CPPE = Total Program Costs / Total Participant or Exiters Earnings in 2nd and 3rd
               post-Program quarters multiplied by $1,000

     Pros:
        •    Potentially applicable to most programs.
        •    Data is readily available.
        •    Relatively low cost and low burden to produce.
        •    It is outcome-based.
        •    Unlike the prior measure, does not weight prior employment earnings against post
             program earnings.
     Cons:
        • Lengthier lags in data (must wait several quarters after the end of program year).
        • Somewhat difficult to understand.

(8) Cost per Exiter or Participant Receiving a Particular Service (CPS) = Total program cost
     of a particular service divided by the number of exiters or participants receiving a particular
     service.

             CPS = Total Cost of Particular Program / Participants or Exiters Who Received
             Particular Service

     Pros:
        • Easy to understand.
        • No lags in data. Data can be immediately generated at the end of each year.

     Cons:
        • Only applicable to programs that distinguish types of service.
        • Data is readily available for some programs, but not all.

         •   Is not an outcome-based efficiency measure.
         •   May be burdensome to generate.

(9) Cost per Placement in Employment or Education (CPEE) = Total program cost divided
     by the number of participants or exiters in employment or enrolled in post-secondary
     education and/or advanced training or advanced training occupational skills in the 1st
     quarter after exit.

             CPEE = Total Program Costs / Number of Exiters or Participants Employed or in
             Post-Secondary Education Programs after 1st Quarter Exit

     Pros:
        • The data is relatively easy to understand.
        • Relatively low cost and low burden to produce.
        • The measure is outcome-based so it is of substantial use to understanding program
           effectiveness.
     Cons:
        • Limited to primarily the Workforce Investment Act Youth program.

(10) Cost per Individual Attaining a Recognized Degree or Certificate (CID) (Credentials
     include but are not limited to, a high school diploma, GED, or other recognized
     equivalents, post-secondary degrees/certificates, recognized skill standards, and licensure
     or industry-recognized certificates.) = Total training program cost divided by the number of
     participants or exiters receiving a training service attaining a recognized credential during
     participation or by the end of the 3rd quarter after exit.

             CID = Total Training Program Costs / Number of Participants or Exiters who
             Attained Certification or Degree by the end of 3rd Quarter after exit

     Pros:
        • The measure is an outcome-based measure, so it is of substantial use in
           understanding program effectiveness.
     Cons:
        • Only applicable to programs that provide services and identify individuals as
           receiving training and types of credentialing.
        • Data is readily available for some programs, but not all.
        • The measure is somewhat difficult to understand.
        • Potentially lengthy lags in data.

(11) Return on Investment (ROI). ROI is a way of summarizing how large the gain is on an
     investment, such as workforce development. In its simplest form, ROI is calculated by
     dividing the gain by the size of the investment. This equation can be written as B/C, where
     B is the sum of all benefits that result from the investment over the period considered and C
     represents the costs. For a workforce program, one would divide the increase in earnings
     due to the program by the cost of the program. In more sophisticated analyses, ROI
     calculations take into account the timing of the gains due to the program. Economists

typically compute a variation called the internal rate of return (IRR), which is based on the
costs and benefits over the life of the investment. The IRR can be calculated, using a
financial calculator or a spreadsheet, by solving the following equation for i:

0 = -C0 + (B1 –C1)/(1+i) + (B2 –C2)/(1+i)2 + (B3 –C3)/(1+i)3 + …. + (BN –CN)/(1+i)N

Where Bt is the benefit received in year i, Ct is the cost incurred in year i, and N is the last
year that benefits or costs occur. (The four dots mean that the formula includes the same
type of term for all years between year 3 and year N.) The IRR is preferred to the simpler
versions of ROI because it takes into account the timing of the costs and benefits.

Pros:
   • Potentially applicable to most programs.
   • Measure is an impact-based efficiency measure, which controls for factors that
      could potentially influence/bias results. Therefore, it is of the greatest utility in
      understanding program cost-effectiveness.
   • This measure controls for difficulty or cost of serving different populations (e.g.,
      hard-to-serve, service mix, and economic conditions).
Cons:
   • Data is very costly to produce.
   • The measure is difficult to understand.
   • Lengthy lags in data.