Program Year 2014 WIA annual report narrative

CLASSIFICATION
   EMPLOYMENT AND TRAINING ADMINISTRATION                 WIA/Performance Reporting System
               ADVISORY SYSTEM                            CORRESPONDENCE SYMBOL
          U.S. DEPARTMENT OF LABOR                        OPDR
             Washington, D.C. 20210                       DATE
                                                          November 4, 2015

ADVISORY:         TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 7-15

TO:               STATE WORKFORCE AGENCIES
                  STATE WORKFORCE ADMINISTRATORS
                  STATE WORKFORCE LIAISONS

FROM:             GERRI FIALA /s/ for PORTIA WU
                  Assistant Secretary

SUBJECT:          Workforce Investment Act (WIA) Program Year (PY) 2014 Annual Report
                  Narrative

1. Purpose. To provide guidance to the states regarding the content of the WIA Annual
   Report narrative; including strategies for serving veterans, acceptable approaches to
   fulfilling the customer satisfaction requirements, and the procedures for submission to the
   Employment and Training Administration (ETA). The Annual Report is due on Tuesday,
   December 15, 2015.

2. References.
   • Workforce Investment Act, Sections 134, 136 and 185;
   • 20 CFR 667.300;
   • Workforce Investment Act Annual Report: General Reporting Instructions and ETA
      Form 9091, Revised 2012 (Office of Management and Budget (OMB) Control No. 1205-
      0420, expires 5/31/2016);
   • Training and Employment Guidance Letter (TEGL) 17-05 and TEGL 17-05, Change 2,
      Common Measures Policy for the Employment and Training Administration’s
      Performance Accountability System and Related Performance Issues;
   • TEGL 09-07, Revised Incentive and Sanction Policy for Workforce Investment Act Title
      IB Programs;
   • TEGL 6-14, Program Year (PY) 2013/Fiscal Year (FY) 2014 Data Validation and
      Performance Reporting Requirements and Associated Timelines;
   • TEGL 9-14, Workforce Investment Act (WIA) Program Year (PY) 2013 Annual Report
      Narrative;
   • OMB Circular No. A-94, Revised Guidelines and Discount Rates for Benefit-Cost
      Analysis of Federal Programs.

 RESCISSIONS                                              EXPIRATION DATE
 None                                                     Continuing

3. Background. Each state that receives an allotment under WIA Section 127 (Youth
   activities) or Section 132 (Adult and Dislocated Worker activities) must prepare and submit a
   WIA Annual Report of performance progress to the Secretary of Labor in accordance with
   WIA Sections 136 and 185.

   There are two components to the WIA Annual Report: (1) the required performance results,
   as specified in ETA Form 9091—WIA Title 1B Annual Report Form (OMB No. 1205-0420)
   found at:
   http://www.doleta.gov/performance/guidance/WIA/WIAAnnualReportSpecifications.pdf,
   and (2) a narrative report. This guidance focuses on the narrative report and what states
   should address in this report.

   The performance accountability requirements prescribed under the Workforce Innovation
   and Opportunity Act (WIOA) begin July 1, 2016. States are expected to submit a WIA
   annual report following the close of PY 2015 (June 30, 2016). States should continue to
   collect the required Workforce Investment Act Standardized Record Data elements through
   that time. Further guidance will be forthcoming as to the reporting transition from WIA to
   WIOA.

4. WIA Annual Report Narrative. The required portions of the WIA Annual Report
   Narrative include the information required by WIA sections 136(d)(1) and (2) and
   185(d). This includes:

       •   Performance data on the core and customer satisfaction measures, including progress
           of local areas in the state in achieving local performance measures;
       •   Information on the status of state evaluation activities;
       •   Information on the cost of workforce investment activities relative to the effect of the
           activities on the performance of participants;
       •   Assurance that all required elements are reported uniformly so that a state-by-state
           comparison can be made;
       •   Information on participants in the workforce investment system (this information is
           also included in the performance results portion of the WIA Annual Report); and
       •   A listing of the waivers for which the state has received approval, information on
           how the waivers have changed the activities of the state and local areas, and how
           activities carried out under the waivers have directly or indirectly affected state
           and local area performance outcomes. To the extent possible, states should
           discuss whether waiver implementation and outcomes advance the President’s
           Job-Driven Elements.

   Additional information regarding the content of the required components is provided below.

   Customer Satisfaction Measures

   The public workforce system established under WIA is intended to serve a dual-customer
   base consisting of job seekers and employers. Recognizing the importance of measuring
   service quality, WIA (Section 136(b)(2)(B)) requires states to measure customer satisfaction

                                                 2

for employers and participants of state and local agencies that provide employment and
training services.

In PY 2005, ETA began approving a waiver to support adoption of the common measures,
commonly referred to as the common measure waiver. States with an approved common
measure waiver must continue to collect customer satisfaction data but only report on
common performance measure outcomes. Instead, states with the common measure waiver
must provide information about their customer service results in the narrative portion of their
WIA Annual Report to the Department of Labor (Department). Currently all but four states
(Michigan, Minnesota, Rhode Island, and Vermont) and Puerto Rico have this approved
waiver for PY 2014 resulting in significantly limited access to customer satisfaction data at
the national level.

In TEGL 9-14, Workforce Investment Act (WIA) Program Year (PY) 2013 Annual Report
Narrative, ETA was interested in what approaches and methodologies states were using to
collect customer satisfaction information. In the PY 2013 WIA Annual Reports states
identified the approaches and methodologies they were using; including descriptions of how
the information was being used to improve customer service. Other states mentioned they
were in the process of developing new customer satisfaction measures. Many states and
local areas have begun piloting new and potentially less costly methods to collect customer
satisfaction data other than the phone surveys required previously when using the American
Customer Satisfaction Index (ACSI), which was the common methodology used to capture
and report customer satisfaction information prior to PY 2012.

In PY 2013, the Department provided states with flexibility in collection of their customer
satisfaction information. States identifying new measures in PY 2013 should provide results
information in their PY 2014 report. Although ACSI is no longer contracted with ETA,
states may elect to use ACSI methodology at their own cost. States utilizing ACSI for
customer satisfaction may still report the results in the ETA Form 9091. For states not
utilizing ACSI, they are encouraged to use robust methodologies that use multiple strategies
for data collection (telephone, email, paper surveys, or other technology methods). All states
must describe their customer service methodologies in their annual narrative. At a minimum,
quality customer satisfaction narratives shall include:

    1. The approach used including whether the approach used a random sample (if possible
      include a sample of the survey);
    2. The number of individuals/employers that were provided customer satisfaction
      outreach;
    3. The response rate;
    4. A summary of the results and whether the results are generalizable to the entire
      population of customers; and
    5. Any processes for incorporating the customer satisfaction feedback

This approach provides states with the flexibility they have requested to use new
technologies which may lead to higher quality services while also providing the Department
with better insight into each state’s customer satisfaction initiatives. ETA believes that
customer service will be enhanced through this increased flexibility and by requiring all

                                             3

states to follow, at a minimum, the above parameters for structuring their description of
customer satisfaction activities. Additionally, this strategy for capturing customer
satisfaction provides a more robust national understanding of state level activities.

Status of State Evaluation Activities

As WIA Section 134(a)(2)(B)(ii) notes, conducting evaluations of workforce investment
activities under WIA section 136(e) is a required statewide activity. States should include
information about all evaluation studies that were started and/or completed during the
program year for which the WIA Annual Report is being submitted. For each evaluation,
the Annual Report Narrative should include:

   • The timeline for starting and completing the evaluation;
   • The questions the evaluation did/will address;
   • A description of the evaluation’s methodology, including description of any control
         or comparison group and description of the analysis technique employed;
   •     The timeline for the final report and other deliverables; and,
   •     Summary of evaluation findings, including summary of best practices, for those
         evaluations completed during the program year for which the WIA Annual report is
         being submitted.

These State-sponsored evaluation studies, conducted under WIA Title 1B, are expected to
promote, establish, and implement methods for continuous improvement in the efficiency and
effectiveness of the statewide workforce investment system in improving employability for
job seekers and competitiveness for employers. The decision to undertake evaluations and
research studies should be guided by the information needs of the state and informed by the
Department’s WIA Five-Year Research and Evaluation Strategic Plan for 2012-2017
(Research Plan). The current Research Plan, prepared in consultation with members of the
public workforce system, identifies the following high priority research topics:

   (1)    Understanding Changing Labor Markets;
   (2)    Identifying Effective Strategies;
   (3)    Improving Workforce System Infrastructure;
   (4)    Addressing the Needs of Special Populations; and
   (5)    Building Research Infrastructure and Support.

The Research Plan is available at http://www.doleta.gov/reports/fiveyear_researchplan.cfm.

Costs of Workforce Investment Activities

States should explain how the mix of services for adults, dislocated workers, and youth
activities affected the outcomes. For adults and dislocated workers, the activities that
states may wish to address are core, intensive, and training services. For youth activities,
states may wish to include information about front-end costs (e.g., intake, assessment and
case management) and aggregated direct service costs for the 10 youth program elements
described in WIA Section 129(c)(2).

                                             4

   ETA has a long-term interest in improving program cost-effectiveness so that both taxpayers
   and customers can be better served. A variety of methods for calculating and presenting
   cost information and measures are described in the attachment of this document. Among the
   most common cost calculations reported by the states is the “cost per participant” indicator,
   which will be a required report element beginning in PY 2016, under the Workforce
   Innovation and Opportunity Act (WIOA).

   In addition to the required components of the WIA Annual Report Narrative, ETA
   encourages states to include the following information in their narrative:

       A. Information from their strategic plans that highlights innovative service delivery
          strategies, including program activities that support dislocated workers, low-
          skilled/low-income adults and disadvantaged youth, the outcomes expected, as well
          as, the actual outcomes for their major customer populations. States may indicate
          actual federal outlays for selected activities, if such information is available.

       B. A discussion of “best practices” and “lessons learned” that focuses on specific
          participants, employers, and communities.

       C. Messages from the governor or other contextual information about state workforce
          investment board members, market analysis, strategies for improvement, and
          effects on major industries may also be included.

       D. A discussion of the activities funded by the state’s discretionary (“8.75 percent”)
          funds. In this section of the narrative report, states may describe activities
          undertaken in whole or in part with their discretionary funds, and how those
          activities directly or indirectly affect performance.

       E. A discussion of programs and strategies for serving employers at the state and local
          level, including the performance metrics used by states or local areas to measure
          the effectiveness of such services and current available performance data. Effects
          on major industries may also be included.

       F. A discussion of the initiatives and activities outlined in the WIA and Wagner-Peyser
          Act State Strategic Plan to improve performance.

       G. A discussion of the programs, initiatives, and strategies for serving veterans at the
          state and local level, including, the performance metrics used by states or local areas
          to measure the effectiveness of such services and current available performance data.
          Include a description of how veterans’ priority of service is being implemented for all
          Department training programs.

5. Due Date. The WIA Annual Report narrative is usually due no later than October 1st
   following each program year. This year, the report for PY 2014 will be due Tuesday,

                                                5

   December 15, 2015. The WIA Annual Report narrative will reflect performance outcome
   information through June 30, 2015.

6. Submission. An electronic copy of the WIA Annual Report narrative should be e-mailed to
   [email protected] by December 15, 2015. States should also submit an electronic copy to
   their respective ETA Regional Administrator and copy their Federal Project Officer. Hard
   copies of the report may be submitted but are no longer required, in an effort to be more
   environmentally friendly. ETA will publish each state’s report on the Internet at
   www.doleta.gov/performance. As was the case for last year’s submission, ETA will now
   only accept 508 compliant PDF formats. Since each state’s report will be posted on ETA’s
   Performance Web site, all WIA Annual Report Narratives must be submitted electronically
   in a machine readable format to comply with requirements set forth in Section 508 of the
   Rehabilitation Act.

7. Action Requested. Distribute this TEGL to those personnel responsible for developing the
   WIA Annual Report narrative, including personnel responsible for performance reporting,
   and to all local areas responsible for administering the WIA programs.

8. Inquiries. Please direct questions concerning this TEGL to your appropriate Regional
   Office.

9. Attachment. Overview of Potential Alternative Efficiency Measures for Consideration

                                              6

Attachment 1

Official attachment: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2015/TEGL_07-15_Attachment_I.pdf

ATTACHMENT

         Overview of Potential Alternative Efficiency Measures for Consideration

Examples of costs in relation to participant services and outcomes

(1) Unit Costs = total cost by service / total participation by service.

            Expenditures            Participation                        Unit Costs
       Core Intensive Training Core Intensive Training             Core Intensive Training
        $       $        $      #       #         #                 $       $        $

       Pros:
          • Applicable to most programs.
          • Easier to understand how costs apply to participant services.
       Cons:
          • Tracking program services and costs by year requires great effort and attention to
             detail; it therefore would be more susceptible to human error.
          • Limited use in assessing program effectiveness, because it is not an outcome-
             based measure.

(2) Cost per Participant (CP) = This measure is calculated by taking the total program costs in
     terms of expenditures and dividing by the number of participants served during the year by
     the particular program.

                                 All Program Expenditures
                        CP =
                                 All Program Participants

     Pros:
        • Applicable to most programs.
        • Data is readily available.
        • Easy to understand.
        • Can be immediately generated each year.
        • Not costly or burdensome.
     Cons:
        • Limited use in assessing program effectiveness, because it is not an outcome-based
           measure.

(3) Cost per Exiter (CE) = It is calculated by taking total program costs in terms of
     expenditures and dividing by the number of exiters terminating the program during the year
     by the particular program.

                                     Total Program Expenditures
                        CE =
                                  Total Exiters Terminating Program

                                                  1

     Pros:
        • Applicable to most programs.
        • Data is readily available.
        • Easy to understand.
        • Can be immediately generated each year.
        • Not costly or burdensome.
     Cons:
        • Limited use in assessing program effectiveness, because it is not an outcome-based
           measure.

(4) Cost per Entered Employment (CEE) = This measure is calculated by taking total program
     costs in terms of expenditures and dividing by the number of exiters entering employment
     in the first quarter following exit from the particular program.

                                                Total Program Costs
                       CEE =
                                     First Quarter Exiters Entering Employment

     Pros:
        •    Applicable to most programs.
        •    Data is readily available.
        •    Easy to understand.
        •    Can be generated about two quarters after the end of each program year.
        •    Not costly or burdensome.
        •    Measure is an outcome-based efficiency measure. Therefore, it is of substantial use
             in understanding program effectiveness.
     Cons:
        • Does not capture those who entered employment in the same quarter of exit.
        • Puts a premium on quick labor exchange at a time we are trying to improve skills.

(5) Cost per Retained Employment (CRE) = This efficiency measure is calculated by taking
     total program costs in terms of expenditures and dividing by the number of exiters who are
     employed in both the second and third quarters after the exit quarter.

                                                Total Program Costs
                       CRE =
                                       Exiters Employed in Q2 & Q3 after Exit

     Pros:
        •    Potentially applicable to most programs.
        •    Data is readily available.
        •    Relatively easy to understand.
        •    Relatively low cost and low burden to produce.
        •    It is an outcome-based efficiency measure. Therefore, it is of substantial use to
             understanding program effectiveness and costs.

                                                2

     Cons:
        • Lengthier lags in data (must wait for several quarters after the end of the program
           year).

(6) Cost per Exiter or Participant Receiving a Particular Service (CPS) = Total program cost
     of a particular service divided by the number of exiters or participants receiving a particular
     service.

                                               Total Cost of Particular Program
                        CPS =
                                   Participants or Exiters Who Received Particular Service

     Pros:
        • Easy to understand.
        • No lags in data. Data can be immediately generated at the end of each year.
     Cons:
        • Only applicable to programs that distinguish types of service.
        • Data is readily available for some programs, but not all.
        • Is not an outcome-based efficiency measure.
        • May be burdensome to generate.

(7) Cost per Placement in Employment or Education (CPEE) = Total program cost divided
     by the number of participants or exiters in employment or enrolled in postsecondary
     education and/or advanced training or advanced training occupational skills in the 1st
     quarter after exit.

                                                    Total Program Costs
                        CPEE =
                                       Number of Exiters or Participants Employed or in
                                    Postsecondary Education Programs after 1st Quarter Exit

     Pros:
        • The data is relatively easy to understand.
        • Relatively low cost and low burden to produce.
        • The measure is outcome-based so it is of substantial use to understanding program
           effectiveness.
     Cons:
        • Limited to primarily the Workforce Investment Act Youth program.

(8) Cost per Individual Attaining a Recognized Degree or Certificate (CID) = Total training
     program cost divided by the number of participants or exiters receiving a training service
     attaining a recognized credential during participation or by the end of the 3rd quarter after
     exit. (Credentials include but are not limited to, a high school diploma, GED, or other
     recognized equivalents, post-secondary degrees/certificates, recognized skill standards, and
     licensure or industry-recognized certificates.)

                                                 3

                                                     Total Training Program Costs
                        CID =
                                           Number of Participants or Exiters who Attained
                                      Certification or Degree by the end of 3rd Quarter after exit

     Pros:
        • The measure is an outcome-based measure, so it is of substantial use in
           understanding program effectiveness.
     Cons:
        • Only applicable to programs that provide services and identify individuals as
           receiving training and types of credentialing.
        • Data is readily available for some programs, but not all.
        • The measure is somewhat difficult to understand.
        • Potentially lengthy lags in data.

(9) Return on Investment (ROI). ROI is a way of quantifying the gain on an investment, such
     as workforce development, actually is. In its simplest form, ROI is calculated by dividing
     the gain by the size of the investment. This equation can be written as B/C, where B is the
     sum of all benefits that result from the investment over the period considered and C
     represents the costs. For a workforce program, one would divide the increase in participant
     earnings by the cost of the program. In more sophisticated analyses, ROI calculations take
     into account the timing of the gains due to the program. Economists typically compute a
     variation called the internal rate of return (IRR), which is based on the costs and benefits
     over the life of the investment. The IRR can be calculated, using a financial calculator or a
     spreadsheet, by solving the following equation for i:

     0 = -C0 + (B1 –C1)/(1+i) + (B2 –C2)/(1+i)2 + (B3 –C3)/(1+i)3 + …. + (BN –CN)/(1+i)N

     Where Bt is the benefit received in year i, Ct is the cost incurred in year i, and N is the last
     year that benefits or costs occur. (The four dots mean that the formula includes the same
     type of term for all years between year 3 and year N.) The IRR is preferred to the simpler
     versions of ROI because it takes into account the timing of the costs and benefits.

     Pros:
        • Potentially applicable to most programs.
        • Measure is an impact-based efficiency measure, which controls for factors that
           could potentially influence/bias results. Therefore, it is of the greatest utility in
           understanding program cost-effectiveness.
        • This measure controls for difficulty or cost of serving different populations (e.g.,
           hard-to-serve, service mix, and economic conditions).
     Cons:
        • Data is very costly to produce.
        • The measure is difficult to understand.
        • Lengthy lags in data.

                                                  4