Program year 2012 WIA annual report narrative

Accessible letter PDF: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2013/TEGL_6_13_Acc.pdf

CLASSIFICATION
  EMPLOYMENT AND TRAINING ADMINISTRATION               WIA/Performance Reporting System
              ADVISORY SYSTEM                          CORRESPONDENCE SYMBOL
         U.S. DEPARTMENT OF LABOR                      OPDR
            Washington, D.C. 20210                     DATE
                                                       October 18, 2013
ADVISORY:          TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 6-13

TO:                STATE WORKFORCE AGENCIES
                   STATE WORKFORCE ADMINISTRATORS
                   STATE WORKFORCE LIAISONS

FROM:              ERIC M. SELEZNOW /s/
                   Acting Assistant Secretary
SUBJECT:           Workforce Investment Act (WIA) Program Year (PY) 2012 Annual Report
                   Narrative

    1. Purpose. To provide guidance to the states regarding the content of the WIA Annual
     Report Narrative including: the procedures for submission of the report to the
     Employment and Training Administration (ETA); acceptable approaches to fulfilling
     the customer satisfaction requirements for PY 2012 and PY 2013; and narrative about
     strategies for serving veterans (as referenced in section 4, part F). The WIA Annual
     Report Narrative is due on Friday, November 15, 2013.

    2. References.

        •   Workforce Investment Act, Sections 134, 136, and 185;
        •   20 Code of Federal Regulations Section 667.300;
        •   Training and Employment Guidance Letter (TEGL) 09-11, Guidance on Revised
            Fiscal Year 2012 “Advance Funding Levels Available October 2011 for
            Workforce Investment Act Programs and Clarification on the Level of
            Governor’s Reserve;
        •   TEGL 21-11, Instructions for Submitting Workforce Investment Act and
            Wagner-Peyser Act State Plans for Program Year 2012;
        •   TEGL 28-11, Change 1, Program Year 2011/Fiscal Year 2012 Performance
            Reporting and Data Validation Timeline;
        •   TEGL 29-11, Guidance on the Workforce Investment Act Annual Report
            Narrative;
        •   Training and Employment Notice (TEN) 31-11, The Rapid Response Framework;
        •   TEN 32-11, The Rapid Response Self-Assessment Tool;
        •   TEGL 17-05 and TEGL 17-05, Change 2, Common Measures Policy for the
             Employment and Training Administration’s Performance Accountability System
            and Related Performance Issues;
        •   TEGL 09-07, Revised Incentive and Sanction Policy for Workforce Investment Act
            Title IB Programs; and

 RESCISSIONS                                          EXPIRATION DATE
 TEGL 29-11                                           Continuing

          •   Office of Management and Budget (OMB) Circular No. A-94, Revised Guidelines
              and Discount Rates for Benefit-Cost Analysis of Federal Programs.

3. Background. Each state that receives an allotment under WIA Section 127 (Youth activities)
or Section 132 (Adult and Dislocated Worker activities) must prepare and submit an Annual
Report of performance progress to the Secretary of Labor in accordance with WIA Sections 136
and 185.

There are two components to the WIA Annual Report: (1) the required data performance results,
as specified in form ETA 9091—the WIA Title 1B Annual Report (OMB No. 1205-0420); and
(2) a narrative report. This guidance focuses on the narrative report and what states should
address in this report.

4. WIA Annual Report Narrative. The required portions of the Annual Report Narrative
include the information required by WIA sections 136(d) (1) through (2) and 185(d). This
includes:

   •   Performance data on the core and customer satisfaction measures, including progress of
       local areas in the state in achieving local performance measures;
   •   Information on the status of state evaluation activities;
   •   Information on the cost of workforce investment activities relative to the effect of the
       activities on the performance of participants;
   •   Assurance that all required elements are reported uniformly so that a state-by-state
       comparison can be made;
   •   Information on participants in the workforce investment system (this information is also
       included in the data performance results portion of the Annual Report); and
   •   A listing of the waivers for which the state has received approval, information on how the
       waivers have changed the activities of the state and local areas, and how activities carried
       out under the waivers have affected state and local area performance outcomes directly or
       indirectly. To access further guidance and information on the current ETA waiver policy,
       see www.doleta.gov/waivers.

Other Recommended Annual Report Narrative Components

ETA receives inquiries from its stakeholders, including members of Congress, the
Government Accountability, the Administration, and others, regarding states’ progress
against their strategic plan and the goals within the plan. The WIA Annual Report narrative
is one of the best ways for states to communicate to ETA and the workforce system
stakeholders the achievements of the past program year.

Additionally, ETA excerpts the success stories and best practices from these Annual Report
narratives to further showcase the accomplishments of the workforce system in assistance
individuals find work, and/or helping them return to work, as well as to assist employers find
skilled workers.


Therefore, in addition to the required components of the WIA Annual Report narrative, ETA
encourages states to include the following information in its narrative:

   A. A discussion of the state’s programs and recent accomplishments in implementing its
      five-year State Integrated or Unified Workforce Plan. This may include a discussion of
      the state’s achievements related to its cross-program strategies, partnerships, leveraging
      of resources, policy alignment, and program successes.

   B. Messages from the governor or other contextual information about state workforce
      investment board members, market analysis, strategies for improvement, and effects on
      major industries may also be included.

   C. A discussion of the activities funded by the state’s discretionary (“5 percent”) funds. In
      this section of the narrative report, states may describe activities undertaken in whole or
      in part with their discretionary funds, and how those activities directly or indirectly affect
      performance.

   D. A discussion of programs and strategies for engaging and serving employers at the state
      and local level, including the performance metrics used by states or local areas to
      measure the effectiveness of such services and current available performance data.
      Effects on major industries may also be included.

   E. A discussion of the programs, initiatives, and strategies for serving Veterans at the state
      and local level; including, the performance metrics used by states or local areas to
      measure the effectiveness of such services and current available performance data.
      Include a description of how the state implemented Veterans’ priority of service for all
      Department of Labor training programs. Also include a description of how the state has
      implemented the Gold Card Initiative, to provide intensive services to post 9/11 Veterans;
      including, how the six month follow-up has been implemented.

Customer Satisfaction Measures

WIA (Section 136(b)(2)(B)) requires states to measure customer satisfaction for employers and
participants of state and local agencies that provide employment and training services.

In PY 2005, ETA began approving a waiver to support adoption of the common measures,
commonly referred to as the Common Measure waiver. States with an approved common
measure waiver are required to report only the common performance measure outcomes. States
with Common Measures waivers are still required to collect customer satisfaction data and
provide information about their customer service results in the narrative portion of their WIA
Annual Report to the Department of Labor. In PY 2012 all but six states (Hawaii, Arizona,
Vermont, Rhode Island, Michigan, and Minnesota) and Puerto Rico have this waiver, resulting in
significantly limited access to customer satisfaction data at the national level.

Many states and local areas have expressed interest in piloting new and potentially less costly
methods to collect customer satisfaction than the phone surveys required previously when using


the American Customer Satisfaction Index (ACSI), which was the common methodology used to
capture and report customer satisfaction information prior to PY 2012. ETA is no longer
contracting with ACSI for the procurement of national weights used in performing the ACSI
calculations as of PY 2012.

For the PY 2012 WIA Annual Reports, states with an approved Common Measure waiver are
directed, similar to PY 2011 guidance, to provide performance data on the core and customer
satisfaction measures, including progress of local areas in the state in achieving local
performance measures as part of their narrative. States that do not have an approved Common
Measure waiver are directed to provide a brief overview of their plan to transition to the PY 2013
reporting requirements described below and are exempt from reporting data in the customer
satisfaction fields, Table A of the ETA form 9091. As described in TEGL 09-07, the omission of
customer satisfaction from form 9091 will have no impact on states’ eligibility for WIA
incentive grants.

Beginning in PY 2013, ETA is providing states with additional flexibility in the collection of
customer satisfaction and each state may select its own approach to capturing this information.
ACSI remains an acceptable methodology that states and locals may elect to use at their own
cost. States must use robust methodologies that use multiple strategies for data collection
(telephone, email, paper surveys, or other technology methods). All states will be required to
describe their customer service methodology as part of their annual narrative. Customer
satisfaction narratives must, at a minimum include:

   •   the approaches/ methodology used;
   •   the number of individuals/employers that were provided customer satisfaction outreach;
   •   the response rate; and
   •   a summary of the results

This new approach provides states with the flexibility they have requested to use new
technologies which may lead to better customer satisfaction data and subsequently, higher
quality services, while also providing the ETA with better insight into states’ customer
satisfaction initiatives. ETA believes that customer service is enhanced through increasing
flexibility and by requiring all states to follow, at a minimum, the above parameters for
structuring their description of customer satisfaction activities. Additionally, this new strategy
for capturing customer satisfaction will provide a more robust national understanding of state
level activities.

ETA recognizes that this transition may be especially challenging for states that currently do not
have the Common Measure waiver and will continue to provide technical assistance to such
states through the appropriate Regional Office. Additionally, ETA will provide additional
guidance if technical assistance efforts demonstrate such a need.

Status of State Evaluation Activities

As WIA section 134(a)(2)(B)(ii) notes, conducting evaluations of workforce investment
activities under WIA section 136(e) is a required statewide activity. States should include


information about all evaluation studies that were started and/or completed during the program
year for which the WIA Annual Report is submitted. For each evaluation, the narrative should
include:

   • The timeline for starting and completing the evaluation;
   • The questions the evaluation did/will address;
   • A description of the evaluation’s methodology, including description of any control or
       comparison group and description of the analysis technique employed;
   •   The timeline for the final report and other deliverables; and,
   •   A summary of evaluation findings, including summary of best practices, for those
       evaluations completed during PY 2012.

If no evaluations were started and/or completed during the PY for which the WIA Annual Report
is submitted, then states, except those with an approved waiver for conducting evaluations,
should submit the following information for each planned evaluation:

   • Expected timeline for starting and completing the evaluation; and
   • The questions the evaluation did/will address.

These evaluation studies, conducted under WIA Title 1B, are to promote, establish, and
implement methods for continuous improvement in the efficiency and effectiveness of the
statewide workforce investment system in improving employability for job seekers and
competitiveness for employers. These evaluation studies identify best practices or replicable
models and tools, as well as challenges and potential workforce solutions. This information will
be helpful to the public workforce system and may inform ETA’s national evaluation and
research agenda. As such, states are encouraged to share a copy of each final evaluation report
with ETA via their Regional Office. ETA will share relevant best practices, lessons learned,
and other resources with the public workforce system through its online technical assistance
(TA) platform (www.Workforce3One.org) and other appropriate TA platforms.

Types and Costs of Workforce Investment Activities

States should explain how the mix of services for adults, dislocated workers, and youth activities
affected the outcomes. For adults and dislocated workers, the activities that states may wish to
address are core, intensive, and training services. For youth activities, states may wish to include
information about front-end costs (e.g., intake, assessment, and case management) and
aggregated direct service costs for the 10 youth program elements described in WIA section
129(c)(2).

ETA has a long-term interest in improving program productivity to serve our customers better. A
variety of methods for calculating and presenting cost- related activities and measures have been
described in prior WIA State Annual Reports. Among the most common cost calculations
reported by the states is the “cost per participant” indicator.

To facilitate further discussion about alternative cost measures (e.g., costs per outcome), OMB
charged ETA with the responsibility of developing and implementing an outcome- based


measure for programs using the common performance measures. The attachment to this TEGL
offers examples of potential, alternative efficiency measures.

5. Due Date. The Annual Report is usually due no later than October 1 following each program
year. For Program Year 2012, however, the due date has been extended to November 15, 2013,
for the Annual Report narrative ONLY, to provide states sufficient time to fully describe its
accomplishments. The Annual Report, which includes the required performance data reports and
the narrative, will reflect accurate performance outcome information that is available by the time
the Annual Report for the program year is due. Usually failure to submit both the complete WIA
Annual Report Narrative and performance report by the deadline may lead to incentive grant
ineligibility for a state. For Program Year 2012, the data tables are still due by October 1, 2013;
only the deadline for the Annual Report narrative is extended.

6. Submission. An electronic copy of the WIA State Annual Report Narrative should be e -
mailed to [email protected] by November 15, 2013. States should also submit an electronic
copy to their respective ETA Regional Administrator. You may submit hard copies of the report
but are no longer required, in an effort to be more environmentally friendly. ETA will publish
each state’s report on the Internet at www.doleta.gov/performance. Acceptable formats include:
WordPerfect, Microsoft Word, PDF, and other commonly used formats.

7. Action Requested. Distribute this TEGL to those personnel responsible for developing the
WIA Annual Report Narrative, including personnel responsible for performance reporting, and
to all local areas responsible for administering the WIA programs.

8. Inquiries. Please direct questions concerning this TEGL to your appropriate Regional
Office.

9. Attachment. Overview of Potential Alternative Efficiency Measures for Consideration.

Attachment 1: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2013/TEGL_6_13_Attachment_%28Accessible%20PDF%29.pdf

ATTACHMENT

         Overview of Potential Alternative Efficiency Measures for Consideration

Examples of costs in relation to participant services and outcomes

(1) Unit Costs = total cost by service / total participation by service.

            Expenditures            Participation                        Unit Costs
       Core Intensive Training Core Intensive Training             Core Intensive Training
        $       $        $      #       #         #                 $       $        $

       Pros:
          • It is applicable to most programs.
          • It makes it easier to understand how costs apply to participant services.
       Cons:
          • Tracking program services and costs by year requires great effort and attention to
             detail; it therefore would be more susceptible to human error.
          • It is of limited use in assessing program effectiveness, because it is not an
             outcome-based measure.

(2) Cost per Participant (CP) = This measure is calculated by taking the total program costs in
    terms of expenditures and dividing by the number of participants served during the year by
    the particular program.

                       CP = All Program Expenditures
                            All Program Participants

     Pros:
        • It is applicable to most programs.
        • Data is readily available.
        • Easy to understand.
        • Can be immediately generated each year.
        • Not costly or burdensome.
     Cons:
        • It is of limited use in assessing program effectiveness, because it is not an outcome-
           based measure.

(3) Cost per Exiter (CE) = It is calculated by taking total program costs in terms of
    expenditures and dividing by the number of exiters terminating the program during the year
    by the particular program.

                       CE = Total Program Expenditures
                            Total Exiters Termination Program


     Pros:
        • It is applicable to most programs.
        • Data is readily available.
        • Easy to understand.
        • Can be immediately generated each year.
        • Not costly or burdensome.
     Cons:
        • It is of limited use in assessing program effectiveness, because it is not an outcome-
           based measure.

(4) Cost per Entered Employment (CEE) = This measure is calculated by taking total program
    costs in terms of expenditures and dividing by the number of exiters entering employment in
    the first quarter following exit from the particular program.

                      CEE = Total Program Costs
                           First Quarter Exiters Entering Employment
     Pros:
        •    It is applicable to most programs.
        •    Data is readily available.
        •    Easy to understand.
        •    Can be generated about two quarters after the end of each program year.
        •    Not costly or burdensome.
        •    Measure is an outcome-based efficiency measure. Therefore, it is of substantial use
             in understanding program effectiveness.
     Cons:
        • Does not capture those who entered employment in the same quarter of exit.
        • Puts a premium on quick labor exchange at a time we are trying to improve skills.

(5) Cost per Retained Employment (CRE) = This efficiency measure is calculated by taking
    total program costs in terms of expenditures and dividing by the number of exiters who are
    employed in both the second and third quarters after the exit quarter.

                      CRE = Total Program Costs
                            Exiters Employed in Q2 & Q3 after Exit

     Pros:
        •    Potentially applicable to most programs.
        •    Data is readily available.
        •    Relatively easy to understand.
        •    Relatively low cost and low burden to produce.
        •    It is an outcome-based efficiency measure. Therefore, it is of substantial use to
             understanding program effectiveness and costs.


     Cons:
        • Lengthier lags in data (must wait for several quarters after the end of the program
           year).


(6) Cost per $1,000 Increase in Earnings (CIE) = Total program cost divided by total earnings
    change from 2nd and 3rd pre-program quarters to 2nd and 3rd post-program quarters for
    participants or exiters.

                      CIE = Total Program Costs
                            Total Participant or Exiters Earnings Change from 2nd and
                            3rd pre-program quarters


     Pros:
        •    Potentially applicable to most programs.
        •    Data is readily available.
        •    Relatively low cost and low burden to produce.
        •    It is an outcome-based efficiency measure. Therefore, it is of substantial use to
             understanding program effectiveness and costs.
     Cons:
        • Somewhat difficult to understand.
        • Lengthier lags in data (must wait several quarters after the end of the program year).

(7) Cost per $1,000 in Post-Program Earnings (CPPE) = Total program cost divided by total
    earnings in 2nd and 3rd post-program quarters for participants of exiters multiplied by
    $1,000.

                      CPPE =         Total Program Costs
                                     Total Participant or Exiters Earnings in 2nd and 3rd
                                     post-program quarters multiplied by $1,000


     Pros:
        •    Potentially applicable to most programs.
        •    Data is readily available.
        •    Relatively low cost and low burden to produce.
        •    It is outcome-based.
        •    Unlike the prior measure, does not weight prior employment earnings against post
             program earnings.
     Cons:
        • Lengthier lags in data (must wait several quarters after the end of program year).
        • Somewhat difficult to understand.


(8) Cost per Exiter or Participant Receiving a Particular Service (CPS) = Total program cost
    of a particular service divided by the number of exiters or participants receiving a particular
    service.

                       CPS = Total Cost of Particular Program
                             Participants or Exiters Who Received Particular Service

     Pros:
        • Easy to understand.
        • No lags in data. Data can be immediately generated at the end of each year.

     Cons:
        • Only applicable to programs that distinguish types of service.
        • Data is readily available for some programs, but not all.
        • Is not an outcome-based efficiency measure.
        • May be burdensome to generate.

(9) Cost per Placement in Employment or Education = Total program cost divided by the
    number of participants or exiters in employment or enrolled in post secondary education
    and/or advanced training or advanced training occupational skills in the 1st quarter after exit.

                       CPEE =         Total Program Costs
                                      Number of Exiters or Participants Employed or in Post
                                      Secondary Education Programs after 1st Quarter Exit

     Pros:
        • The data is relatively easy to understand.
        • Relatively low cost and low burden to produce.
        • The measure is outcome-based so it is of substantial use to understanding program
           effectiveness.
     Cons:
        • Limited to primarily the Workforce Investment Act Youth program.

(10) Cost per Individual Attaining a Recognized Degree or Certificate (Credentials include
     but are not limited to, a high school diploma, GED, or other recognized equivalents, post-
     secondary degrees/certificates, recognized skill standards, and licensure or industry-
     recognized certificates.) = Total training program cost divided by the number of
     participants or exiters receiving a training service attaining a recognized credential during
     participation or by the end of the 3rd quarter after exit.

                       CID = Total Training Program Costs
                             Number of Participants or Exiters who Attained
                             Certification or Degree by the end of 3rd Quarter after exit


     Pros:
        • The measure is an outcome-based measure, so it is of substantial use in
           understanding program effectiveness.
     Cons:
        • Only applicable to programs that provide services and identify individuals as
           receiving training and types of credentialing.
        • Data is readily available for some programs, but not all.
        • The measure is somewhat difficult to understand.
        • Potentially lengthy lags in data.

(11) Return on Investment (ROI). ROI is a way of summarizing how large the gain on an
     investment, such as workforce development, actually is. In its simplest form, ROI is
     calculated by dividing the gain by the size of the investment. This equation can be written
     as B/C, where B is the sum of all benefits that result from the investment over the period
     considered and C represents the costs. For a workforce program, one would divide the
     increase in earnings due to the program by the cost of the program. In more sophisticated
     analyses, ROI calculations take into account the timing of the gains due to the program.
     Economists typically compute a variation called the internal rate of return (IRR), which is
     based on the costs and benefits over the life of the investment. The IRR can be calculated,
     using a financial calculator or a spreadsheet, by solving the following equation for i:

     0 = -C0 + (B1 –C1)/(1+i) + (B2 –C2)/(1+i)2 + (B3 –C3)/(1+i)3 + …. + (BN –CN)/(1+i)N

     Where Bt is the benefit received in year i, Ct is the cost incurred in year i, and N is the last
     year that benefits or costs occur. (The four dots mean that the formula includes the same
     type of term for all years between year 3 and year N.) The IRR is preferred to the simpler
     versions of ROI because it takes into account the timing of the costs and benefits.

     Pros:
        • Potentially applicable to most programs.
        • Measure is an impact-based efficiency measure, which controls for factors that
           could potentially influence/bias results. Therefore, it is of the greatest utility in
           understanding program cost-effectiveness.
        • This measure controls for difficulty or cost of serving different populations (e.g.,
           hard-to-serve, service mix, and economic conditions).
     Cons:
        • Data is very costly to produce.
        • The measure is difficult to understand.
        • Lengthy lags in data.