Revising WIA negotiated performance levels

Accessible letter PDF: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2011/TEGL11-01c1acc.pdf

                                                    CLASSIFICATION
  EMPLOYMENT AND TRAINING ADMINISTRATION            WIA/Performance Levels
              ADVISORY SYSTEM                       CORRESPONDENCE SYMBOL
         U.S. DEPARTMENT OF LABOR                   OPDR-OWI
            Washington, D.C. 20210                  DATE
                                                    August 12, 2011



ADVISORY: TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 11-01,
          Change 1

TO:            ALL STATE WORKFORCE AGENCIES
               ALL STATE WORKER ADJUSTMENT LIASONS
               ALL STATE WORKFORCE LIAISONS
               ALL ONE-STOP CAREER CENTER SYSTEM LEADS

FROM:          JANE OATES /s/
               Assistant Secretary

SUBJECT:       Guidance on Revising Workforce Investment Act (WIA) State
               Negotiated Levels of Performance

1. Purpose. This change Training and Employment Guidance Letter (TEGL) is being
   issued to reflect modifications made to the original TEGL 11-01, which include the
   following: updated information regarding the regression modeling tool for target
   setting, revised time periods to submit renegotiated goals, and updated language to
   include the common measures waiver in negotiations.

   This guidance letter describes criteria and procedures the Department of Labor
   (DOL) will use when considering state requests to revise negotiated levels of
   performance established under title IB of WIA.

2. References.
       WIA Section 136.
       WIA regulations at 20 Code of Federal Regulations (CFR) part 666.
       WIA regulations at 20 CFR 661.230(b)(2).

3. Background. Under WIA title I, subtitle B, each state workforce agency must
   negotiate expected levels of performance with DOL for the adult, dislocated worker,
   and youth programs, as well as customer satisfaction, for a total of 17 measures
   (unless the state has a common measure waiver, in which case the state would only
   be required to negotiate the nine common measures).




 RESCISSIONS                                       EXPIRATION DATE
 TEGL 11-01                                        Continuing
   Section 136(b)(3)(A)(vi) of WIA permits each Governor to request revisions to the
   state negotiated levels of performance in the event “unanticipated circumstances
   arise in a state resulting in a significant change in the factors” described in section
   136(b)(3)(A)(iv)(II) of WIA. These factors include the economic conditions for the
   state, the characteristics of participants, and the services provided to participants.

   TEGL 11-01 was published in February 2002 and was based in large part on a
   workgroup of state and local representatives who convened in 2001. At the time, a
   negotiations-based accountability framework represented a new approach to the
   determination of performance levels. Therefore, obtaining input on a renegotiation
   framework from the workforce system was critical, and TEGL 11-01 reflected the
   workgroup’s contributions. This guidance letter rescinds the original policy and
   provides updated information and language.

4. Policy on Requesting Revisions. DOL views the process of establishing and
   revising negotiated levels of performance as a way to promote performance
   accountability and continuous improvement. Beginning with Program Year 2011,
   the Governor may request a revision to the corresponding Regional Office for one or
   more negotiated performance levels by the end of the third quarter (March 31) of the
   current program year for which the revised level(s) would apply. Further, to be
   considered for incentive awards, the Governor’s request (including new proposed
   goals) must be received by the corresponding Regional Office by the end of the
   second quarter of the current program year (December 31). This policy allows
   adjustments to negotiated performance goals in order to account for changes in
   economic conditions, changes in the characteristics of the participants served by the
   program, and changes in service delivery design. DOL expects a request for
   revisions to performance levels to be submitted by a state workforce agency as soon
   as possible after the identification of the unanticipated circumstance.

5. Negotiation Process for Revising Levels of Performance. Each request submitted
   by a state workforce agency for a revision to one or more performance levels will be
   reviewed by the Employment and Training Administration (ETA) Regional Office
   based on its own merits.

      1) The Request. Each state workforce agency seeking a revision will develop and
      submit a written request to the ETA Regional Office serving the state.

      2) Review of the Request. The ETA Regional Office will review the request and
      determine whether the state request appropriately documents and satisfies each
      of the following three conditions listed below. Each of these conditions are
      discussed in more detail in Attachment I.

          a) Condition 1. The request must articulate an unanticipated circumstance.



                                            -2-
         b) Condition 2. The unanticipated circumstance, in turn, must impact one or
         more factors. There should be evidence to associate the unanticipated
         circumstance with an actual change in one or more of the factors.

         c) Condition 3. The stated variations from expected outcomes identified in
         the request must be linked to the stated significant changes in factors. There
         should be sufficient and appropriate documentation to explain and justify the
         proposed revised levels of performance.

      3) The Negotiation. The revised levels of performance will be negotiated
      between the ETA Regional Office and state workforce agency staff to ensure that
      reasonable and appropriate levels are set. The ETA Regional Office will identify
      where data or information are insufficient to justify the request. Both the ETA
      Regional Office and state staff should work together to ensure the necessary
      information is included and considered when processing the request.

      4) The Decision. The ETA Regional Office will convey the results of the
      negotiation to the state workforce agency within 30 working days after receipt of
      the written request. All approved revisions will be incorporated into the State
      Plan through a plan modification. Please note that modifications to the State
      Plan are subject to the same public review and comment requirements that apply
      to the development of the original State Plan. The ETA Regional Office will
      maintain all documentation supporting its decision. Approved revised levels of
      performance will be effective on the date agreement is reached between the ETA
      Regional Office and the state workforce agency, and will be applied to the entire
      program year in question. In the event one or more of these revised levels are
      changed afterwards as a result of the public review and comment requirement,
      the impacted request and decision will be reconsidered by the ETA Regional
      Office. States should direct any questions about the plan modification process to
      their ETA Regional Office.

6. Factors to be Included in the Renegotiation Justification. When determining
   whether sufficient information is available to document and satisfy the three
   conditions, the ETA Regional Office will look for:

            A description of the nature of the problem or mitigating circumstance,
             including a description of when the unanticipated circumstance occurred
             and its duration or expected duration (Condition 1).

            The performance measure(s) and program year(s) affected by the request
             (Condition 2).




                                         -3-
             Evidence of the change in the factors taken into account in the earlier
              negotiation(s), the forecasted factor values, if appropriate, and the
              estimated impact on the performance outcome(s) (Condition 2 and
              Condition 3).

             A description of the approach(s) used to determine revised levels of state
              negotiated performance, including methods used to forecast annual factor
              values, if appropriate (Condition 3).

             A description of the data source(s) used to demonstrate change in the
              factors and a description of the data sources used to forecast values for the
              factors, if appropriate (Condition 3).

             The computations for the revised performance level(s) included in the
              request (Condition 3).

   As noted, this guidance for justifying revisions to negotiated levels of performance
   reflects input from a workgroup of state and local officials. Attachment I
   summarizes, in greater detail, each of the three conditions cited earlier and largely
   reflects comments from the stakeholders’ workgroup. Attachment II describes one
   approach that state workforce agencies may find useful when calculating
   appropriate levels of performance. ETA Regional Offices will use these attachments
   as a guide when reviewing requests. States may also want to consider them when
   analyzing performance levels and developing appropriate requests for revisions.

7. Action Requested. States should distribute this TEGL to all officials within the state
   who need such information to implement the performance accountability policies
   under the WIA.

8. Inquiries. Questions concerning this issuance should be directed to your
   appropriate ETA Regional Office.

9. Attachment.

      Attachment I:         Relating the Three Conditions for Revising Levels of
                            Negotiated Performance
      Attachment II:        The Weighted Average Approach to Revising Levels of
                            Negotiated Performance




                                           -4-

Official attachment PDF: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2011/TEGL11-01c1a1.pdf

                                      Attachment I

                   Relating the Three Conditions for Revising Levels of
                                 Negotiated Performance

Background

Section 136(b)(3)(A)(vi) of the Workforce Investment Act (WIA) specifies that a
Governor may request revisions to state negotiated levels of performance in the event
“unanticipated circumstances arise in a state resulting in a significant change in the
factors” that include factors described in section 136(b)(3)(A)(iv)(II) of WIA.
Additionally, 20 CFR 666.130(a) states the change in factors is to be determined from
when the initial plan was submitted and approved. These factors are widely accepted
variables known to impact resulting outcomes on one or more of the 17 statutory WIA
performance measures, or nine common measures. These factors include differences in
economic conditions, the characteristics of participants, and the services provided.

There are three conditions for any state workforce agency request for revisions to state
negotiated levels of performance to be approved by the Employment and Training
Administration (ETA) Regional Office. These conditions are graphically displayed in a
logic model below.

                              Relating the Three Conditions

     Condition 1                   Condition 2                   Condition 3
                                                               Factor(s) Known
    Unanticipated                 Demonstrated                     to Impact
    Circumstance                  Change in the                Performance and
       Occurs                       Factor(s)                   Revised Levels
                                                              can be Objectively
                                                                 Determined

  The process for                 The unanticipated            The change in the
  requesting                      event must impact            factor(s), in turn,
  revisions to earlier            one or more of the           must be related to
  negotiated levels of            factors taken into           performance on
  performance is                  consideration in the         one or more of the
  triggered by an                 earlier negotiations         measures. The
  unanticipated                   of performance               degree of the
  circumstance. The               levels. It must be           impact on
  unanticipated                   reasonable to                performance may
  circumstance must               associate the event          be estimated using
  occur for a request             with a change in             historical data.
  to be considered by             the factor(s).
  the Secretary.

                                            -1-
Condition 1: Unanticipated Circumstances

Actual performance on the 17 statutory WIA title I measures of performance, along with
the nine common measures, varies above or below expected levels of performance for a
number of reasons. Variations from expected outcomes can also result from significant
changes in factors that are beyond the control of the state workforce agency. In these
situations, state workforce agencies should consider submitting requests to revise
negotiated levels of performance impacted by the significant change in factors.

Variations from expected outcomes can also occur from significant changes in
administrative practices within the control of the state workforce agency. Changes in
expected performance under these circumstances do not merit revisions to negotiated
levels of performance. By themselves, these changes would not be considered the result
of “unanticipated circumstances.”

Examples of unanticipated circumstances are listed below. Please note, the list of
unanticipated circumstances is not exhaustive and does not constrain a state workforce
agency from documenting its own experiences that have prompted the need to seek a
revision.

   ·   Changes in the characteristics of program participants, such as service to hard-to-
       serve customer groups, not taken into account in the earlier negotiation(s) on
       state levels of performance.

   ·   Changes in economic conditions, such as plant closings and mass layoffs.

   ·   Changes in economic assumptions and outlooks.

   ·   Disasters, such as floods, earthquakes, hurricanes, tornadoes, and acts of war.

   ·   Significant changes in Federal funds allotted to the state to implement WIA.

   ·   Unanticipated legal or legislative actions that increase the quality of participant
       outcomes and, as a result, increase the level of effort needed by participants to
       attain these outcomes. For example, a new state law may impose higher
       standards for high school graduation. This action would increase the standard
       needed to attain a diploma or equivalent and could decrease the numbers of
       graduates.




                                            -2-
Condition 2: Demonstrated Change in the Factor(s)

The unanticipated circumstance, by itself, does not provide sufficient information to
justify a revision in state negotiated levels of performance. There should be sufficient
evidence to demonstrate change in one or more factors initially considered when
reaching agreement on the state negotiated levels of performance. Those factors might
have been an annual unemployment rate of 5.4 percent, a baseline adult entered
employment rate of 72 percent, or 53 percent of the adult participants demonstrating
one or more significant barriers to employment. It is worthwhile to keep in mind that
an unanticipated circumstance may impact performance measures covering multiple
program years.

Condition 3: Factor(s) Known to Impact Performance and Revised Levels can be
Objectively Determined

Because unanticipated circumstances are, by definition, outside the normal operating
conditions, there can be no predetermined numerical guidelines within which
negotiated levels of performance might be revised. However, any justification for
revising performance levels should be based on generally acceptable approaches and
data sources.

A. Guidelines for Approaches and Data Sources

   ·   The methods used to determine revisions to negotiated levels of performance
       must:

       –   Adhere to widely accepted statistical practices, including predictive or
           forecasting techniques where appropriate.

       –   Demonstrate a reasonable cause and effect relationship between one or more
           factors and performance on a measure.

       –   Be fair, objective, and yield quantifiable results.

       –   Support, and not undermine, state efforts in achieving continuous
           improvement of workforce investment activities.

   ·   The source data must be:

       –   Developed by 1) a Federal, state or local governmental agency, or 2) some
           other reputable source such as the state’s management information system
           unit, a university, or a private research foundation.




                                             -3-
       –   Gathered according to acceptable data collection techniques.

       –   Compiled according to widely accepted analytical procedures.

   ·   The factors used to determine revisions to negotiated levels of performance
       include:

       –   Differences in economic conditions.

       –   The characteristics of participants at the time of registration.

       –   Services to be provided to participants.

B. Using Predictive Statistical Models to Estimate Revised Levels

State workforce agencies may develop their own statistical forecasting models using the
multiple regression approach or use one or more other statistical methods used to
predict selected outcomes. The multiple regression statistical method is a widely used
technique that determines the relationship between a selected performance outcome
and multiple explanatory factors or variables.

While state workforce agencies may develop their own statistical models, ETA
recognizes that many states do not have the statistical expertise or the necessary
resources to develop these models. As a result, states are encouraged to use the model
ETA has developed for use in the negotiations process to quantify the degree of
relationship between the factors and each of the performance measures in the
renegotiation process, should the factors be included in the set of variables used in the
model. To compute the adjustment, the state would multiply the change in the factor
by that factor’s coefficient. This value would then be added to the negotiated target to
arrive at the renegotiated target.

C. The Importance of Negotiation in Revising State Levels of Performance

Under certain circumstances, the predictive statistical models may yield levels of
performance that are unrealistic. Imprecision in the models, extreme factor values, and
the occurrence of rare, isolated economic changes (such as those due to natural
disasters) are examples of conditions that may yield unrealistically high or low levels of
expected performance. In these and similar situations, the proposed revised levels of
performance should be negotiated between the ETA Regional Office and state staff to
ensure reasonable and appropriate levels are set.

The expectation for performance is derived from state experience with similar
unanticipated circumstances or suggested by research studies. However, the revised



                                             -4-
level should not be based solely on past performance or experience under similar
circumstances. The revised level of performance should reflect what is ideally
attainable, given the change in the environment and any needed changes to the delivery
of services.

Since environmental conditions are likely to be unique for each unanticipated
circumstance arising in a state, defining an appropriate performance level will likely be
somewhat subjective. In reaching agreement on a definition for exemplary
performance, the ETA Regional Office and state staff should consider the following
questions:

        · What is the unanticipated circumstance?

        · What are the changes in the factors considered in formulating the original
          state negotiated levels of performance?

        · Who are the participants impacted by the changes in the factors (e.g.,
          population group)?

        · What is the estimated performance impact of the changes in the factors on the
          population group?

        · What changes can be feasibly made by the state to the design and delivery of
          services to address the impact of the unanticipated circumstance? What
          programmatic changes are being proposed by the state?

        · Given the changes in the factors, what level of performance is estimated to
          occur?

              - Without changes to the design and delivery of services to the target
              group?

              - With feasible changes to the design and delivery of services to address
               changes in the factors caused by the unanticipated circumstance?

        · What level of performance does the state cite for the balance of exiters not
          included in the population groups impacted by the unanticipated
          circumstance? Is this level different than the initial state negotiated level of
          performance for the measure? Why? Is the level reasonable and appropriate?

        · What level of performance does the state suggest as appropriate for the
          measure? Why?




                                           -5-
· What level of performance does the ETA Regional Office cite as ideally
  attainable? Why?




                                 -6-

Official attachment PDF: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2011/TEGL11-01c1a2.pdf

                                    Attachment II
                     The Weighted Average Approach to Revising
                          Levels of Negotiated Performance

Overview

The weighted average approach views the state negotiated levels of performance as
aggregate levels of local Workforce Investment Board (WIB) negotiated performance
and statewide project performance goals. Similarly, local WIB negotiated levels of
performance should be viewed as aggregate levels of project and/or target group
performance goals.

Under the weighted average approach, the overall state negotiated level of performance
on a measure is disaggregated into expected levels of performance for one or more
affected target groups and for the balance of exiters included in the calculation of
performance on the measure. Agreed-upon revised levels of performance are then
applied to each target group and the results are then aggregated to derive a revised
state negotiated level of performance.

In some situations, the unanticipated circumstance may equally impact the expected
outcomes of all exiters included in the calculation of performance on a measure. In
these situations, the weighted average approach should not be used to derive revised
performance levels. Instead, the Employment and Training Administration (ETA)
Regional Office and designated state staff should reach agreement on a level for all
exiters included in the calculation of performance on a specific measure.

Defining a Special Population Group for Use in the Formula

A “special population group” may be categorized in a number of ways, including: 1)
participants affected by significant changes in economic conditions; 2) the demographic
characteristics of participants; and 3) the type of services provided to participants. For
the purposes of this paper, a special population group is a collection of individuals
whose outcomes on a measure are expected to be uniquely impacted by an
unanticipated circumstance that results in a change in one or more of the factors
considered in reaching agreement on the state negotiated levels of performance. If
more than one special population group is identified as being affected by the
unanticipated circumstance, it is important to ensure these groups are mutually
exclusive.

Negotiating Appropriate Performance Expectations

Both the ETA Regional Office and designated state staff should reach agreement on an
expectation for each special population group and the balance of exiters identified for a



                                           -1-
specific performance measure. The expectation may be derived from state experience
with similar unanticipated circumstances or suggested by special research studies.

The Formula

The weighted average is calculated by multiplying the agreed-upon performance level
for each group by the number of expected exiters in each group, totaling these results,
and then dividing this result by the total number of expected participants exiting
services in the state.

       SGE1  SGP1   SGE 2  SGP2   SGE3  SGP3   TE  SGE1  SGE 2  SGE3   TP 
RP 
                                                           TE
Where…

         SGE(1,2 or 3) is the number of estimated exiters in the special population group 1 .

         SGP(1,2, or 3) is the negotiated performance level for exiters in the special
         population group1.

         TE is the total number of estimated exiters in the state.

         TP is the negotiated level of performance level for all exiters not included in the
         special population groups. In most situations, this value is the same as the
         original state negotiated level of performance for the measure.

         RP is the revised negotiated level of performance for the measure.

Example Application of the Weighted Average Method

Note: The following example was created to illustrate how a state request for a revised
performance level might be developed for submission to the Secretary. For the sake of simplicity,
the following example request is for a proposed revision to one measure for a single program year
impacted by an unanticipated circumstance. It is quite likely, however, several measures
covering multiple program years would be impacted by the unanticipated circumstance.

The Governor made a request to the Secretary to revise the state negotiated
performance level for the dislocated worker entered employment rate measure for
Program Year (PY) 2010. The Governor requested that the level be revised from the PY
2010 negotiated level of 77 percent to 72.4 percent to account for the loss of job openings
and the increased number of unemployed in selected communities of the state that

1
 Additional special target groups may be included in the formula as needed to accurately reflect the impact of an
unanticipated circumstance.




                                                        -2-
resulted from the closure of one of the state’s largest employers (Condition 1). The
employer ceased operation on August 15, 2010.

The state request indicated that there are 1,300 dislocated workers included in the
calculation of the PY 2010 dislocated worker entered employment rate, with 350 of the
1,300 dislocated worker program exiters impacted by the closure. The 350 dislocated
worker participants resided in the impacted communities and exited services during the
first quarter of PY 2010. The unemployment rate for the impacted communities rose
from an average of 5.2 percent for the nine months before the closure to 9.7 percent for
the three months after the closure of the employer (Condition 2). The state contacted a
representative sample of public (non-Workforce Investment Act) and private placement
and temporary staffing agencies serving the impacted communities and found the
average placement rates for these agencies dropped 18 percent from 78 percent during
the first quarter of PY 2009 to 60 percent during the first quarter of PY 2010 (Condition
3).

The state used the weighted average approach to determine a revised level for its
dislocated worker entered employment rate. The target group used in the state’s
computation is dislocated worker participants from communities expected to be impacted by the
plant closings. Based on the state’s review of the impact of the closure on both public
and private employment agencies, the state set a goal of 60 percent (the average rate
reported by employment agencies) for the estimated 350 exiters in the target group.
The goal for the remaining 950 dislocated worker exiters included in the calculation was
77 percent (or the original state negotiated level of performance).

To compute the revised level for the state overall, the state used the weighted average
method to address the estimated impact of the target group on PY 2001 performance on
the dislocated worker entered employment rate measure. The following scaled-down
formula to compute this revised level was used to determine the proposed revised
performance level requested by the state.


                          RP 
                                 SGE  SGP   TE  SGE   TP 
                                                  TE

Where…

        SGE = 350 (the number of estimated exiters in the special population group)

        SGP = 60 percent (the negotiated performance level for exiters in the special
        population group)

        TE = 1,300 (or the sum of 350 and 950 - the total number of estimated exiters in
        the state)



                                              -3-
       TP = 77 percent (the original state negotiated level of performance level for the
       measure)

Applying the above values in the formula, RP - the revised negotiated level of
performance for the measure – equals 72.4 percent, the level proposed in the state’s
request.


                        RP 
                               350  0.60  1,300  0350  0.77 
                                                 1,300

                                    210  731.5
                            RP                  0.724  72.4%
                                      1,300

Using the results of the weighted average formula, the Governor requested the level be
revised from the current negotiated level of 77 percent for PY 2010 to 72.4 percent to
account for the loss of job openings and the increased number of unemployed in
selected communities of the state that resulted from the closure of one of the state’s
largest employers.




                                              -4-