Evaluation and evidence requirements for RESEA programs

Official signed letter: TEGL 6-19.

CLASSIFICATION
   EMPLOYMENT AND TRAINING ADMINISTRATION                 Unemployment Insurance
               ADVISORY SYSTEM                            CORRESPONDENCE SYMBOL
          U.S. DEPARTMENT OF LABOR                        OUI/DUIO
             Washington, D.C. 20210                       DATE
                                                          October 31, 2019

ADVISORY:       TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 6-19

TO:             STATE WORKFORCE AGENCIES

FROM:           JOHN PALLASCH /s/
                Assistant Secretary

SUBJECT:        Expectations for States Implementing the Reemployment Service and
                Eligibility Assessment (RESEA) Program Requirements for Conducting
                Evaluations and Building Program Evidence

1. Purpose. To provide States with guidance and expectations regarding the implementation of
   the RESEA evaluation and evidence requirements.

2. Action Requested. The Department of Labor’s (DOL’s) Employment and Training
   Administration (ETA) requests State Workforce Administrators to provide information
   contained in this Unemployment Insurance Program Letter (UIPL) to appropriate program
   and other staff in the state’s workforce system. This information should be shared with staff
   in, but not limited to, the Unemployment Insurance (UI) program, workforce programs
   administered under the Workforce Innovation and Opportunity Act (WIOA), including the
   Wagner-Peyser Employment Service, and workforce information/labor market information
   programs.

3. Summary and Background.

      Summary – In accordance with the statutory provisions for RESEA contained in the
       Social Security Act (SSA), states are expected to begin conducting evaluations of
       RESEA interventions and service delivery strategies no later than fiscal year (FY) 2020
       to support building new evidence on effective RESEA interventions that all states can
       rely on in designing and delivering the RESEA program.

      Background – On February 9, 2018, the President signed the Bipartisan Budget Act of
       2018, Public Law No. 115-123 (BBA), which amended the SSA and created a permanent
       authorization for the RESEA program. The RESEA provisions are contained in Section
       30206 of the BBA, enacting new Section 306 of the SSA.

       Section 306 of the SSA includes a tiered evidence approach for the RESEA program to
       encourage states to use evidence-based strategies, where they exist, and to conduct
       evaluations and build evidence for other interventions and service delivery strategies.
       The goal is to ensure that each state employs RESEA interventions and service delivery

 RESCISSIONS                                             EXPIRATION DATE
 None                                                    Continuing

        strategies that are based on rigorous causal evidence from evaluations rated as “high or
        moderate causal” and are shown to reduce benefit duration as a result of improved
        employment outcomes. In addition, states using interventions or service delivery
        strategies without such evidence must be under evaluation at the time of use to determine
        their effectiveness in achieving this goal. Over time, as the RESEA program uses this
        tiered evidence approach,1 states will add to the evidence base; grow the workforce
        system’s understanding of what interventions work well, for whom, and in what contexts;
        and expand the use of interventions with strong evidence of success. Similar tiered
        evidence models are used across federal government programs such as the Department of
        Health and Human Services’ Home Visiting Program.
        The statute, in section 306(c), SSA, requires states to use RESEA grant funds for
        evidence-based interventions or service delivery strategies that reduce the average
        number of weeks participants receive benefits by improving employment outcomes,
        including earnings. Specifically, it requires the following with regard to evidence
        building and evaluations:

                 “(c) EVIDENCE-BASED STANDARDS.--

                          “(1) IN GENERAL.--In carrying out a State program of reemployment
                          services and eligibility assessments using grant funds awarded to the State
                          under this section, a State shall use such funds only for interventions
                          demonstrated to reduce the number of weeks for which program
                          participants receive unemployment compensation by improving
                          employment outcomes for program participants.

                       “(2) EXPANDING EVIDENCE-BASED INTERVENTIONS.--In
                       addition to the requirement imposed by paragraph (1), a State shall –
                           (A) for fiscal years 2023 and 2024, use no less than 25 percent of the
                               grant funds awarded to the State under this section for
                               interventions with a high or moderate causal evidence rating that
                               show a demonstrated capacity to improve employment and
                               earnings outcomes for program participants.
                           (B) for fiscal years 2025 and 2026, use no less than 40 percent of such
                               grant funds for interventions described in sub-paragraph (A); and
                           (C) for fiscal years beginning after fiscal year 2026, use no less than 50
                               percent of such grant funds for interventions described in sub-
                               paragraph (A).
                 “(d) EVALUATIONS.--

1
 “Tiered evidence” refers to a policy tool that allows federal agencies to tie federal funding to strategies with
evidence, to encourage the use of interventions that have strong evidence of success and test promising new ideas.
With the RESEA program, the legislation ties certain levels of future funding to interventions with moderate or high
causal evidence ratings, to encourage the use of those interventions that have stronger evidence that they “work,”
and requires interventions without those ratings to be to under evaluation at the time of use.

                     “(1) REQUIRED EVALUATIONS.--Any intervention without a high or
                     moderate causal evidence rating used by a State in carrying out a State
                     program or reemployment services and eligibility assessments under this
                     section shall be under evaluation at the time of use.
                     “(2) FUNDING LIMITATION.--A State shall use not more than 10
                     percent of grant funds awarded to the State under this section to conduct
                     or cause to be conducted evaluations of interventions used in carrying out
                     a program under this section (including evaluations conducted pursuant to
                     paragraph (1).”

      ETA provided preliminary guidance with regard to these provisions for FY 2019 in UIPL
      No. 07-19 (https://wdr.doleta.gov/directives/corr_doc.cfm?DOCN=8397).

      This guidance provides information on the new RESEA evidence-based requirements and
      provides definitions of high and moderate causal evidence. High or moderate causal
      intervention ratings are based on how many good quality studies show positive impacts
      of that intervention. To provide states a solid foundation on the meaning of good quality
      studies, this guidance presents a description of how DOL rates studies’ quality of
      evidence through its Clearinghouse for Labor Evaluation and Research (CLEAR) (see
      Section 6, below). The guidance also discusses the standards for rating intervention
      effectiveness and identifies relevant interventions that currently meet those standards.
      In addition to setting standards and intervention ratings, this guidance also suggests
      RESEA components that are in need of expanded evidence and includes a discussion of
      evaluation approaches and strategies for carrying out evaluations. Finally, the guidance
      points to resources that are available to states to better understand and use existing
      evidence and to help states initiate rigorous high-quality evaluations to build evidence on
      the effectiveness of interventions in their RESEA programs.
      While the intent is that states will implement interventions and service delivery strategies
      supported by rigorous evidence, there is not yet a large body of such evidence related to
      the new parameters for the permanent RESEA program. States must begin conducting
      rigorous studies to produce new evidence that helps determine the success of the
      interventions and service delivery strategies that meet the goals of the RESEA program.

4. Expectation that States Begin RESEA Evaluations No Later than FY 2020.

   Reemployment evaluations, to date, have focused mainly on broad categories of services or
   services at a program level. These evaluated programs have similarities to RESEA, but also
   many differences. A primary goal of the RESEA legal requirement for evaluations is to
   expand the evidence base by conducting new high-quality evaluations of states’ RESEA
   programs, particularly to build evidence about specific program components or activities.

   Congress, as reflected in the provisions of section 306, SSA, intended the evidence base for
   RESEA to expand and to improve the program through state use of evidence-based
   interventions with high or moderate causal ratings. While there is a modest and growing
   evidence base from which to synthesize and draw conclusions about RESEA interventions’

   effectiveness, there is an immediate need to grow and expand it to address new RESEA
   program components. Previous evaluations of the Reemployment and Eligibility Assessment
   (REA) program, the predecessor to the RESEA program, were based on the whole program
   and the need now is to develop and expand evidence on more well-defined activities,
   program components, and service delivery approaches that states use in operating the
   RESEA program. Development of a culture of continuous improvement and evidence
   building around the RESEA program will strengthen it over time and improve reemployment
   outcomes for unemployment compensation (UC) claimants.

   To meet Congressional intent with regard to causal evidence ratings in the tiered evidence
   approach and to ensure states’ ability to comply with the evidence and evaluation provisions
   in the statute, states are expected to begin evaluating RESEA interventions and service
   delivery strategies as soon as feasible and no later than the end of FY 2020 for the following
   reasons:
        The requirement that states use only interventions with high or moderate causal
           evidence ratings or have them under evaluation is in effect in FY 2020;
        RESEA, while modeled in part after the former REA, is a different program and
           includes the actual delivery of reemployment services in addition to the foundational
           elements of the REA program, so evidence beyond evaluations of the REA program
           is needed;
        Expanded evidence is needed to ensure that states have sufficient evidence to support
           program delivery when the minimum percentage requirements for use of
           interventions with high or moderate causal ratings begin in FY 2023;
        In the new RESEA state plan, required in FY 2020, states must articulate a
           description of their evaluation structure for RESEA interventions without a high or
           moderate causal evidence rating; and
        Rigorous impact evaluations sufficient to achieve a high or moderate causal rating
           are most often multi-year in length, and states need to begin conducting evaluations
           now to obtain sufficient evidence to support delivery strategies and interventions of
           their RESEA programs in FY 2023 and beyond.

5. Evaluation Parameters.
   Given that a key goal of the RESEA program is reduced average duration of UC benefit
   receipt as a result of improved employment outcomes, states’ RESEA impact evaluations
   must include duration of UC and employment as primary outcomes:

      Unemployment Compensation Duration: This outcome is measured as the number of
       weeks RESEA participants receive UC; and

      Employment: For RESEA participants, employment and earnings outcomes can be
       measured in the second full calendar quarter following the start of a participant’s UC
       claim similar to the WIOA measures, or sooner in the claims cycle to the extent that data
       is available.
   States are also encouraged to propose additional outcomes that could provide early
   indications that the RESEA program is working as intended. Examples of outcomes that

   states might consider include increased participation in or completion of the RESEA program
   activities, or time to reemployment following the start of RESEA interventions.
   States should consider, when feasible, coordinating their RESEA evaluations with their
   WIOA-mandated evaluation projects which can create economies of scale and generate
   synergies across programs. States’ new evaluations must meet evidence standards for study
   quality and find favorable impacts with at least a reasonable degree of statistical confidence
   to allow the intervention under examination to potentially qualify for a high or moderate
   rating, as defined in Section 8, below. The goal of this evidence-generating approach is to
   provide states operating RESEA programs with a sufficient number of new studies that meet
   these standards, which can support, along with current evidence, the statutory requirement
   for states to use interventions demonstrated to be effective.
   DOL recognizes that all findings, whether positive, negative, or null, are important
   contributions to the evidence base, and DOL is committed to learning from and using
   evaluations and data to inform program improvements. As such, it is both critical and
   expected that all evaluations conducted of RESEA interventions be publicly available,
   regardless of the outcomes. States are also encouraged to share links to their publicly posted,
   completed evaluations with CLEAR to ensure their inclusion in future evidence reviews.
   RESEA evaluations will play an important role in building the reemployment evidence base,
   and in helping states and other program decision-makers make more informed choices about
   how to bundle RESEA program components and strategies to best meet the needs of the
   people being served by them.

6. Clearinghouse for Labor and Evaluation Research (CLEAR).

   A first step in identifying interventions with high and moderate causal ratings is determining
   which existing studies provide evidence about them that is relevant and credible. DOL will
   leverage CLEAR to identify evaluations in the evidence base that are relevant to the RESEA
   program and determine which impact studies have high, moderate, or low causal evidence
   ratings. DOL established CLEAR to make research on labor topics more accessible to
   practitioners, policymakers, researchers, and the public so that evidence can inform policy
   and program decisions. To achieve this goal, CLEAR conducts systematic evidence reviews
   of research and evaluation reports on labor topics, and then reviews and summarizes those
   studies. CLEAR also rates studies that estimate causal impact.

   CLEAR currently has over 600 studies summarized across 18 labor-related topic areas,
   including “Reemployment,” and is continually growing. The “Reemployment” evidence
   review identifies, summarizes, and determines the quality of existing causal evidence on
   reemployment service delivery strategies intended to promote reemployment of UC
   claimants while also reducing UC receipt duration. Under the “Reemployment” topic area,
   CLEAR has reviewed 45 publications published between 1978 and 2018, and has developed
   one-page summary profiles of and ratings for each of these studies. The reviewed studies use
   “causal” designs, otherwise known as “impact studies,” and assess the effectiveness or
   impact of an intervention. These studies identify how a particular intervention changes
   claimants’ outcomes relative to a comparison group, such as those that receive a different
   intervention or those that did not receive the intervention.

       Many impact studies use random assignment designs. Such designs randomly (i.e., through
       the functional equivalent of a coin toss) assign some eligible individuals to a “treatment
       group” or groups that may participate in the intervention and others to a “control group” that
       do not participate in the intervention. These designs use random assignment to prevent
       systematic pre-existing differences between the two groups from creating bias in an
       evaluation. Thus, systematic differences in outcomes between the two groups can reasonably
       be attributed to the intervention. Other causal impact studies may use “quasi-experimental”
       designs that estimate impact, but do not use random assignment. Instead, quasi-experimental
       designs use administrative data and statistical techniques to identify a comparison group that
       is similar to the treatment group to act as a control group.

       The credibility of the evidence from an impact study depends on how it is designed and
       carried out. Currently, CLEAR has established causal evidence guidelines,2 which identify
       the criteria CLEAR uses to assess the strength of a study’s causal evidence. CLEAR’s causal
       evidence ratings are an indicator of the quality of the study and the level of confidence you
       can have that the study’s findings truly reflect the causal impact of the intervention studied
       and not some other factor. CLEAR also has guidelines for high-quality, quantitative,
       descriptive, and implementation studies, but does not currently assign evidence ratings to
       those types of studies.3

       CLEAR currently assesses its causal evidence ratings based on the rigor of the study as
       follows.
            Studies receive a “high” rating for causal evidence if there is confidence that the
              study’s estimated effects are “solely attributable to the intervention being examined.”
            Studies receive a “moderate” rating for study quality if there is “some confidence that
              the estimated effects are attributable to the intervention studied, but there might be
              other contributing factors that were not included in the analysis.”
            Studies that do not meet the criteria for a high or moderate rating receive a “low”
              rating, which indicates that it is “not possible to be confident” that the estimated
              effects are attributable to the intervention studied. In these instances, other factors
              likely contributed to the estimated effects.

       Moving forward and as described below, CLEAR will assign causal evidence ratings to new
       RESEA studies based on both study quality and effectiveness of the intervention examined in
       a study, as appropriate. DOL will publicly and transparently post information about this
       process on the CLEAR website when future evidence reviews begin.

2
    Find CLEAR’s causal evidence guidelines here:
    https://clear.dol.gov/sites/default/files/CLEAR_EvidenceGuidelines_V2.1.pdf
3
 Find CLEAR’s quantitative descriptive guidelines and guidelines for reviewing implementation studies here, under
“Reference Documents”: https://clear.dol.gov/about

7. The Need for Expanded Evaluations of Interventions.

   The statute, in section 306(i)(3), SSA, defines an intervention as “a service delivery strategy
   for the provision of State reemployment services and eligibility assessment activities.” In
   operating RESEA programs, states bundle or mix multiple components and activities
   together in ways that may vary in their details or emphasis. They may also vary in the
   strategies or approaches for carrying them out. For instance, all RESEA programs include a
   claimant selection component, but states may vary in how they select claimants for
   participation. For evidence rating purposes, an evaluation intervention may be a whole
   program or any component of it.

   RESEA is different from any of the interventions for which evidence currently exists. It
   shares some elements with earlier programs, particularly with REA, but it also has new
   elements. For example, it places a greater emphasis on more intensive reemployment
   services for claimants and states now have greater freedom in deciding how to select
   claimants. States need to develop a substantial body of high-quality evidence about the
   effectiveness of RESEA strategies and components. Exhibit 1 lists components for which
   evidence needs to be built, in order to meet the basic requirement of demonstrating
   effectiveness and to provide meaningful findings to help states design and implement their
   RESEA programs. Other gaps in the evidence base are expected to emerge as more is
   learned about states’ current RESEA programs. As new evidence is produced, the list of
   interventions that have been demonstrated to be effective will be refined.

          Exhibit 1. RESEA Components in Need of Expanded Evidence
         Component Sub-component                           Specific Evidence to Build
                                    Claimant       What selection approach (if any) identifies
                                    Selection       claimants that will be most favorably affected
          Selecting Claimants and
           Scheduling Meetings

                                    Mechanism       by RESEA selection?
                                                         o Selecting those with high exhaustion
                                                             risk?
                                                         o Selecting those with low exhaustion
                                                             risk?
                                                         o Selecting based on other criteria?
                                    Timing of      Is it better to select claimants as soon as
                                    Claimant        possible (i.e., after the first payment has been
                                    Selection       made) or later in the life of the claim?

Component Sub-component                             Specific Evidence to Build
                          Scheduling         How soon after claimant selection should
                          RESEA               RESEA meetings be required to occur?
                          Meetings           What scheduling strategies are most likely to
                                              ensure the claimant fully participates in all
                                              RESEA services? Does getting the claimant to
                                              initially show up increase the likelihood of full
                                              participation in RESEA services?
                                             What is the effect of having multiple RESEA
                                              meetings, rather than just one?
                          Reemployment       What strategies are most effective to support
                          services            development of a reemployment plan that the
                                              claimant “owns” and implements?
                                             What is the impact of more “basic” assistance
                                              (e.g., American Job Center orientation, self-
                                              directed use of online tools, general labor
                                              market information) vs. the impact of more
                                              intensive, individualized career services?
                                             What is the impact of more intensive one-on-
                                              one career counseling?
                                             What are effective strategies for delivering job
                                              search assistance?
Reemployment Activities

                                             Are there types of training that can help get
                                              individuals to work quickly (e.g., on-the-job
                                              training, apprenticeship)?
                                             Are there particular assessments or other ways
                                              of identifying claimants’ needs that create a
                                              more effective reemployment plan, better
                                              connects claimants with services, and
                                              ultimately leads to more positive outcomes?
                                             What is the effect of case management? Is
                                              increased frequency of intensive case
                                              management (e.g., more regular contact) more
                                              effective?
                          Activities to      What strategies that support a claimant’s work
                          support work        search compliance impact employment
                          search              outcomes?
                          compliance         What is the effect of review of continued
                                              eligibility for benefits on the claimant’s
                                              employment outcomes?
                                             How do the strength, immediacy, and surety of
                                              penalties for failure to attend (FTA) affect job
                                              search efforts and outcomes?

8. Criteria for Causal Evidence Ratings.

    Section 306(c)(2), SSA, conditions funding for RESEA programs on states using
    interventions either demonstrated as effective with a “high or moderate causal evidence
    rating” or being under evaluation. Beginning in FY 2020, the definitions established below
    will be in effect and explain how an intervention can qualify for a high rating or a moderate
    rating.4 These ratings examine available evidence and determine whether the interventions
    have favorable impacts on both employment and benefit duration outcomes.5 The high and
    moderate causal evidence standards described below rely on evidence of impact exclusively
    from studies that received a high or moderate rating for study quality in CLEAR. These
    studies are identified in the definitions as credible studies.
             High: For an intervention to qualify for a high causal evidence rating, there must be
              at least two credible impact studies of the intervention (as reviewed by CLEAR) that
              have each found favorable impacts on employment and UC duration, with a strong
              degree of statistical confidence.6
             Moderate: For an intervention to qualify for a moderate causal evidence rating, there
              must be at least one credible impact study of the intervention that found a favorable
              impact on employment and one credible impact study of the intervention that found a
              favorable impact on UC duration. Again, these ratings of the study or studies are as
              reviewed by CLEAR. Each study must have at least a modest degree of statistical
              confidence.7 The findings on employment and benefit duration may both come from
              the same study or from different studies.
    DOL also defines two additional categories: “potentially promising” and “no rating.”
             Potentially Promising: A potentially promising rating indicates that there is some
              suggestive evidence that an intervention may be effective. Such interventions are
              candidates for further evaluation that possibly would allow the intervention to qualify
              for a higher rating. For an intervention to qualify for a potentially promising causal
              evidence rating, there must be one impact study reviewed by CLEAR (irrespective of

4
 As the evidence base grows, more information will be available to help distinguish which approaches have the
strongest evidence of effectiveness. At that time, the standards for evidence of effectiveness may evolve as well, in
order to help better support those distinctions.
5
  Specifically, the ratings criteria are based on interventions’ estimated impacts on (1) reduced UC duration and (2)
increases either employment rates or earnings, as measured in the second full calendar quarter after the claim began.
6
 A “strong statistical confidence” is defined as an estimated impact that is “statistically significant” (different from
zero) at the 5% level. (p<.05). This means that there is less than a 5% chance that the study’s results are due to
chance and not actually the intervention. Impact estimates must meet that threshold for both outcomes—UC
duration and employment.
7
 A “modest degree of statistical confidence” is defined as an estimated impact that is “statistically significant”
(different from zero) at the 10% level. (p<.10). This means that there is less than a 10% chance that the study’s
results are due to chance and not actually the intervention. Impact estimates must meet that threshold for both
outcomes—UC duration and employment.

             the causal evidence rating it received)8 that has found significant favorable impacts on
             either employment or UC duration with at least a moderate degree of statistical
             confidence.9

            No Rating: All interventions that do not qualify for any of the three ratings above
             will receive no rating, regardless of the rating given by CLEAR for the quality of
             studies of that intervention. These may be interventions for which no impact studies
             have been conducted, interventions with an impact study that have not been reviewed
             by CLEAR yet, or interventions whose studies have been reviewed by CLEAR but
             have not shown any favorable impacts.

9. High and Moderate Causal Ratings for Existing and Future Interventions Beginning in
   FY 2020.

     Existing impact studies of approaches to speed the reemployment of UC claimants typically
     focus on broadly defined sets of services and activities. CLEAR’s 2018 Reemployment
     Research Synthesis on reemployment interventions10 identified the following broad
     intervention categories from the existing evidence base that are relevant to RESEA:11
            Reemployment and Eligibility Assistance (REA): The REA program, the predecessor
             to RESEA, provided claimants up to three mandatory in‐person sessions in which
             workforce staff assessed their continued eligibility for UC, provided them with labor
             market information, and supported their development of a reemployment plan. In
             some cases, they also provided referrals to reemployment services. Failure to attend
             REA sessions without good cause affects continued receipt of UC.
            Job Search Assistance (JSA): JSA interventions provide claimants assistance and
             training in job search techniques, including job search workshops, preparing a
             resume, and interview training. The JSA interventions that included strong linkages

8
  CLEAR also rates some studies as low. These studies are not used when considering whether an intervention is
eligible for a high or moderate effectiveness rating. However, studies rated as low can contribute to a potentially
promising rating. The potentially promising rating indicates that some suggestive evidence exists that an
intervention might be effective. While evidence from a low rated study is not a strong basis for concluding that an
intervention is effective, it can suggest that the intervention may be worth considering for more rigorous testing.
Some studies that are rated as low may still be considered promising and thus, a candidate for further evaluation.
9
 As noted earlier, for moderate effectiveness ratings, a “modest degree of statistical confidence” is defined as an
estimated impact that is “statistically significant” (different from zero) at the 10% level.
10
  Find CLEAR’s Reemployment Research Synthesis here:
https://clear.dol.gov/sites/default/files/CLEAR%20Reemployment%20Synthesis%20November%202018.pdf
11
  All descriptions are adapted from CLEAR’s 2018 research synthesis, What do we know about the effect of
reemployment initiatives?, which can be found at:
https://clear.dol.gov/sites/default/files/ResearchSynthesis_Reemployment.pdf

       between UI and workforce partners and required claimants at risk for benefit
       exhaustion to report for job search assistance demonstrated positive impacts.
      Profiling: Profiling interventions identify claimants at higher risk of exhausting UC
       and offer or require enhanced employment services. These services may include an
       orientation, providing labor market information, and referrals to job search training or
       resume training workshops. Claimants that fail to participate in required services
       without good cause lose UC.
      More Stringent Employer Contact Requirements: This type of intervention increases
       the amount of work search effort required of claimants to continue receiving UC,
       strengthen verification of work search efforts, or both.
      Less Stringent Employer Contact Requirements: This type of intervention reduces
       the amount of work search effort required of claimants to continue receiving UC,
       loosen verification of work search efforts, or both.
These broadly defined interventions often involve partially overlapping services and
activities. Beginning in FY 2020, these interventions will receive effectiveness ratings using
the definitions above. To the extent that the states’ programs use interventions that have not
received high or moderate evidence ratings, those states must be conducting high-quality
impact evaluations using the CLEAR guidelines for study quality.
Interventions Receiving a High Rating

Of the interventions considered, only REA currently receives a high causal evidence rating.
If a state’s RESEA program has components that are sufficiently similar to the evaluated
REA program components, a state can demonstrate that those pieces or components of its
RESEA program are evidence-based, by referring to this intervention and its rating. While
no evaluation of REA components that are part of a state’s RESEA program is required,
states are encouraged to continue to evaluate these interventions in order to build rigorous
evidence in the context of the new RESEA program. As noted above, RESEA is not
identical to the REA program and has different components, so REA interventions alone will
not be sufficient to meet RESEA requirements, and states should continue to consider
implementing new interventions. If a state’s RESEA program includes other components
that are not evidence-based, those components must be under evaluation at the time of use.

Interventions Receiving a Moderate Rating

Applying the criteria above, JSA and profiling interventions receive a moderate causal
evidence rating. If a state’s RESEA program includes components like the JSA and profiling
strategies described above in this guidance, a state can demonstrate the corresponding
components of its RESEA program are evidence-based by referring to these components and
interventions and their ratings. Again, while no evaluation of JSA components of a state’s
RESEA program are required, states are encouraged to continue to evaluate their
interventions in order to continue to grow the base of evidence regarding their use in the
RESEA program. Additional evidence on these interventions is still valuable and could
result in raising the causal evidence rating for the interventions to the high category. If a
state’s RESEA program includes other components that are not evidence-based, those
components must be under evaluation at the time of use.

   Interventions Receiving a Potentially Promising Rating

   Applying the criteria above, the component of requiring more stringent employer contacts
   receives a potentially promising causal evidence rating. This rating indicates that the
   component or intervention may be of interest to consider adopting or testing, as it might be
   effective. Additional evidence on these interventions might also support a change of causal
   evidence rating for the intervention. States implementing interventions with only a
   potentially promising rating must be evaluated at the time of use.

   Interventions Receiving No Rating

   Applying the criteria above, the component of less stringent employer contacts receives no
   rating. Additionally, the more detailed components included in Exhibit 1, as well as any
   additional RESEA interventions or program components not identified here as being
   demonstrated effective by current evidence, also currently receive no rating. Such
   interventions must be evaluated if states choose to implement them.

10. Evaluation Approaches.

   Section 306(c), SSA, gives states time to evaluate RESEA interventions before the
   percentage requirements for use of interventions with high or moderate causal ratings begin
   in FY 2023. States’ evaluations will need to meet the causal evidence standards described in
   Section 8 of this guidance. To help states’ impact evaluations have the best chance of
   meeting CLEAR’s standards, states are strongly encouraged to: (1) choose an experienced
   evaluator; (2) choose a simple impact study design (the simplest being random assignment
   with administrative data follow-up); and (3) take advantage of the evaluation technical
   assistance (EvalTA) guidance being provided (described in more detail in Section 13, below).

   Importantly, there are multiple types of evaluations and evaluation-related activities that
   ultimately support a strong impact evaluation that produces high or moderate causal
   evidence. For example, it may be appropriate to conduct an evaluability assessment
   (discussed in more detail below) or feasibility study before embarking on an impact
   evaluation to identify any challenges or barriers, such as data availability or limited sample
   size, to conducting an evaluation of a specific intervention or service delivery design.
   Alternatively, it may be desirable to pair both an implementation evaluation and an impact
   evaluation. DOL considers activities leading up to an impact evaluation that has the
   capability of producing a high or moderate causal rating to be interventions designated as
   “under evaluation.” Examples of these activities include an evaluability assessment as
   described below, and an implementation study that helps refine the specific intervention and
   research questions to be addressed in the impact evaluation.

   Evaluation Design

   DOL encourages states’ evaluation designs to specify use of, or be building evidence to
   move toward the use of, approaches capable of earning high or moderate quality ratings, with

the goal of producing both the strongest possible evidence and the highest possible causal
evidence rating for the intervention being studied. As noted previously, impact evaluations
are necessary to achieve these ratings; however, there may be other types of evaluations or
pre-evaluation activities that should be conducted first or along with an impact study to
maximize learning about the intervention.

The most common types of evaluation designs are:
       Impact Evaluation: This type of evaluation assesses the impact of a program or
          component of a program on outcomes, typically relative to a counterfactual
          situation. This evaluation provides some estimate of what would have happened
          in absence of the program or component of the program. Impact evaluation
          includes both experimental (i.e., randomized controlled trials) and quasi-
          experimental designs. These types of evaluations speak to the "does it work?"
          question.
       Outcome Evaluation: This type of evaluation measures the extent to which a
          program has achieved its intended outcome(s), and focuses on outputs and
          outcomes to assess effectiveness. Unlike an impact evaluation, an outcome
          evaluation cannot show causal impacts. An outcome evaluation can help answer
          questions like, “Did the program, policy, or organization do what it intended to
          do?”
       Process or Implementation Evaluation: These types of evaluations assess how the
          program or service is delivered relative to its intended theory of change, and often
          include information on content, quantity, quality, and structure of services
          provided. Process or implementation evaluations can be conducted on their own,
          but are often conducted along with impact and/or outcomes evaluations. Process
          or implementation evaluations can help answer questions like, “Was the program
          or policy implemented as intended?” or “How is the program, policy, or
          organization operating in practice?”
       Formative Evaluation: This type of evaluation, typically done before full
          implementation of a program, assesses whether a program or component of a
          program is feasible, appropriate, and acceptable before it is fully implemented. It
          may include some of the activities described above, such as process evaluation or
          outcome evaluation. However, unlike summative evaluation designs like impact
          and outcome evaluations, which seek to answer whether or not the program met
          its intended goal(s) or had the intended impact(s), a formative evaluation focuses
          solely on learning and improvement and does not answer questions of overall
          effectiveness.

Selecting an Evaluator

While DOL recognizes there is value in all types of evaluations, the RESEA evidence-
generating approach specifically requires impact evaluations of interventions to help
determine causal evidence ratings for those interventions. While states may have evaluation
capacity within the agency operating the RESEA program, DOL recommends that states use
an experienced and independent evaluator that can identify the most appropriate and rigorous

design to answer research questions and learn about the RESEA program and program
components and interventions.

Deciding What to Evaluate

DOL recognizes that each state’s RESEA program is a uniquely implemented bundle of
different interventions and service delivery strategies, or components. However, this can
make it difficult to know which program components, and how these components, are
generating the observed outcomes of the intervention. Therefore, states are strongly
encouraged to work with their independent evaluators to develop evaluations that seek to
estimate the impact of individual RESEA program components and interventions, or to
develop evaluations of whole programs where the components are well defined and the
effectiveness of which could be evaluated at a later time through meta-analyses. Building
this type of evidence will further states’ understanding of the effectiveness of components
that could be mixed and matched to develop a program that meets the needs of a specific
state. However, DOL recognizes that evaluating only a component of the program implies
the need for the evaluation to detect smaller impacts, which requires much larger samples.
Again, states are encouraged to work with experienced evaluators and explore partnerships
with other states to develop the most rigorous and appropriate designs to determine the
effectiveness of program components. See Section 11, below, for more discussion on
evaluation partnerships across states.

Pre-evaluation activities states can begin doing now to support getting to a firm evaluation
plan include the following activities:

      States are encouraged to conduct evaluability assessments of their RESEA
       programs. Evaluability assessments will help states: define specific interventions
       that are evaluation-ready to test in a feasible, measurable way; ensure that the
       intervention and the component(s) to test are well-understood by all stakeholders;
       confirm availability of data and other operational resources needed to conduct an
       evaluation; and build consensus on evaluation goals to ensure results are relevant to
       stakeholders. Evaluability assessments also are useful for identifying a program’s
       potential strengths and challenges with planning and executing an evaluation. For
       example, they may assess whether adequate access to information technology (IT)
       and data resources exist and are available to support the evaluation or if program staff
       has sufficient evaluation expertise. The results of an evaluability assessment refine a
       state’s broad learning goals with more narrowly-focused research questions that
       explore the RESEA program’s influence on a particular population’s outcomes of
       interest, and identify the type of evaluation that can best answer these questions.
       Evaluability assessments highlight evaluation feasibility issues, such as operational
       gaps that must be addressed to successfully execute the evaluation, such as
       availability of IT resources and data availability, staffing to increase evaluation
       capacity, developing partnerships with organizations that have appropriate evaluation
       expertise, and creating evaluation procedures and training staff. Addressing these
       operational gaps identified through an evaluability assessment strengthens a state’s
       ability to produce a high-quality evaluation that meets CLEAR standards. States can

               find an Evaluation Design Assessment Tool developed by IMPAQ International to
               support WIOA evaluations here:
               https://evalhub.workforcegps.org/resources/2018/09/07/19/53/Evaluation-Design-
               Assessment-Tool
              States are also encouraged to develop logic models when formulating evaluation
               plans. Logic models are graphical representations of interventions and how they
               operate. They are designed to show the following regarding a RESEA intervention:
                       Inputs – such as staff time, RESEA funds, and other resources used to deliver
                        the program;
                       Activities – such as meeting with American Job Center (AJC) staff to create
                        an individual reemployment plan, provision of reemployment services,
                        conducting the eligibility assessment, and other activities the program
                        regularly operates;
                       Outputs (the immediate results of the program) – such as improved job
                        readiness skills or enhancing labor market knowledge; and
                       Outcomes (the expected short-term and long-term goals of the program) –
                        such as reduced UC duration, faster return to employment, and improved
                        earnings.
               Logic models define the inputs, activities, or other tangible activities that lead to
               outputs and outcomes for the RESEA program. Logic models and other similar
               program mapping activities demonstrate how RESEA interventions drive the change
               in outcomes for claimants. States can find more information about developing logic
               models for labor programs in a webinar titled Fully Articulating Your Vision: Using
               Logic Models to Support Innovation:
               https://evalhub.workforcegps.org/sitecore/content/global/resources/2015/05/07/11/07/
               Fully_Articulating_Your_Vision_Using_Logic_Models_to_Support_Innovation
              States’ evaluability assessments and logic models ultimately help states identify
               specific research questions that may be added to a multi-year learning strategy or
               agenda. Organizing learning priorities is an approach that is gaining traction across
               the Federal government, most recently supported in the Foundations for Evidence-
               Based Policymaking Act of 2018 (Public Law No. 115-435),12 which requires
               Federal government agencies to produce evidence-building plans. Learning agendas
               can also serve as roadmaps to help states plan for immediate and future evaluations,
               by clarifying learning goals, research questions, the types of evaluations that would
               answer those questions, and the states’ priorities in building evidence.

11. Strategies to Meet RESEA Evaluation Requirements.

       Some states may be interested in conducting their own individual impact evaluations. As
       indicated previously in UIPL No. 7-19, DOL encourages states to consider evaluation

12
     https://www.congress.gov/bill/115th-congress/house-bill/4174

   partnerships with other states, so that states may consider conducting pooled evaluations of
   similar RESEA interventions. This approach has the benefit of potentially yielding sample
   sizes large enough to demonstrate effectiveness. Smaller states, in particular, might benefit
   from this strategy. It also has the benefit of allowing states to pool their limited evaluation
   funding.

   From previous research, we know that sample sizes required to detect impacts on labor
   market outcomes of the kind required by section 306, SSA, are large. DOL recognizes that
   many states do not have a sufficiently large number of RESEA-eligible claimants in a single
   year, and some states do not have that many RESEA-eligible claimants in several years. A
   pooling strategy, including a well-defined intervention aligned across states, can help
   overcome this challenge. States can pool their data and yield samples large enough to detect
   effects and potentially demonstrate effectiveness on a more reasonable timeframe. As
   mentioned previously, states are encouraged to work with experienced evaluators and explore
   partnerships with other states to develop the most rigorous and appropriate study designs to
   evaluate program components and interventions.

12. Evaluation Resources.

   In addition to the tools linked to above, DOL’s CLEAR also has several additional tools to
   help states better understand the evidence in the Reemployment topic area.
        CLEAR’s Reemployment Synthesis is a short, high-level, plain-language report that
           summarizes studies of interventions that are relevant to RESEA. It describes key
           takeaways from the reemployment evidence base, provides an overview of the
           interventions studied, and identifies gaps in the research. This report may be useful
           as RESEA program managers begin to focus on conducting evaluability assessments
           and efforts to build the evidence base.
        As a companion to the Synthesis, CLEAR’s Reemployment Synthesis Supplement
           gives states a more detailed look at the information provided in the synthesis. It
           provides brief descriptions of the findings for all the reports reviewed in the
           Reemployment topic area. This supplemental tool also includes links that lead
           directly to the study profiles in CLEAR where more information about the specific
           studies and interventions is available. It also is organized by sections that correspond
           to the intervention categories identified in the Reemployment Synthesis that will
           receive ratings beginning in FY 2020, as described above.

   CLEAR and its resources are at the links below:
      CLEAR: https://clear.dol.gov/
      Reemployment topic area: https://clear.dol.gov/topic-area/reemployment
      Reemployment Synthesis landing page: https://clear.dol.gov/synthesis-
        report/reemployment-synthesis, and a downloadable brief report:
        https://clear.dol.gov/sites/default/files/ResearchSynthesis_Reemployment.pdf.
      Reemployment Synthesis Supplement:
        https://clear.dol.gov/sites/default/files/ResearchSynthesis_Reemploy_Sup.pdf.

Another important resource for states will be DOL’s RESEA Evidence Building and
Implementation Study. In September 2018, DOL’s Chief Evaluation Office awarded a
three-year contract to Abt Associates, the Urban Institute, Capitol Research Corporation, and
National Association of State Workforce Agencies (“the RESEA study team”) to provide
support on implementing the evaluation requirements in section 306, SSA. Among the tasks
planned for the RESEA study team is an implementation study of states’ RESEA programs.
DOL is conducting this implementation study to examine how RESEA programs and
strategies are operated, understand how states are bundling various services to improve
outcomes for RESEA participants, and identify new, innovative, and potentially promising
strategies being implemented. Findings from this implementation evaluation will inform an
evaluation report that will further describe research and evaluation options for DOL and
states to consider and will contribute to the RESEA evidence base. Reports from the study
will be publicly available when completed. A brief description of DOL’s RESEA study on
the Chief Evaluation Office’s website is available here:
https://www.dol.gov/asp/evaluation/currentstudies/Reemployment-Services-and-Eligibility-
Assessments-Research.htm.

Finally, it is DOL’s intent that states also leverage other available evaluation capacity-
building resources. These include, but are not limited to:
     Evaluation and Research Hub: A new community of practice, created with input
         from state and local workforce agency representatives across the country. While it is
         available to address the evaluation requirements of the WIOA, the resources included
         on the Hub can inform or support the evaluation needs of all ETA-funded programs.
         You can find it here: https://evalhub.workforcegps.org/about
     WIOA Evaluation Technical Assistance Tools: State and local workforce agencies
         participated in ETA’s peer learning effort to share and disseminate evaluation
         resources as well as address questions such as, “Where and how do we start?” Key
         tools are included here:
              o Evaluation Readiness Assessment Tool:
                  https://evalhub.workforcegps.org/resources/2018/09/07/19/45/Evaluation-
                  Readiness-Assessment-Tool
              o Evaluation Design Assessment Tool:
                  https://evalhub.workforcegps.org/resources/2018/09/07/19/53/Evaluation-
                  Design-Assessment-Tool
              o Evidence Says: Work-based Learning:
                  https://evalhub.workforcegps.org/resources/2018/09/07/16/10/The-Evidence-
                  Says-Work-Based-Learning
     Workforce System Strategies (WSS): A research clearinghouse that profiles
         evidence-based and emerging practices in workforce development to help the field
         make informed decisions about improving outcomes for job seekers and employers.
         Its resource library contains more than 1,200 profiles of evaluation reports, policy
         and practice briefs, and how-to guides. It is available here:
         https://strategies.workforcegps.org/announcements/2018/05/04/20/17/Connect-Your-
         Peers-to-Workforce-System-Strategies.

13. Evaluation Technical Assistance (EvalTA).

   As described above, a critical piece of the DOL’s RESEA project is to provide EvalTA to
   states. The EvalTA will include a suite of tools and resources to help states meet evaluation
   and evidence-building needs for their RESEA programs. Experienced staff from the RESEA
   study team will develop and deliver EvalTA.

   Beginning in summer 2019, the RESEA study team is offering generalized EvalTA, which
   has been informed by state feedback from webinars, clarifying calls, and a review of
   available documents on states’ FY 2019 RESEA programs. The goals of this generalized
   EvalTA is to help states with the following: 1) gradually and continually increase their
   evaluation capacity so states are prepared to begin evaluation-related work by FY 2020; 2) to
   describe evaluation activities in their FY 2020 RESEA state plans; and 3) to meet evidence-
   related statutory requirements, both now and moving forward. The EvalTA will be provided
   through resources DOL will make broadly available, such as webinars, toolkits, briefs,
   templates, and videos, to explain key topics to improve states’ understanding of basic
   evaluation concepts and begin to plan and carry out evaluations. These resources will build
   on existing DOL evaluation technical assistance resources (e.g., for WIOA, as described
   above) and focus particularly on knowledge required for evaluations of RESEA program
   components and interventions that can meet evidence standards. When they are finalized, a
   schedule of EvalTA activities, as well as all resources developed through the EvalTA, will be
   available on the Reemployment Connections community of practice on WorkforceGPS
   (www.workforcegps.org).

   In addition to this generalized EvalTA, the RESEA team will also offer customized EvalTA
   to individual states or small groups of states, as appropriate, that are planning or carrying out
   evaluations. Customized EvalTA is anticipated to begin in fall 2019 and is likely to include
   detailed verbal and written technical assistance to states at key points during individual
   evaluations. The RESEA study team may provide customized EvalTA in areas such as
   procurement and selection of an independent evaluator; selection of methods and
   development of evaluation design plans; monitoring random assignment and dealing with
   unanticipated issues; methods of analysis and reviews of analysis plans; reporting and
   dissemination; and other issues as appropriate and needed. Plans for customized EvalTA will
   be updated by DOL’s RESEA project yearly, as states’ needs are better understood and as
   new RESEA interventions and evaluations are planned.

   While not every state is expected to need or participate in customized EvalTA, all states are
   encouraged to take advantage of the generalized EvalTA being offered. Previous experience
   from other tiered evidence initiatives across the government suggest that an adequate
   planning period combined with a comprehensive EvalTA strategy can improve evaluation
   quality. Together, these efforts can help meet one of the primary goals of this early phase of
   RESEA implementation, to expand the evidence base by supporting states in conducting new
   high-quality evaluations of RESEA program components and interventions.

14. Inquiries. For further information, please direct inquiries to the appropriate ETA Regional
    Office.

15. References.

          Section 306, Social Security Act, 42 U.S.C. § 506;
          The Bipartisan Budget Act of 2018, Public Law No. 115-123;
          The Foundations for Evidence-Based Policymaking Act of 2018, Public Law No.
           115-435; and
          Unemployment Insurance Program Letter No. 07-19, Fiscal Year (FY) 2019 Funding
           Allotments and Operating Guidance for Unemployment Insurance (UI)
           Reemployment Services and Eligibility Assessment (RESEA) Grants, issued January
           11, 2019.

16. Attachment(s). Not Applicable.