Customer satisfaction survey methodology changes

U. S. Department of Labor
Employment and Training Administration
Washington, D.C. 20210
CLASSIFICATION: WIA/Performance Accountability
CORRESPONDENCE SYMBOL: OWS
DATE: October 31, 2001

TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 6-00, CHANGE 1

TO: ALL STATE WORKFORCE LIAISONS
ALL STATE WORKFORCE AGENCIES
ALL ONE-STOP CENTER SYSTEM LEADS
ALL STATE WORKER ADJUSTMENT LIAISONS

/s/
FROM: EMILY STOVER DeROCCO
Assistant Secretary

SUBJECT: Changes to Guidance on Capturing Customer Satisfaction Results for Participants and Employers Receiving Services Under Title I of the Workforce Investment Act .

1. Purpose. To provide revisions to technical guidance for states to implement the core and customer
   satisfaction performance measures and calculate performance levels required under the Workforce Investment Act in order that the guidance contained in Training and Employment Guidance Letter (TEGL) No. 6-00 conform to the requirements contained in TEGL No. 14-00.

2. References. Workforce Investment Act of 1998 (WIA), section 136; WIA Final Rules, 20 CFR
   part 666; TEGL No. 7-99; TEGL No. 8-99; TEGL No. 6-00; TEGL No. 14-00; Planning Guidance and Instructions for Submission of the Strategic Five-Year Plan for Title I of WIA and the Wagner- Peyser Act; WIA: Final Unified Planning Guidance; Notice of Office of Management and Budget (OMB) Action No. 1205-0420, dated March 1, 2001.

3. Background. Section 136 of WIA specifies core indicators of performance and their application to
   workforce investment activities in adult, dislocated worker, and youth programs. In TEGL No. 7-99, the Department issued technical guidance to states to use when calculating outcomes on the 15 core performance measures and the two customer satisfaction performance measures. Four core measures apply separately to the adult, dislocated worker and older youth participants; three measures apply to younger youth; and two measures of customer satisfaction apply across the three funding streams.

RESCISSIONS: None
EXPIRATION DATE: Continuing

TEGL No. 6-00 outlined the parameters states are to use in conducting customer satisfaction surveys at the state level and detailed the parameters states are to use in conducting surveys and reporting results for local workforce investment boards (WIBs). Formal instructions for reporting performance outcomes (including customer satisfaction results) to the Department were released in TEGL No. 14-00 as part of the overall WIA performance accountability reporting system approved by OMB on March 1, 2001. The approved performance reporting system contained several changes to the customer satisfaction survey methodology in response to comments forwarded to OMB by states. These changes increase the methodological rigor of the customer satisfaction surveys.

This guidance letter outlines changes to the parameters detailed in TEGL No. 6-00 that are needed to conform to the increased methodological rigor published in TEGL No. 14-00.

4. Changes to the Customer Satisfaction Survey Methodology.

A. Revisions to Original Guidance on Conducting Surveys to Obtain State Level Results
• For both the participant and employer surveys, states must achieve at least 500
completed interviews per group and meet the minimum required response rates. For the
first year of data collection only, there is a minimum response rate of 50% for the
employer survey and 50% for the participant survey for determining the usefulness of
the data for performance measurement purposes. Response levels below 50% during
the first year will invalidate the results for performance measurement purposes as a
significant bias in the results is likely. For the following years of data collection, states
are required to achieve response rates of 70% for both the participant survey and the
employer survey.

           [Affected item(s) in TEGL No. 6-00: Section 5, Parameters for Conducting
           Surveys to Obtain State Level Results, items D and F on page 4.]

• States may randomly sample participants and employers to obtain the desired number
of completed surveys. States are required to determine the appropriate sample sizes
and sampling percentages using the required response rate and the required number of
completed interviews. Original guidance required states to
draw minimum sample sizes of 1,000 for the participant survey and 1,000 for the
employer survey.
[Affected item(s) in TEGL No. 6-00: Section 5, item E on page 4.]
• Respondents must be told that responding to the survey is voluntary and that the
information they provide will be kept confidential. States must not make any assurances
regarding confidentiality without ensuring that they have the legal authority to make such
assurances. Original guidance stated that sharing information on the voluntary nature of the
survey and the confidentiality of results with respondents was optional for states.

           [Affected item(s) in TEGL No. 6-00: Section 5, item C on page 4.]

• States must create a process for creating an up-to-date customer list, capturing
information on each customer’s address and telephone number. Original guidance from
the Department encouraged, but did not require, states to create customer lists in a
timely fashion prior to selecting samples for the surveys.

[Affected item(s) in TEGL No. 6-00: Section 5, item H on page 5.]

• The sampling frame used to obtain state-level results and the sampling frame used to
obtain local area results includes all exiters and employers eligible for the survey, with
the exception of those who are deceased, institutionalized, incarcerated, or unable to
complete the survey due to illness or injury. Original guidance from the Department
stated the sampling frame only included participants and employers eligible for the
surveys who had valid contact information.

The response rate is calculated as the number of respondents with complete customer
satisfaction information divided by the total number in the sample size. If sampling is not
used, the response rate is calculated as the number of respondents with complete
customer satisfaction information divided by the total number eligible for the survey (also
known as the sampling frame). A survey is considered complete when valid answers
are provided by the respondents for each of the core American Customer Satisfaction
Index (ACSI) questions. Valid answers are responses 1 through 10 for each of the
three ACSI questions. Responses 11 (don’t know) and 12 (refused) are considered
missing values and are not considered valid responses for computation purposes.

The calculation of the ACSI includes only the results from complete surveys.

[Affected item(s) in TEGL No. 6-00: Section 5, item F on page 4.]

• The introductory statements for both the participant survey and the employer survey
have been revised to better set the context for the interviews. These revised
introductory statements are to be read to each respondent at the beginning of the
interviews. The wording of an introductory statement may be modified to meet the
needs of the state. In addition to contact information, survey administrators will need to
know the service(s) provided to each customer, the date(s) of service, and the firm(s)
providing the service(s) in order to complete the interviews. The service(s) should be
worded in terms recognizable to the customer.

The following introductory script replaces the introductory statements for both the
participant and employer surveys published in the original guidance:

[Introductory Script]

My name is (interviewer) with (firm or agency) and I am conducting a survey for the
(program name). I would like to speak to Ms./Mr. (name of participant or name of


       employer contact).

       Are you the Ms./Mr. (name of participant or name of employer contact) who received a
       (name of service(s)) from the                   on (date or date range)? [IF THE
       RESPONDENT CONFIRMS THE RECEIPT OF SERVICES, PROCEED WITH THE INTERVIEW. IF
       NO, END THE INTERVIEW AND THANK THE RESPONDENT FOR HIS OR HER TIME.]


       I would like to ask you some questions about your recent experience with the services
       provided to you [SUBSTITUTE AYOU@ WITH AYOUR BUSINESS@ FOR EMPLOYER
       RESPONDENTS]. Our purpose is to learn from you how to improve programs and
       services offered to people [SUBSTITUTE APEOPLE@ WITH AAREA BUSINESSES@ FOR
       EMPLOYER RESPONDENTS]. The survey was approved by the Office of Management
       and Budget to collect information on your experience with services we provide to
       individuals [SUBSTITUTE AINDIVIDUALS@ WITH AEMPLOYERS@ FOR EMPLOYER
       RESPONDENTS]. The questionnaire is voluntary and should take no longer than XX
       minutes to complete.

       Please know the answers you provide to my questions will be kept confidential. Do
       you have any questions for me before we start the interview? [IF NO, CONTINUE. IF
       YES, PROBE TO UNDERSTAND THE QUESTIONS, PROVIDE ANSWERS AND RESTATE THE
       IMPORTANCE OF THE RESPONDENT’S PARTICIPATION IN THE INTERVIEW.]


       First, I want to know your overall experience with the services. I will read to you three
       statements and ask you to rate your experience. [PROCEED WITH THE CORE ACSI
       QUESTIONS.]


       [Affected item(s) in TEGL No. 6-00: Section 5, item J on pages 5 through 7]

B. Additions to Original Guidance on Conducting Surveys to Obtain State Level Results

• Local program staff must inform the customer during the registration process about the importance of satisfying customers and the possibility of being contacted for information on his or her experience with the services.
• The interview should be limited to 15 minutes or less. Shorter questionnaires produce better response rates when compared to longer questionnaires.
• A minimum of five follow-up attempts is required, including contact at various times of the day before closing the record.
• Local programs should collect alternate contact information for a person known by the customer who would know the whereabouts of the customer in the event the customer cannot be reached at the address and telephone number recorded for the customer.
• A letter in advance of the survey should be sent out informing the customer that he or she can expect to receive a telephone call about his or her satisfaction with the services.
  States should place a phone number on the letter, suggesting the customer call if he or she needs help or call if his or her home phone number changed.

[The above items should be added to section 5, Parameters for Conducting Surveys to Obtain State Level Results, in TEGL No. 6-00.]

C. Updated Table of Weights to Use in Calculating the ACSI

Updated weights were calculated for use in PY 2001 by a statistical algorithm in order to minimize random survey noise or measurement error that exists in all survey data. Please refer to TEGL No. 6-00 for a description of the formulas for calculating the ACSI scores.
Specific weighting factors were developed for each state for PY 2001 and can be found attached to this TEGL. New weighting factors will be published annually. The weights listed in the attachment are to be used by each state in computing performance on the ACSI for employers and participants.

5. Action Required. States shall distribute this guidance letter to all officials within the state who need
   such information to implement WIA core and customer satisfaction measures.

6. Inquiries. Questions concerning this issuance may be directed to your appropriate Regional Office.

7. Attachments. ACSI Formula Weights

Weights for Use in Calculating State-Level ACSI Scores September 14, 2001

Weights
State Overall Confirm Compared to Name Satisfaction Expectations Ideal
AK 0.3621 0.2650 0.3729
AL 0.4411 0.3096 0.2493
AR 0.4366 0.3213 0.2421
AZ 0.4047 0.3198 0.2755
CA 0.4244 0.3238 0.2518
CO 0.3974 0.3304 0.2722
CT 0.4328 0.3407 0.2265
DC 0.4879 0.3208 0.1912
DE 0.4430 0.3076 0.2494
FL 0.4331 0.3054 0.2615
GA 0.4252 0.3076 0.2672
HI 0.1958 0.6136 0.1906
IA 0.3960 0.3190 0.2850
ID 0.3769 0.3236 0.2995
IL 0.4023 0.3222 0.2755
IN 0.4274 0.3360 0.2366
KS 0.4092 0.3426 0.2482
KY 0.4263 0.3030 0.2707
LA 0.4350 0.3078 0.2573
MA 0.4188 0.3283 0.2529
MD 0.4215 0.3375 0.2410
ME 0.4526 0.2804 0.2670

Weights
State Overall Confirm Compared to
Name Satisfaction Expectations Ideal
MI 0.4223 0.3237 0.2539
MN 0.4232 0.3425 0.2342
MO 0.4288 0.3300 0.2412
MS 0.4240 0.3171 0.2589
MT 0.3908 0.3488 0.2604
NC 0.4113 0.3311 0.2577
ND 0.4297 0.2790 0.2913
NE 0.4046 0.3375 0.2579
NH 0.4266 0.3247 0.2487
NJ 0.4309 0.3371 0.2320
NM 0.4312 0.3193 0.2495
NV 0.3810 0.3493 0.2698
NY 0.4129 0.3336 0.2535
OH 0.4217 0.3251 0.2532
OK 0.4382 0.3111 0.2507
OR 0.4247 0.3274 0.2479
PA 0.4314 0.3287 0.2399
RI 0.3932 0.3498 0.2570
SC 0.4551 0.2986 0.2464
SD 0.4039 0.3263 0.2698
TN 0.4314 0.3177 0.2509
TX 0.4186 0.3313 0.2501
UT 0.4063 0.3323 0.2615
VA 0.4313 0.3111 0.2576
VT 0.4225 0.3266 0.2509
WA 0.4059 0.3244 0.2697
WI 0.4282 0.3133 0.2585
WV 0.4408 0.3145 0.2447
WY 0.4488 0.3272 0.2240

Source: National Quality Research Center, University of Michigan Business School