2011 WIA customer satisfaction survey weights

Accessible letter PDF: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2013/TEGL_12_12_Acc.pdf

CLASSIFICATION
   EMPLOYMENT AND TRAINING ADMINISTRATION                   ACSI
               ADVISORY SYSTEM                              CORRESPONDENCE SYMBOL
                                                            OWI/DAS
          U.S. DEPARTMENT OF LABOR
                                                            DATE
             Washington, D.C. 20210
                                                            January 7, 2013


TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 12-12

TO:                ALL STATE WORKFORCE AGENCIES
                   ALL STATE WORKFORCE LIAISONS

FROM:              JANE OATES /s/
                   Assistant Secretary

SUBJECT:           Table of Weights for Use in Calculating State-Level American Customer
                   Satisfaction Index (ACSI) Scores for Participant and Employer Customer
                   Satisfaction Surveys

1. Purpose. To provide states with a table of weights for use in calculating Program Year (PY)
   2011 Workforce Investment Act customer satisfaction results (See Attachment A). This
   action will enable states to fulfill statutorily required obligations to collect data and measure
   customer satisfaction.

2. References. Workforce Investment Act of 1998 (WIA), Section 136; WIA Final Rule 20
   CFR Part 666; Training and Employment Guidance Letter (TEGL)
   No. 17-05 (“Common Measures Policy for the Employment and Training Administration’s
   (ETA) Performance Accountability System and Related Performance Issues,” February 17,
   2006).

3. Background. The Department of Labor (Department) uses the American Customer
   Satisfaction Index (ACSI) to meet the customer satisfaction measurement requirements of
   WIA. The ACSI survey is a weighted approach that captures common customer satisfaction
   information, which is then aggregated and compared at different levels. Please refer to the
   WIA Information Management System data collection package (Office of Management and
   Budget No. 1205-0420) for information on creating and administering the ACSI at the state
   level. Guidance on Capturing Customer Satisfaction Results can be found in Attachment B
   of this TEGL.

   The ACSI is a weighted score computed from the customer responses to a small set of
   required questions, with the weighted factors based on the relative distribution of ACSI
   respondent data for non-regulatory federal agencies previously collected and analyzed by the
   ACSI, LLC. The ACSI trademark is the proprietary property of ACSI, LLC.

4. ACSI Table of Weights. Weights were calculated by using a statistical algorithm in order to
   minimize random survey variation or measurement errors that exist in all survey data. The


RESCISSIONS                                               EXPIRATION DATE
TEGL No. 36-10                                            Continuing


   weights listed in the attachment are to be used by a state in computing performance on the
   ACSI for employers and participants and for the Annual Report for PY 2011.

5. Calculating DOL ACSI scores by using the weights. Weights for the 50 states plus the
   District of Columbia are included on the attached table. The ACSI scores of different states
   for DOL services are calculated by the weighted sum of the three ACSI question means
   (measured on 1-10 scales) transformed to 0-100 scale values.

   For example, if the mean values of the three ACSI questions are:

       Overall satisfaction                                 8.3
       Confirmation/disconfirmation of expectations         7.9
       Comparison to ideal                                  7.0

   Values from raw data must first be transformed to the values on the 0-100 scale. This is done
   by subtracting 1 from the mean value, dividing the result by 9, which is the value range of the
   1 to 10 raw data scale, and multiplying by 100.

   0 to 100 Overall Satisfaction = ((8.3 - 1)/9) x 100 = 81.1
   0 to 100 Confirmation/disconfirmation of expectations = ((7.9-1)/9) x 100=76.7
   0 to 100 Comparison to ideal = ((7.0-1)/9) x 100=66.7

   Applying the Michigan weights to this example, the ACSI score as the weighted sum of these
   values would then be: (.3913 x 81.1) + (.3272 x 76.7) + (.2815 x 66.7) = 75.63 (rounds to 76).

6. Action Required. States shall distribute this guidance to all officials within the state who
   need such information to implement WIA customer satisfaction measures. All states must
   administer customer satisfaction surveys and collect data on customer satisfaction regardless
   of whether the state has a waiver for Common Measures. The only difference is that states
   with a waiver for Common Measures are exempted from reporting results to ETA.

   In addition, states will need to footnote their customer satisfaction survey results in their
   WIA Annual Report narrative as follows: the response rate was “X” percent; results are not
   intended to make any inferences beyond the survey’s respondents.

7. Inquiries. Questions concerning this issuance may be directed to the appropriate Regional
   Office.

8. Attachment.

   A. ACSI Formula Weights for use in PY 2011.
   B. Guidance On Capturing Customer Satisfaction Results

Attachment 1: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2013/TEGL_12_12_Att_A.pdf

Attachment A
                     American Customer Satisfaction Index
                     Fomula Weights for Program Year 2011
                                Satisfaction Confirm      Ideal
               State              (SATIS)   (CONFIRM)   (IDEAL)
               Alabama             0.3932     0.3246     0.2822
               Alaska              0.3507     0.3545     0.2948
               Arizona             0.4032     0.3186     0.2781
               Arkansas            0.3919     0.3232     0.2849
               California          0.3905     0.3317     0.2778
               Colorado            0.4053     0.3203     0.2745
               Connecticut         0.3853     0.3298     0.2849
               Delaware            0.3866     0.3330     0.2804
               DC                  0.3750     0.3319     0.2931
               Florida             0.3906     0.3282     0.2812
               Georgia             0.4011     0.3265     0.2724
               Hawaii              0.4048     0.3190     0.2761
               Idaho               0.3797     0.3524     0.2679
               Illinois            0.3857     0.3302     0.2841
               Indiana             0.4081     0.3161     0.2758
               Iowa                0.3899     0.3235     0.2866
               Kansas              0.4002     0.3314     0.2684
               Kentucky            0.3862     0.3307     0.2831
               Louisiana           0.3940     0.3241     0.2819
               Maine               0.3952     0.3233     0.2815
               Maryland            0.3762     0.3302     0.2936
               Massachusetts       0.3879     0.3315     0.2806
               Michigan            0.3913     0.3272     0.2815
               Minnesota           0.3931     0.3278     0.2791
               Mississippi         0.3957     0.3307     0.2736
               Missouri            0.3979     0.3276     0.2745
               Montana             0.4007     0.3215     0.2778
               Nebraska            0.3913     0.3352     0.2735
               Nevada              0.3995     0.3198     0.2807
               New Hampshire       0.3937     0.3284     0.2779
               New Jersey          0.3834     0.3269     0.2896
               New Mexico          0.4178     0.3224     0.2598
               New York            0.3913     0.3338     0.2749
               North Carolina      0.4012     0.3225     0.2763
               North Dakota        0.3991     0.3131     0.2877
               Ohio                0.3940     0.3294     0.2766
               Oklahoma            0.3945     0.3294     0.2761
               Oregon              0.3997     0.3246     0.2758
               Pennsylvania        0.3939     0.3282     0.2779
               Rhode Island        0.4118     0.3399     0.2483
               South Carolina      0.4054     0.3151     0.2795
               South Dakota        0.4017     0.3155     0.2828
               Tennessee           0.4018     0.3180     0.2802
               Texas               0.4016     0.3232     0.2752
               Utah                0.4013     0.3198     0.2789
               Vermont             0.3864     0.3678     0.2457
               Virginia            0.3996     0.3284     0.2720
               Washington          0.3944     0.3227     0.2829
               West Virginia       0.3881     0.3281     0.2838
               Wisconsin           0.3950     0.3236     0.2813
               Wyoming             0.3730     0.3526     0.2744
               Aggregate           0.4127     0.3688     0.3229

Attachment 2: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2013/TEGL_12_12_Att_B.pdf

Attachment B


               GUIDANCE ON CAPTURING CUSTOMER
                    SATISFACTION RESULTS


                Using the American Customer Satisfaction Index
                       to Measure Customer Satisfaction


A. Overview of Measurement Approach

   The Employment and Training Administration (ETA) will use the American Customer
   Satisfaction Index (ACSI) to meet the customer satisfaction measurement needs of the
   Workforce Investment Act (WIA). The survey approach that will be utilized allows
   grantees flexibility and, at the same time, captures common customer satisfaction
   information that can be aggregated and compared at a grant recipient, state, regional and
   national levels. This will be done through the use of a small set of required questions that
   will form a customer satisfaction index. ETA will use the ACSI, which is created by
   combining scores from three specific questions that address different dimensions of
   customers' experience.

   The ACSI is a widely used customer satisfaction measurement approach. It is used
   extensively in the business community, including at over 150 Fortune 500 companies, and
   in many European countries. Twenty-nine agencies of the Federal government have used
   the ACSI. The ACSI will allow the workforce investment system to not only look at
   performance within the system, but also be able to gain perspective on the workforce
   system’s performance by benchmarking against organizations and industries outside of the
   system. The ACSI also has a history of being useful in tracking change in customer
   satisfaction over time, making it an ideal way to gauge grantees’ progress toward
   continuously improving in performance.

   Since the ACSI trademark is proprietary property of the University of Michigan and the
   Claes Fornell International Group (CFI), ETA established a license agreement with the
   University of Michigan that will allow grant recipients the use of the ACSI for a statewide
   sample of participants and employers.

   It is worthwhile to note that WIA state grantees may use the ACSI questions and approach
   to measure satisfaction with WIA services at the local board level if individual license
   agreements are purchased from CFI and the ACSI questions and measurement approach are
   uniformly administered throughout the state. States using the ACSI methodology to
   measure local board performance are not required to conduct separate statewide customer
   satisfaction surveys to obtain State level WIA customer satisfaction results as long as:

      •   the surveys are administered by telephone,

      •   respondents are contacted within the window of opportunity outlined for
          participants and employers, and at least 500 surveys are completed statewide for
          each group -- participants and employers,

      •   the surveys comply with the methodology spelled out in this guidance, and


      •   the Local Board results can be weighted to account for sampling differences, if
          appropriate, and aggregated to obtain state level outcomes.

   WIA state grantees should refer to Training and Employment Guidance Letter (TEGL) No.
   17-05, for information on purchasing licenses to use the ACSI methodology to measure
   satisfaction at the local level. State grantees following this guidance may aggregate local
   outcomes to obtain State level outcomes on the customer satisfaction measures. These
   aggregated results will be reported in the quarterly reports.


B. WIA Participant Customer Satisfaction Measures

   The weighted average of participant ratings on each of the three questions regarding overall
   satisfaction is reported on a 0-100 point scale. The score is a weighted average, not a
   percentage.

    1. Who Will Be Surveyed?

      WIA Title I-B participants with a last expected service date are eligible to be chosen for
      inclusion in the random sample. WIA Title I-B participants include individuals who
      are either adults, dislocated workers, older youth (ages 19-21), and younger youth (ages
      14-18).

    2. How Many (Number to be Surveyed)?

      States may randomly sample individuals to obtain the desired number of completed
      surveys. States are required to determine the appropriate sample sizes and sampling
      percentages using the required response rates and the required number of interviews.
      500 completed participant surveys must be obtained each year for calculation of the
      measures. Participants should be contacted as soon as possible on or after the last
      expected service date and no later than 60 days after the closure date. Closure date is
      another term for the last expected service date.

      A completed survey is defined as a survey in which all three questions regarding
      overall satisfaction have been answered. The standard of 500 from a sample of the
      whole population of customers provides accuracy such that there is only a 5 in 100
      chance that the results would vary by more than ±5 points from the score obtained from
      surveying the whole population.


3. How (methodology) and When to Conduct Surveys?

   To ensure ACSI results for individuals are collected in a consistent and uniform
   manner, procedures used by states to obtain participant customer satisfaction
   information must satisfy the following criteria:

      •   Participant customer satisfaction responses must be collected by way of
          telephone interviews. In-person interviews and mail questionnaires may be
          used only in situations where the individual does not have a telephone;

      •   Participants should be contacted as soon as possible on or after the last expected
          service date and no later than 60 days after the closure date. Closure date is
          another term for the last expected service date;

      •   States must complete a minimum of 500 surveys during the program year to
          accurately assess performance on the ACSI;

      •   If sampling is used, states must randomly select participants to obtain the
          desired number of completed surveys. States are required to determine the
          appropriate sample sizes and sampling percentages using the required response
          rate and the required number of completed interviews;

      •   The sampling methodology used to select potential respondents for the surveys
          must ensure the consistent random selection of a sample of participants eligible
          for the surveys. With this in mind, no individual in the participant group
          eligible for the survey may be arbitrarily excluded from a sample. Procedures
          used to draw a sample must conform to accepted statistical practices, such as
          using a table of random numbers. Participant samples must be drawn on at least
          a monthly basis;

      •   The minimum response rate is 70 percent. The response rate is calculated as the
          number of respondents with complete customer satisfaction information divided
          by the total number in the sampling frame. A survey is considered complete
          where valid answers are provided by respondents for each of the core questions
          outlined by ETA;

      •   It is very important that all records for eligible participants be entered in a
          timely fashion into the databases. The information in these databases will be
          used to develop lists of participants who received services that are the subject of
          the survey. Every precaution must be taken to prevent a response bias;


       •    States must develop a process for creating an up-to-date customer list, capturing
            information on each customer’s address and telephone number. Local programs
            should collect alternate contact information from a person known by the
            customer who would know the whereabouts of the customer in the event the
            customer cannot be reached at the address and telephone number recorded for
            the customer;

       •    A letter in advance of the survey should be sent out informing the customer that
            he or she can expect to receive a telephone call about his or her satisfaction with
            the services. States should place a phone number on the letter, suggesting that
            customers call if they need help or call if their phone numbers changed;

       •    Attempts must be made to contact all eligible participants in the sample to avoid
            introducing bias. A minimum of five follow-up attempts is required, involving
            various times of the day before closing the record;

       •    Respondents must be told that responding to the survey is voluntary and that the
            information they provide will be kept confidential. However, states must not
            make any assurances regarding confidentiality without ensuring that they have
            the legal authority to make such assurances. Such authority may be found in
            state privacy laws, for example;

       •    States must use the minimum core customer satisfaction questions specified by
            ETA in collecting and reporting satisfaction outcomes. These mandatory items
            should be located at the beginning of the questionnaire. States may include
            other questions about the service experience; and

       •    The interview should be limited to 15 minutes or less. Shorter questionnaires
            produce better response rates when compared to longer questionnaires.

4. What are the Core Questions?

   The following introductory statement and core ACSI questions are to be read to the
   respondent at the beginning of the interview 1. The introductory script sets the context
   for the interview by focusing the interview on the service experience, date(s) of service,
   and the firm(s) or organization(s) delivering the service(s). In addition to contact
   information, the survey administrators will need to know the service(s) provided to
   each customer, the date(s) of service, and the firm(s) providing the service(s) in order to
   complete the interviews. The service(s) should be worded in terms recognizable to the
   customer.

              1
                  Note: The first question can be modified to suit the individual needs of the State and the
    names for program services recognizable for their population. The lead-in question provided is a model
    to be used as guidance.


[Introductory Script]

My name is (interviewer) with (firm or agency) and I am conducting a survey for the
(program name). I would like to speak to Ms./Mr. (name of participant).

Are you the Ms./Mr. (name of participant) who received a (name of service(s)) from the
on (date or date range)? [IF THE RESPONDENT CONFIRMS PARTICIPATION, PROCEED WITH
THE INTERVIEW. IF NO, END THE INTERVIEW AND THANK THE RESPONDENT FOR HIS OR
HER TIME.]


I would like to ask you some questions about your recent experience with the services
you received. Our purpose is to learn from you how to improve programs and services
offered to people. Under the public burden statement required by the Paperwork
Reduction Act of 1995, our OMB control number for this information collection is
1205-0420 and permission to collect this expires on 12/31/2012. Responding to this
questionnaire is completely voluntary. The survey will take about 15 minutes. Your
responses are private and will not be shared with the U.S. Department of Labor or any
other agency except as required by law. Do you have any questions for me?
If you have any questions regarding this estimate or any other aspects of this collection,
including suggestions for reducing this burden, send them to Larry Burns, U.S.
Department of Labor, Employment and Training Administration, Office of Workforce
Investment, Room S-4209, Washington, D.C. 20210 (Paperwork Reduction Act 1205-
0420). [IF NO, CONTINUE. IF YES, PROBE TO UNDERSTAND THE QUESTIONS, PROVIDE
ANSWERS AND RESTATE THE IMPORTANCE OF THE RESPONDENT’S PARTICIPATION IN THE
INTERVIEW.]

[The Questions]

First, I want you to rate your overall experience with the services. I will read to you
three statements and ask you to rate your experience.

Utilizing a scale of 1 to 10 where a 1 means “Very Dissatisfied” and 10 means “Very
Satisfied” what is your overall satisfaction with the services provided from     ?

Very                                                                Very     DK 2   REF 3
Dissatisfied                                                   Satisfied
1        2          3        4        5    6   7     8       9        10       11    12


           2
               DK = Don’t Know

           3
               REF = Refused to Answer 6

Considering all of the expectations you may have had about the services, to what extent
have the services met your expectations? A 1 now means “Falls Short of Your
Expectations” and 10 means “Exceeds Your Expectations.”

Falls Short of                                               Exceeds      DK     REF
Expectations                                              Expectations
1      2       3       4     5       6       7     8       9        10    11     12

Now think of the ideal program for people in your circumstances. How well do you
think the services you received compare with the ideal set of services? A 1 now means
“Not very close to the Ideal” and 10 means “Very Close to the Ideal.”

Not Close                                                  Very Close     DK   REF
To Ideal                                                     To Ideal
1        2         3   4     5       6       7     8       9       10     11    12


C. Measuring One-Stop Employer Customer Satisfaction

   The weighted average of employer ratings on each of the three questions regarding overall
   satisfaction is reported on a 0-100 point scale. The score is a weighted average, not a
   percentage.

    1. Who Will Be Surveyed?

      One-Stop Employers

      Those eligible for surveying include employers who received a substantial service
      where the service has been completed or, if it is an ongoing service, when a full
      segment of service has been provided. All employers who received a substantial
      service involving personal contact with WIA staff are eligible to be chosen for
      inclusion in the random sample. Examples of substantial services include staff
      facilitated job orders, customized job training, customized labor market information
      requests, and on-the-job training activities.

      When an employer has received multiple services, priority should be given to the
      service that required the greatest expenditure of financial assistance or staff time 4, and
      the survey conducted regarding their satisfaction with that service. An employer should
      only be surveyed one time during the course of the program year.

    2. How Many (number obtained)?

      Except in small states, a sample will be taken from these employers. At least 500
      completed interviews are required for the employer survey for calculation of the
      indicator. States are required to determine the appropriate sample sizes and sampling
      percentages using the required response rate and the required number of completed
      interviews. The sampling percentage should be constant over time to avoid under-
      representing or over-representing time intervals. A completed employer survey is
      defined as a survey in which all three questions regarding overall satisfaction have been
      answered. The standard of 500 from a sample of the whole population of customers
      provides accuracy such that there is only a 5 in 100 chance that the results would vary
      by more than 5 points from the score obtained from surveying the whole population.


                      Where an employer has received multiple services in a given time period, and there is
        separate contact information for each service, the contact information for the priority service should be
        used for surveying.


3. How (methodology) and When to Conduct Surveys?

   To ensure ACSI results for individuals are collected in a consistent and uniform
   manner, procedures used by states to obtain employer customer satisfaction information
   must satisfy the following criteria:

      •   Employer customer satisfaction responses must be collected by way of
          telephone interviews. In-person interviews and mail questionnaires may be
          used only in situations where the individual does not have a telephone;

      •   Employers should be contacted as soon as possible after the completion of the
          service and no later than 60 days after the completion of the service. For
          employers who listed a job order where no referrals were made, contact should
          occur 30 to 60 days after a job order was listed;

      •   States must complete a minimum of 500 employer surveys during the program
          year to accurately assess performance on the ACSI;

      •   If sampling is used, states must randomly sample employers to obtain the
          desired number of completed surveys. States are required to determine the
          appropriate sample sizes and sampling percentages using the required response
          rate and the required number of completed interviews;

      •   The sampling methodology used to select potential respondents for the surveys
          must ensure the consistent random selection of a sample of employers eligible
          for the surveys. With this in mind, no individual in the employer group eligible
          for the survey may be arbitrarily excluded from a sample. Procedures used to
          draw a sample must conform to accepted statistical practices, such as using a
          table of random numbers;

      •   Employer samples must be drawn on at least a monthly basis. To obtain
          sufficient numbers, smaller states will need to survey on an ongoing basis;

      •   The minimum response rate for the employer survey is 70 percent. The
          response rate is calculated as the number of respondents with complete customer
          satisfaction information divided by the total number in the sampling frame. A
          survey is considered complete where valid answers are provided by respondents
          for each of the core questions outlined by ETA;

      •   It is very important that all records for eligible employers be entered in a timely
          fashion into the databases. The information in these databases will be used to
          develop lists of employers who received services that are the subject of the
          surveys. Every precaution must be taken to prevent a response bias.


       •    States must develop a process for creating an up-to-date customer list, capturing
            information on each customer’s address and telephone number. Local programs
            should collect alternate contact information from a person known by the
            customer who would know the whereabouts of the customer in the event the
            customer cannot be reached at the address and telephone number recorded for
            the customer.

       •    A letter in advance of the survey should be sent out informing the customer that
            he or she can expect to receive a telephone call about his or her satisfaction with
            the services. States should place a phone number on the letter, suggesting that
            customers call if they need help or call if their phone numbers changed.

       •    Attempts must be made to contact all eligible employers in the samples to avoid
            introducing bias. A minimum of five follow-up attempts is required, involving
            various times of the day before closing the record.

       •    Respondents must be told that responding to the survey is voluntary and that the
            information they provide will be kept confidential. However, states must not
            make any assurances regarding confidentiality without ensuring that they have
            the legal authority to make such assurances. Such authority may be found in
            state privacy laws, for example.

       •    States must use the minimum core customer satisfaction questions specified by
            ETA in collecting and reporting satisfaction outcomes. These mandatory items
            should be located at the beginning of the questionnaire. States may include
            other questions about the service experience.

       •    The interview should be limited to 15 minutes or less. Shorter questionnaires
            produce better response rates when compared to longer questionnaires.

4. What are the Core Questions?

   The following introductory statement and core ACSI questions are to be read to the
   respondent at the beginning of the interview 5. The introductory script sets the context
   for the interview by focusing the interview on the service experience, date(s) of service,
   and the firm(s) or organization(s) delivering the service(s). In addition to contact
   information, the survey administrators will need to know the service(s) provided to
   each customer, the date(s) of service, and the firm(s) providing the service(s) in order to
   complete the interviews. The service(s) should be worded in terms recognizable to the
   customer.
              5
                  Note: The first question can be modified to suit the individual needs of the State and the
    names for program services recognizable for their population. The lead-in question provided is a model
    to be used as guidance.


[Introductory Script]

My name is (interviewer) with (firm or agency) and I am conducting a survey for the
(program name). I would like to speak to Ms./Mr. (name of employer contact).

Are you the Ms./Mr. (name of contact) who received a (name of service(s)) from the
on (date or date range)? [IF THE RESPONDENT CONFIRMS PARTICIPATION, PROCEED WITH
THE INTERVIEW. IF NO, END THE INTERVIEW AND THANK THE RESPONDENT.]

I would like to ask you some questions about your recent experience with the services
you received. Our purpose is to learn from you how to improve programs and services
offered to people. Under the public burden statement required by the Paperwork
Reduction Act of 1995, our OMB control number for this information collection is
1205-0420 and permission to collect this expires on 12/31/2012. Responding to this
questionnaire is completely voluntary. The survey will take about 15 minutes. Your
responses are private and will not be shared with the U.S. Department of Labor or any
other agency except as required by law. Do you have any questions for me?
If you have any questions regarding this estimate or any other aspects of this collection,
including suggestions for reducing this burden, send them to Larry Burns, U.S.
Department of Labor, Employment and Training Administration, Office of Workforce
Investment, Room S-4209, Washington, D.C. 20210 (Paperwork Reduction Act 1205-
0420).

[IF NO, CONTINUE. IF YES, PROBE TO UNDERSTAND THE QUESTIONS, PROVIDE ANSWERS
AND RESTATE THE IMPORTANCE OF THE RESPONDENT’S PARTICIPATION IN THE
INTERVIEW.]

[The Questions]

First, I want you to rate your overall experience with the services. I will read to you
three statements and ask you to rate your experience.

Utilizing a scale of 1 to 10 where 1 means “Very Dissatisfied” and 10 means “Very
Satisfied” what is your overall satisfaction with the services provided from   ?

Very                                                               Very
Dissatisfied                                                     Satisfied   DK 6   REF 7
1        2          3        4           5   6    7   8      9        10       11    12


           6
               DK = Dont Know

           7
               REF = Refused to Answer       11

      Considering all of the expectations you may have had about the services, to what extent
      have the services met your expectations? A 1 now means “Falls Short of Your
      Expectations” and 10 means “Exceeds Your Expectations.”

        Falls Short of                                                 Exceeds      DK    REF
      Expectations                                                   Expectations
      1         2        3   4       5      6       7      8       9      10         11     12

      Now think of the ideal program for people in your circumstances. How well do you
      think the services you received compare with the ideal set of services? A 1 now means
      “Not very close to the Ideal” and 10 means “Very Close to the Ideal.”

      Not Close                                                      Very Close     DK      REF
      To Ideal                                                         To Ideal
      1        2         3   4       5      6       7      8       9      10        11       12


D. Definition of Terms

   Sample - A group of cases selected from a population by a random process where everyone
   has an equal probability of being selected.

   Last Expected Service – Occurs when the participant completes the activities outlined in his
   or her service strategy or service plan and there are no additional services expected other
   than supportive or follow-up services. Last expected service may also occur in situations
   where the participant voluntarily or involuntarily discontinues his or her participation in
   services outlined in the service plan.

   Last Expected Service Date – This date is used to determine when a customer becomes a
   part of the sampling frame for the participant customer satisfaction survey. In many
   instances, this date will be the same as the exit date. In situations where a case was ended,
   reopened within 90 days of the original closure date, and then ended again, the date used to
   determine inclusion in the sampling frame is the initial last expected service date.

   Response rate - The percentage of people in the sample who are contacted and respond to
   the core questions on the survey.


E. Calculation of the ACSI

   The ACSI scores represent the weighted sum of the three ACSI questions’ values which are
   transformed into 0 to 100 scale values. The weights are applied to each of the three
   questions to account for differences in the characteristics of the State’s customer groups.

   For example, assume the mean values of three ACSI questions for a state are:

        1. Overall Satisfaction       = 8.3
        2. Met Expectations           = 7.9
        3. Compared to Ideal          = 7.0

   Then, these mean values from raw data must first be transformed to the value on a 0 to 100
   scale. This is done by subtracting 1 from these mean values, dividing the results by 9
   which is the value of range of a 1 to 10 raw data scale, and multiplying the whole by 100:

        1. Overall Satisfaction       = (8.3 - 1)/9 x 100 = 81.1
        2. Met Expectations           = (7.9 - 1)/9 x 100 = 76.7
        3. Compared to Ideal          = (7.0 - 1)/9 x 100 = 66.7

   The ACSI score is calculated as the weighted averages of these values. Assuming the
   weights for the example state are 0.3804, 0.3247 and 0.2949 for questions 1, 2 and 3,
   respectively, the ACSI score for the state would be calculated as follows:

              (0.3804 x 81.1) + (0.3247 x 76.7) + (0.2949 x 66.7) = 75.4

   Weights were calculated by a statistical algorithm to minimize measurement error or
   random survey noise that exists in all survey data. State-specific weights were calculated
   using the relative distribution of ACSI respondent data for non-regulatory Federal agencies
   previously collected and analyzed by CFI and the University of Michigan.

   Specific weighting factors have been developed for each State. New weighting factors will
   be published annually.