2010 WIA customer satisfaction weights

Accessible letter PDF: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2011/TEGL36-10ACC.pdf

                                                             CLASSIFICATION
   EMPLOYMENT AND TRAINING ADMINISTRATION                    ACSI
                ADVISORY SYSTEM                              CORRESPONDENCE SYMBOL
          U.S. DEPARTMENT OF LABOR                           OWI/DAS
               Washington, D.C. 20210                        DATE
                                                             June 27, 2011



TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 36-10

TO:                ALL STATE WORKFORCE AGENCIES
                   ALL STATE WORKFORCE LIAISONS

FROM:              JANE OATES /s/
                   Assistant Secretary

SUBJECT:           Table of Weights for Use in Calculating State-Level American
                   Customer Satisfaction Index (ACSI) Scores for the Participant and
                   Employer Customer Satisfaction Surveys

1. Purpose. To provide states the table of weights for use in calculating Program Year
   (PY) 2010 Workforce Investment Act customer satisfaction results (See Attachment
   A). This action will enable states to continue to fulfill statutorily required obligations
   to collect data and measure customer satisfaction.

2. References. Workforce Investment Act of 1998 (WIA), Section 136; WIA Final
   Rule 20 CFR Part 666; Training and Employment Guidance Letter (TEGL)
   No. 17-05 (“Common Measures Policy for the Employment and Training
   Administration’s (ETA) Performance Accountability System and Related
   Performance Issues,” February 17, 2006).

3. Background. The Department of Labor (Department) uses the ACSI to meet the
   customer satisfaction measurement requirements of WIA. The ACSI survey
   approach captures common customer satisfaction information that can be aggregated
   and compared at different levels. Please refer to the WIA Information Management
   System data collection package (Office of Management and Budget No. 1205-0420)
   for information on creating and administering the ACSI at the state level. Guidance
   on Capturing Customer Satisfaction Results can be found in Attachment B of this
   TEGL as well as on the ETA Web site at:
   http://www.doleta.gov/performance/guidance/WIA/Guidance-Collecting-Customer-
   Satisfaction-Data-Expires-02282009.doc.

   The ACSI is a weighted score computed from the customer responses to a small set of
   required questions, with the weighted factors based on the relative distribution of
   ACSI respondent data for non-regulatory Federal agencies previously collected and
   analyzed by the ACSI, LLC. The ACSI trademark is the proprietary property of
   ACSI, LLC.




RESCISSIONS                                               EXPIRATION DATE
TEGL No. 07-09                                            Continuing
                                            2

4. ACSI Table of Weights. Weights were calculated by using a statistical algorithm in
   order to minimize random survey variation or measurement errors that exist in all
   survey data. The weights listed in the attachment are to be used by a state in
   computing performance on the ACSI for employers and participants, and for the
   Annual Report for PY 2010.

   The weights for the 48 states plus the District of Columbia are included on the
   attached table.

5. Calculating DOL ACSI Scores by Using the Weights. ACSI is unable to produce
   weights for Alaska and Hawaii due to unusually small survey sample sizes. In the
   Annual Report, these states may a) submit data using the weights last made available
   or b) submit unweighted results. The ACSI scores of different states for DOL
   services are calculated by the weighted sum of the three ACSI question means
   (measured on 1-10 scales) transformed to 0-100 scale values.

   For example, if the mean values of the three ACSI questions for a state are:

       Overall satisfaction                                  8.3
       Confirmation/disconfirmation of expectations          7.9
       Comparison to ideal                                   7.0

   These mean values from raw data must first be transformed to the values on the 0-100
   scale. This is done by subtracting 1 from the mean values, dividing the result by 9,
   which is the value range of the 1 to 10 raw data scale, and multiplying by 100.

   0 to 100 Overall Satisfaction = ((8.3 - 1)/9) x 100 = 81.1
   0 to 100 Confirmation/disconfirmation of expectations = ((7.9-1)/9) x 100 = 76.7
   0 to 100 Comparison to ideal = ((7.0-1)/9) x 100 = 66.7

   These values are then multiplied by the weighted values for the particular state. For
   example, the ACSI values in the example, using ACSI’s weighted values for
   illustrative purposes would be: (.3911 x 81.1) + (.3283 x 76.7) + (.2806 x 66.7) = 75.6
   (rounds to 76).

6. Action Requested. States shall distribute this guidance to all officials within the
   state who need such information to implement WIA customer satisfaction measures.
   All states must administer customer satisfaction surveys and collect data on customer
   satisfaction regardless of whether the state has a waiver of the Common Measures.
   The only difference is that those states with a waiver of the Common Measures are
   exempted from reporting results to ETA. As mentioned in section four of this TEGL,
   Hawaii and Alaska have two options that will fulfill the reporting requirements.

   In addition, states will need to footnote their customer satisfaction survey results in
   their WIA Annual Report narrative as follows: the response rate was “X” percent;
   results are not intended to make any inferences beyond the survey’s respondents.
                                         3

7. Inquiries. Questions concerning this issuance may be directed to the appropriate
   Regional Office.

8. Attachment.

   A. ACSI Formula Weights for use in PY 2010.
   B. Guidance On Capturing Customer Satisfaction Results

Official attachment PDF: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2011/TEGL36-10ATT-A.pdf

                                                        ATTACHMENT A



                 American Customer Satisfaction Index
                 Formula Weights for PY 2010

                 Satisfaction     Confirm   Ideal
                  (SATIS)       (CONFIRM) (IDEAL)


State
Alabama             0.3921        0.3280      0.2799
Arizona             0.3553        0.3683      0.2764
Arkansas            0.3927        0.3195      0.2878
California          0.3862        0.3321      0.2817
Colorado            0.3908        0.3291      0.2802
Connecticut         0.3955        0.3272      0.2773
Delaware            0.3857        0.3243      0.2901
District of
Columbia            0.3900        0.3276      0.2824
Florida             0.4188        0.3325      0.2486
Georgia             0.3935        0.3277      0.2788
Idaho               0.3772        0.3427      0.2801
Illinois            0.3898        0.3307      0.2795
Indiana             0.3893        0.3283      0.2824
Iowa                0.3898        0.3287      0.2815
Kansas              0.3863        0.3382      0.2755
Kentucky            0.3902        0.3285      0.2813
Louisiana           0.3952        0.3250      0.2798
Maine               0.3796        0.3348      0.2856
Maryland            0.3940        0.3317      0.2743
Massachusetts       0.3847        0.3365      0.2788
Michigan            0.3911        0.3283      0.2806
Minnesota           0.3883        0.3371      0.2746
Mississippi         0.3967        0.3218      0.2814
Missouri            0.3942        0.3294      0.2764
Montana             0.3815        0.3393      0.2792
Nebraska            0.3762        0.3460      0.2778
Nevada              0.3878        0.3306      0.2816
New
Hampshire           0.3838        0.3319      0.2843
New Jersey          0.3929        0.3286      0.2785
New Mexico          0.3904        0.3309      0.2787
New York            0.3902        0.3298      0.2800
North Carolina      0.4055        0.3201      0.2744
North Dakota        0.3759        0.3126      0.3115
Ohio                0.3962        0.3264      0.2774
Oklahoma            0.3908        0.3320      0.2772
Oregon              0.3856        0.3215      0.2929
Pennsylvania     0.3971   0.3282   0.2747
Rhode Island     0.3719   0.3248   0.3033
South Carolina   0.4047   0.3190   0.2763
South Dakota     0.3921   0.3501   0.2579
Tennessee        0.4076   0.3142   0.2782
Texas            0.3852   0.3309   0.2838
Utah             0.3770   0.3323   0.2906
Vermont          0.3859   0.3435   0.2706
Virginia         0.3916   0.3298   0.2786
Washington       0.3925   0.3308   0.2767
West Virginia    0.3925   0.3248   0.2828
Wisconsin        0.3845   0.3272   0.2883
Wyoming          0.3642   0.3234   0.3125

Official attachment PDF: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2011/TEGL36-10ATT-B.pdf

                             ATTACHMENT B




GUIDANCE ON CAPTURING CUSTOMER
     SATISFACTION RESULTS
                Using the American Customer Satisfaction Index
                       to Measure Customer Satisfaction


A. Overview of Measurement Approach

   The Employment and Training Administration (ETA) will use the American Customer
   Satisfaction Index (ACSI) to meet the customer satisfaction measurement needs of the
   Workforce Investment Act (WIA). The survey approach that will be utilized allows
   grantees flexibility and, at the same time, captures common customer satisfaction
   information that can be aggregated and compared at a grant recipient, state, regional and
   national levels. This will be done through the use of a small set of required questions that
   will form a customer satisfaction index. ETA will use the ACSI, which is created by
   combining scores from three specific questions that address different dimensions of
   customers' experience.

   The ACSI is a widely used customer satisfaction measurement approach. It is used
   extensively in the business community, including at over 150 Fortune 500 companies, and
   in many European countries. Twenty-nine agencies of the Federal government have used
   the ACSI. The ACSI will allow the workforce investment system to not only look at
   performance within the system, but also be able to gain perspective on the workforce
   system’s performance by benchmarking against organizations and industries outside of the
   system. The ACSI also has a history of being useful in tracking change in customer
   satisfaction over time, making it an ideal way to gauge grantees’ progress toward
   continuously improving in performance.

   Since the ACSI trademark is proprietary property of the University of Michigan and the
   Claes Fornell International Group (CFI), ETA established a license agreement with the
   University of Michigan that will allow grant recipients the use of the ACSI for a statewide
   sample of participants and employers.

   It is worthwhile to note that WIA state grantees may use the ACSI questions and approach
   to measure satisfaction with WIA services at the local board level if individual license
   agreements are purchased from CFI and the ACSI questions and measurement approach are
   uniformly administered throughout the state. States using the ACSI methodology to
   measure local board performance are not required to conduct separate statewide customer
   satisfaction surveys to obtain State level WIA customer satisfaction results as long as:

         the surveys are administered by telephone,

         respondents are contacted within the window of opportunity outlined for
          participants and employers,

         at least 500 surveys are completed statewide for each group -- participants and
          employers,
         the surveys comply with the methodology spelled out in this guidance, and

         the Local Board results can be weighted to account for sampling differences, if
          appropriate, and aggregated to obtain state level outcomes.

B. WIA Participant Customer Satisfaction Measures

   The weighted average of participant ratings on each of the three questions regarding overall
   satisfaction is reported on a 0-100 point scale. The score is a weighted average, not a
   percentage.

    1. Who Will Be Surveyed?

      WIA Title I-B participants with a last expected service date are eligible to be chosen for
      inclusion in the random sample. WIA Title I-B participants include individuals who
      are either adults, dislocated workers, older youth (ages 19-21), and younger youth (ages
      14-18).

    2. How Many (Number to be Surveyed)?

      States may randomly sample individuals to obtain the desired number of completed
      surveys. States are required to determine the appropriate sample sizes and sampling
      percentages using the required response rates and the required number of interviews.
      At least 500 completed participant surveys must be obtained each year for calculation
      of the measures.

      A completed survey is defined as a survey in which all three questions regarding
      overall satisfaction have been answered. The standard of 500 from a sample of the
      whole population of customers provides accuracy such that there is only a 5 in 100
      chance that the results would vary by more than ±5 points from the score obtained from
      surveying the whole population.

    3. How (methodology) and When to Conduct Surveys?

      To ensure ACSI results for individuals are collected in a consistent and uniform
      manner, procedures used by states to obtain participant customer satisfaction
      information must satisfy the following criteria:

             Participant customer satisfaction responses must be collected by way of
              telephone interviews. In-person interviews and mail questionnaires may be
              used only in situations where the individual does not have a telephone;

             Participants should be contacted as soon as possible on or after the last expected
              service date and no later than 60 days after the closure date. Closure date is
              another term for the last expected service date;

             States must complete a minimum of 500 surveys during the program year to
              accurately assess performance on the ACSI;
   If sampling is used, states must randomly select participants to obtain the
    desired number of completed surveys. States are required to determine the
    appropriate sample sizes and sampling percentages using the required response
    rate and the required number of completed interviews;

   The sampling methodology used to select potential respondents for the surveys
    must ensure the consistent random selection of a sample of participants eligible
    for the surveys. With this in mind, no individual in the participant group
    eligible for the survey may be arbitrarily excluded from a sample. Procedures
    used to draw a sample must conform to accepted statistical practices, such as
    using a table of random numbers. Participant samples must be drawn on at least
    a monthly basis;

   The minimum response rate is 70 percent. The response rate is calculated as the
    number of respondents with complete customer satisfaction information divided
    by the total number in the sampling frame. A survey is considered complete
    where valid answers are provided by respondents for each of the core questions
    outlined by ETA;

   It is very important that all records for eligible participants be entered in a
    timely fashion into the databases. The information in these databases will be
    used to develop lists of participants who received services that are the subject of
    the survey. Every precaution must be taken to prevent a response bias;

   States must develop a process for creating an up-to-date customer list, capturing
    information on each customer’s address and telephone number. Local programs
    should collect alternate contact information from a person known by the
    customer who would know the whereabouts of the customer in the event the
    customer cannot be reached at the address and telephone number recorded for
    the customer;

   A letter in advance of the survey should be sent out informing the customer that
    he or she can expect to receive a telephone call about his or her satisfaction with
    the services. States should place a phone number on the letter, suggesting that
    customers call if they need help or call if their phone numbers changed;

   Attempts must be made to contact all eligible participants in the sample to avoid
    introducing bias. A minimum of five follow-up attempts is required, involving
    various times of the day before closing the record;

   Respondents must be told that responding to the survey is voluntary and that the
    information they provide will be kept confidential. However, states must not
    make any assurances regarding confidentiality without ensuring that they have
    the legal authority to make such assurances. Such authority can be found in
    state privacy laws, for example;

   States must use the minimum core customer satisfaction questions specified by
            ETA in collecting and reporting satisfaction outcomes. These mandatory items
            should be located at the beginning of the questionnaire. States may include
            other questions about the service experience; and

           The interview should be limited to 15 minutes or less. Shorter questionnaires
            produce better response rates when compared to longer questionnaires.

4. What are the Core Questions?

   The following introductory statement and core ACSI questions are to be read to the
   respondent at the beginning of the interview1. The introductory script sets the context
   for the interview by focusing the interview on the service experience, date(s) of service,
   and the firm(s) or organization(s) delivering the service(s). In addition to contact
   information, the survey administrators will need to know the service(s) provided to
   each customer, the date(s) of service, and the firm(s) providing the service(s) in order
   to complete the interviews. The service(s) should be worded in terms recognizable to
   the customer.

   [Introductory Script]

   My name is (interviewer) with (firm or agency) and I am conducting a survey for the
   (program name). I would like to speak to Ms./Mr. (name of participant).

   Are you the Ms./Mr. (name of participant) who received a (name of service(s)) from
   theon (date or date range)? [IF THE RESPONDENT CONFIRMS PARTICIPATION, PROCEED
   WITH THE INTERVIEW. IF NO, END THE INTERVIEW AND THANK THE RESPONDENT FOR HIS
   OR HER TIME.]

   I would like to ask you some questions about your recent experience with the services
   provided to you. Our purpose is to learn from you how to improve programs and
   services offered to people. The survey was approved by the Office of Management and
   Budget under OMB Control No. 1205-0420, expiring 02/28/2009 to collect information
   on your experience with services we provide to individuals. Without this approval, we
   would not be able to conduct this survey. The questionnaire is voluntary and should
   take no longer than 15 minutes to complete.

   Please know the answers you provide to my questions will be kept confidential. If you
   have any questions regarding this estimate or any other aspect of this data collection,
   including suggestions for reducing this burden, send them to the U.S. Department of
   Labor, Office of Workforce Development, Room C-4526, Washington, D.C. 20210
   (Paperwork Reduction Project 1205-0420). Do you have any questions for me before
   we start the interview? [IF NO, CONTINUE. IF YES, PROBE TO UNDERSTAND THE

              1
                  Note: The first question can be modified to suit the individual needs of the State and the
    names for program services recognizable for their population. The lead-in question provided is a model
    to be used as guidance.
      QUESTIONS, PROVIDE ANSWERS AND RESTATE THE IMPORTANCE OF THE RESPONDENT’S
      PARTICIPATION IN THE INTERVIEW.]

      [The Questions]

      First, I want you to rate your overall experience with the services. I will read to you
      three statements and ask you to rate your experience.

      Utilizing a scale of 1 to 10 where a 1 means “Very Dissatisfied” and 10 means “Very
      Satisfied” what is your overall satisfaction with the services provided from        ?

      Very                                                                Very     DK2         REF3
      Dissatisfied                                                   Satisfied
      1        2            3        4           5   6   7   8     9        10          11         12

      Considering all of the expectations you may have had about the services, to what extent
      have the services met your expectations? A 1 now means “Falls Short of Your
      Expectations” and 10 means “Exceeds Your Expectations.”

      Falls Short of                                                  Exceeds      DK         REF
      Expectations                                                 Expectations
      1      2       3               4           5   6   7   8      9        10    11         12

      Now think of the ideal program for people in your circumstances. How well do you
      think the services you received compare with the ideal set of services? A 1 now means
      “Not very close to the Ideal” and 10 means “Very Close to the Ideal.”

      Not Close                                                    Very Close      DK        REF
      To Ideal                                                       To Ideal
      1        2            3        4           5   6   7   8     9      10       11        12

C. Measuring One-Stop Employer Customer Satisfaction

   The weighted average of employer ratings on each of the three questions regarding overall
   satisfaction is reported on a 0-100 point scale. The score is a weighted average, not a
   percentage.

    1. Who Will Be Surveyed?

      One-Stop Employers

      Those eligible for surveying include employers who received a substantial service
      where the service has been completed or, if it is an ongoing service, when a full

                   2
                       DK = Don’t Know

                   3
                       REF = Refused to Answer
   segment of service has been provided. All employers who received a substantial
   service involving personal contact with WIA staff are eligible to be chosen for
   inclusion in the random sample. Examples of substantial services include staff
   facilitated job orders, customized job training, customized labor market information
   requests, and on-the-job training activities.

   When an employer has received multiple services, priority should be given to the
   service that required the greatest expenditure of financial assistance or staff time4, and
   the survey conducted regarding their satisfaction with that service. An employer
   should only be surveyed one time during the course of the program year.

2. How Many (number obtained)?

   Except in small states, a sample will be taken from these employers. At least 500
   completed interviews are required for the employer survey for calculation of the
   indicator. States are required to determine the appropriate sample sizes and sampling
   percentages using the required response rate and the required number of completed
   interviews. The sampling percentage should be constant over time to avoid under-
   representing or over-representing time intervals. A completed employer survey is
   defined as a survey in which all three questions regarding overall satisfaction have been
   answered. The standard of 500 from a sample of the whole population of customers
   provides accuracy such that there is only a 5 in 100 chance that the results would vary
   by more than 5 points from the score obtained from surveying the whole population.

3. How (methodology) and When to Conduct Surveys?

   To ensure ACSI results for individuals are collected in a consistent and uniform
   manner, procedures used by states to obtain employer customer satisfaction information
   must satisfy the following criteria:

           Employer customer satisfaction responses must be collected by way of
            telephone interviews. In-person interviews and mail questionnaires may be
            used only in situations where the individual does not have a telephone;

           Employers should be contacted as soon as possible after the completion of the
            service and no later than 60 days after the completion of the service. For
            employers who listed a job order where no referrals were made, contact should
            occur 30 to 60 days after a job order was listed;
           States must complete a minimum of 500 employer surveys during the program
            year to accurately assess performance on the ACSI;

           If sampling is used, states must randomly sample employers to obtain the
            desired number of completed surveys. States are required to determine the

              4
                  Where an employer has received multiple services in a given time period, and there is
    separate contact information for each service, the contact information for the priority service should be
    used for surveying.
    appropriate sample sizes and sampling percentages using the required response
    rate and the required number of completed interviews;

   The sampling methodology used to select potential respondents for the surveys
    must ensure the consistent random selection of a sample of employers eligible
    for the surveys. With this in mind, no individual in the employer group eligible
    for the survey may be arbitrarily excluded from a sample. Procedures used to
    draw a sample must conform to accepted statistical practices, such as using a
    table of random numbers;

   Employer samples must be drawn on at least a monthly basis. To obtain
    sufficient numbers, smaller states will need to survey on an ongoing basis;

   The minimum response rate for the employer survey is 70 percent. The
    response rate is calculated as the number of respondents with complete
    customer satisfaction information divided by the total number in the sampling
    frame. A survey is considered complete where valid answers are provided by
    respondents for each of the core questions outlined by ETA;

   It is very important that all records for eligible employers be entered in a timely
    fashion into the databases. The information in these databases will be used to
    develop lists of employers who received services that are the subject of the
    surveys. Every precaution must be taken to prevent a response bias.

   States must develop a process for creating an up-to-date customer list, capturing
    information on each customer’s address and telephone number. Local programs
    should collect alternate contact information from a person known by the
    customer who would know the whereabouts of the customer in the event the
    customer cannot be reached at the address and telephone number recorded for
    the customer.

   A letter in advance of the survey should be sent out informing the customer that
    he or she can expect to receive a telephone call about his or her satisfaction with
    the services. States should place a phone number on the letter, suggesting that
    customers call if they need help or call if their phone numbers changed.

   Attempts must be made to contact all eligible employers in the samples to avoid
    introducing bias. A minimum of five follow-up attempts is required, involving
    various times of the day before closing the record.

   Respondents must be told that responding to the survey is voluntary and that the
    information they provide will be kept confidential. However, states must not
    make any assurances regarding confidentiality without ensuring that they have
    the legal authority to make such assurances. Such authority can be found in
    state privacy laws, for example.

   States must use the minimum core customer satisfaction questions specified by
    ETA in collecting and reporting satisfaction outcomes. These mandatory items
            should be located at the beginning of the questionnaire. States may include
            other questions about the service experience.

           The interview should be limited to 15 minutes or less. Shorter questionnaires
            produce better response rates when compared to longer questionnaires.

4. What are the Core Questions?

   The following introductory statement and core ACSI questions are to be read to the
   respondent at the beginning of the interview5. The introductory script sets the context
   for the interview by focusing the interview on the service experience, date(s) of service,
   and the firm(s) or organization(s) delivering the service(s). In addition to contact
   information, the survey administrators will need to know the service(s) provided to
   each customer, the date(s) of service, and the firm(s) providing the service(s) in order
   to complete the interviews. The service(s) should be worded in terms recognizable to
   the customer.

   [Introductory Script]

   My name is (interviewer) with (firm or agency) and I am conducting a survey for the
   (program name). I would like to speak to Ms./Mr. (name of employer contact).

   Are you the Ms./Mr. (name of contact) who received a (name of service(s)) from the on
   (date or date range)? [IF THE RESPONDENT CONFIRMS PARTICIPATION, PROCEED WITH
   THE INTERVIEW. IF NO, END THE INTERVIEW AND THANK THE RESPONDENT.]

   I would like to ask you some questions about your recent experience with the services
   provided to your business. Our purpose is to learn from you how to improve programs
   and services offered to area businesses. The survey was approved by the Office of
   Management and Budget under OMB Control No. 1205-0420, expiring 02/28/2009 to
   collect information on your experience with services we provide to employers.
   Without this approval, we would not be able to conduct this survey. The questionnaire
   is voluntary and should take no longer than 15 minutes to complete.

   Please know the answers you provide to my questions will be kept confidential. If you
   have any questions regarding this estimate or any other aspects of this collection,
   including suggestions for reducing this burden, send them to the U.S. Department of
   Labor, Office of ___________, Room S-4231, Washington, D.C. 20210 (Paperwork
   Reduction Project 1205-0420). Do you have any questions for me before we start the
   interview? [IF NO, CONTINUE. IF YES, PROBE TO UNDERSTAND THE QUESTIONS, PROVIDE
   ANSWERS AND RESTATE THE IMPORTANCE OF THE RESPONDENT’S PARTICIPATION IN THE
   INTERVIEW.]

              5
                  Note: The first question can be modified to suit the individual needs of the State and the
    names for program services recognizable for their population. The lead-in question provided is a model
    to be used as guidance.
      [The Questions]

      First, I want you to rate your overall experience with the services. I will read to you
      three statements and ask you to rate your experience.

      Utilizing a scale of 1 to 10 where 1 means “Very Dissatisfied” and 10 means “Very
      Satisfied” what is your overall satisfaction with the services provided from      ?

      Very                                                               Very
      Dissatisfied                                                     Satisfied      DK6    REF7
      1        2            3        4           5   6   7   8     9        10          11    12

      Considering all of the expectations you may have had about the services, to what extent
      have the services met your expectations? A 1 now means “Falls Short of Your
      Expectations” and 10 means “Exceeds Your Expectations.”

      Falls Short of                                                     Exceeds       DK    REF
      Expectations                                                     Expectations

      1        2            3        4           5   6   7   8     9        10          11      12

      Now think of the ideal program for people in your circumstances. How well do you
      think the services you received compare with the ideal set of services? A 1 now means
      “Not very close to the Ideal” and 10 means “Very Close to the Ideal.”

      Not Close                                                        Very Close     DK
   REF
      To Ideal                                                           To Ideal
      1        2            3        4           5   6   7   8     9        10         11        12

D. Definition of Terms

   Sample - A group of cases selected from a population by a random process where everyone
   has an equal probability of being selected.

   Last Expected Service – Occurs when the participant completes the activities outlined in his
   or her service strategy or service plan and there are no additional services expected other
   than supportive or follow-up services. Last expected service may also occur in situations
   where the participant voluntarily or involuntarily discontinues his or her participation in
   services outlined in the service plan.




                   6
                       DK = Don t Know

                   7
                       REF = Refused to Answer
   Last Expected Service Date – This date is used to determine when a customer becomes a
   part of the sampling frame for the participant customer satisfaction survey. In many
   instances, this date will be the same as the exit date. In situations where a case was ended,
   reopened within 90 days of the original closure date, and then ended again, the date used to
   determine inclusion in the sampling frame is the initial last expected service date.

   Response rate - The percentage of people in the sample who are contacted and respond to
   the core questions on the survey.

E. Calculation of the ACSI

   The ACSI scores represent the weighted sum of the three ACSI questions’ values which are
   transformed into 0 to 100 scale values. The weights are applied to each of the three
   questions to account for differences in the characteristics of the State’s customer groups.

   For example, assume the mean values of three ACSI questions for a state are:

        1. Overall Satisfaction        = 8.3
        2. Met Expectations            = 7.9
        3. Compared to Ideal           = 7.0

   Then, these mean values from raw data must first be transformed to the value on a 0 to 100
   scale. This is done by subtracting 1 from these mean values, dividing the results by 9
   which is the value of range of a 1 to 10 raw data scale, and multiplying the whole by 100:

        1. Overall Satisfaction        = (8.3 - 1)/9 x 100 = 81.1
        2. Met Expectations            = (7.9 - 1)/9 x 100 = 76.7
        3. Compared to Ideal           = (7.0 - 1)/9 x 100 = 66.7

   The ACSI score is calculated as the weighted averages of these values. Assuming the
   weights for the example state are 0.3804, 0.3247 and 0.2949 for questions 1, 2 and 3,
   respectively, the ACSI score for the state would be calculated as follows:

              (0.3804 x 81.1) + (0.3247 x 76.7) + (0.2949 x 66.7) = 75.4

   Weights were calculated by a statistical algorithm to minimize measurement error or
   random survey noise that exists in all survey data. State-specific weights were calculated
   using the relative distribution of ACSI respondent data for non-regulatory Federal agencies
   previously collected and analyzed by CFI and the University of Michigan.

   Specific weighting factors have been developed for each State. New weighting factors will
   be published annually.