2008 customer satisfaction score weights

Accessible letter PDF: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2009/TEGL07-09acc.pdf

CLASSIFICATION
 EMPLOYMENT AND TRAINING ADMINISTRATION            ASCI
             ADVISORY SYSTEM                       CORRESPONDENCE SYMBOL
        U.S. DEPARTMENT OF LABOR                   OWI-DAS
             Washington, D.C. 20210
                                                   DATE
                                                   September 30, 2009



ADVISORY:        TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 7-09

TO:              ALL STATE WORKFORCE AGENCIES
                 ALL STATE WORKFORCE LIAISONS

FROM:            JANE OATES /s/
                 Assistant Secretary

SUBJECT:         Updated Table of Weights for Use in Calculating State-Level
                 American Customer Satisfaction Index (ACSI) Scores for the
                 Participant and Employer Customer Satisfaction Surveys

1. Purpose. To provide states with the Program Year (PY) 2008 ACSI Table of
   Weights.

2. References. Workforce Investment Act of 1998 (WIA), Section 136; WIA Final
   Rule 20 CFR Part 666; Training and Employment Guidance Letter (TEGL)
   No. 17-05 (“Common Measures Policy for the Employment and Training
   Administration’s (ETA) Performance Accountability System and Related
   Performance Issues,” February 17, 2006).

3. Background. The Department uses the ACSI to meet the customer
   satisfaction measurement needs of WIA. The ACSI survey approach captures
   common customer satisfaction information that can be aggregated and
   compared at different levels. Please refer to the WIA Information
   Management System data collection package (OMB No. 1205-0420) for
   information on creating and administering the ACSI at the state level.

   The ACSI is a weighted score computed from the customer responses to a
   small set of required questions, with the weighted factors based on the
   relative distribution of ACSI respondent data for non-regulatory federal
   agencies previously collected and analyzed by the ACSI, LLC. The
   Department publishes new weighting factors annually.

RESCISSIONS                                      EXPIRATION DATE
TEGL No. 10-05                                   Continuing

                                         -2-

   It should be mentioned that unlike the previous two years, weights are now
   available for Alaska and Hawaii.

4. ACSI Table of Weights. Updated weights were calculated for use in PY 2008
   by a statistical algorithm in order to minimize random survey variation or
   measurement errors that exist in all survey data. The weights listed in
   attachment I are to be used by a state in computing performance on the ACSI
   for employers and participants and for the Annual Report for PY 2008. Also
   see attachment II for additional guidance.

5. Calculating DOL ACSI scores by using the weights. The weights for the 50
   states plus the District of Columbia are in the attached table. The ACSI scores
   of different states for DOL services are calculated by the weighted sum of the
   three ACSI question mean values (measured on 1-10 scales) transformed to 0-
   100 scale values.

   For example, if the mean values of the three ACSI questions for a state are:

      Overall satisfaction                                    8.3
      Confirmation/disconfirmation of expectations            7.9
      Comparison to ideal                                     7.0

   These mean values from raw data must first be transformed to the values on
   the 0-100 scale. This is done by subtracting 1 from the mean values, dividing
   the result by 9, which is the value range of the 1 to 10 raw data scale, and
   multiplying by 100.

   0 to 100 Overall Satisfaction = ((8.3 - 1)/9) x 100 = 81.1
   0 to 100 Confirmation/disconfirmation of expectations = ((7.9-1)/9) x 100 = 6.7
   0 to 100 Comparison to ideal = ((7.0-1)/9) x 100 = 66.7

    The ACSI score for the state as the weighted sum of these values would be:
   (.3911 x 81.1) + (.3283 x 76.7) + (.2806 x 66.7) = 75.6 (rounds to 76).

6. Action Requested. States shall distribute this guidance to all officials within
   the state who need such information to implement WIA customer satisfaction
   measures. States that do not have a waiver for Common Measures must
   comply with the survey’s requirements as stated in attachment D to TEGL 17-
   05, the Common Measures Policy for the Employment and Training
   Administration’s Performance Accountability System and Related
   Performance Issues. Those states that do not report to DOL on customer
   satisfaction should still administer customer satisfaction surveys. In addition,

                                          -3-

   states will need to footnote their customer satisfaction survey results in their
   WIA Annual Report narrative as follows: the response rate was “X” percent;
   results are not intended to make any inferences beyond the survey’s respondents.

7. Inquiries. Questions concerning this issuance may be directed to the
   appropriate Regional Office.

8. Attachments.

   I. ACSI Formula Weights for PY 2008
   II. WIA Guidance: Collecting Customer Satisfaction

Linked official document 1: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2009/TEGL07-09a1.pdf

ATTACHMENT I
                 ACSI Formula Weights for PY 2008
                 Satisfaction       Confirm            Ideal
State
Alabama          0.405295739      0.320805478       0.273898784
Alaska           0.342054981      0.363879607       0.294065412
Arizona          0.403608894      0.320108893       0.276282213
Arkansas         0.407205706      0.326607333       0.266186961
California       0.399858744      0.329241281       0.270899975
Colorado         0.39631094       0.32611452        0.27757454
Connecticut      0.395958683      0.328082549       0.275958768
Delaware         0.393781952      0.324455987       0.281762061
DC               0.377423195      0.378787577       0.243789228
Florida          0.401969467      0.321907786       0.276122747
Georgia          0.405603158      0.32155544        0.272841402
Hawaii           0.393547186      0.324338506       0.282114308
Idaho            0.386291279      0.320454798       0.293253923
Illinois         0.395024321      0.326329047       0.278646632
Indiana          0.398500236      0.323602652       0.277897113
Iowa             0.407603279      0.318639984       0.273756737
Kansas           0.388376347      0.333587268       0.278036385
Kentucky         0.398372441      0.322573893       0.279053666
Louisiana        0.397588786      0.327249796       0.275161418
Maine            0.403370625      0.335405489       0.261223886
Maryland         0.397162825      0.327001265       0.27583591
Massachusetts    0.395763869      0.336152812       0.268083319
Michigan         0.393269798      0.323865221       0.282864981
Minnesota        0.392708744      0.335092621       0.272198635
Mississippi      0.423182164       0.3162742        0.260543636
Missouri         0.397860007      0.323563038       0.278576955
Montana          0.399157944      0.351021542       0.249820515
Nebraska         0.39095626       0.328455287       0.280588454
Nevada           0.406750008      0.332865362       0.26038463
New Hampshire    0.398735217      0.338348605       0.262916178
New Jersey       0.399809275      0.328822544       0.271368181
New Mexico       0.392627272      0.331405898       0.27596683
New York         0.397819098      0.330428309       0.271752593
North Carolina   0.396280803      0.328632808       0.275086389
North Dakota     0.394397155      0.313171371       0.292431474
Ohio             0.398400957      0.323278311       0.278320732
Oklahoma         0.398513352      0.329012597       0.272474051
Oregon           0.402408121      0.326902365       0.270689514
Pennsylvania     0.400487205      0.32867245        0.270840346
Rhode Island     0.393520615      0.329008214       0.277471171
South Carolina   0.409993708      0.321622447       0.268383845
South Dakota     0.400272261      0.333890875       0.265836864
Tennessee        0.407262133      0.322288066       0.270449801
Texas            0.397075502      0.321660708        0.28126379
Utah             0.386166674      0.332648758       0.281184567
Vermont          0.402398577      0.331244046       0.266357377
Virginia         0.41283192       0.31909219        0.268075891
Washington       0.400066474      0.328420957       0.271512569
West Virginia    0.402558575      0.328104683       0.269336741
Wisconsin        0.396916861      0.324923098       0.278160042
Wyoming          0.400888619      0.340767109       0.258344273

Linked official document 2: https://www.dol.gov/sites/dolgov/files/ETA/advisories/TEGL/2009/TEGL07-09a2.pdf

ATTACHMENT II




GUIDANCE ON CAPTURING CUSTOMER
     SATISFACTION RESULTS

                Using the American Customer Satisfaction Index
                       to Measure Customer Satisfaction


A. Overview of Measurement Approach

   The Employment and Training Administration (ETA) will use the American Customer
   Satisfaction Index (ACSI) to meet the customer satisfaction measurement needs of the
   Workforce Investment Act (WIA). The survey approach that will be utilized allows
   grantees flexibility and, at the same time, captures common customer satisfaction
   information that can be aggregated and compared at a grant recipient, state, regional and
   national levels. This will be done through the use of a small set of required questions that
   will form a customer satisfaction index. ETA will use the ACSI, which is created by
   combining scores from three specific questions that address different dimensions of
   customers' experience.

   The ACSI is a widely used customer satisfaction measurement approach. It is used
   extensively in the business community, including at over 150 Fortune 500 companies, and
   in many European countries. Twenty-nine agencies of the Federal government have used
   the ACSI. The ACSI will allow the workforce investment system to not only look at
   performance within the system, but also be able to gain perspective on the workforce
   system’s performance by benchmarking against organizations and industries outside of the
   system. The ACSI also has a history of being useful in tracking change in customer
   satisfaction over time, making it an ideal way to gauge grantees’ progress toward
   continuously improving in performance.

   Since the ACSI trademark is proprietary property of the ACSI, LLC, ETA established a
   license agreement that will allow grant recipients the use of the ACSI for a statewide
   sample of participants and employers.

   It is worthwhile to note that WIA state grantees may use the ACSI questions and approach
   to measure satisfaction with WIA services at the local board level if individual license
   agreements are purchased from ACSI, LLC to ensure the questions and measurement
   approach are uniformly administered throughout the state. States using the ACSI
   methodology to measure local board performance are not required to conduct separate
   statewide customer satisfaction surveys to obtain State level WIA customer satisfaction
   results as long as:

      •   the surveys are administered by telephone,

      •   respondents are contacted within the window of opportunity outlined for
          participants and employers, at least 500 surveys are completed statewide for each
          group -- participants and employers,

      •   the surveys comply with the methodology spelled out in this guidance, and

      •   the Local Board results can be weighted to account for sampling differences, if
          appropriate, and aggregated to obtain state level outcomes.

   State grantees following this guidance may aggregate local outcomes to obtain State level
   outcomes on the customer satisfaction measures. These aggregated results will be reported
   in the quarterly and annual reports.


B. WIA Participant Customer Satisfaction Measures

   The weighted average of participant ratings on each of the three questions regarding overall
   satisfaction is reported on a 0-100 point scale. The score is a weighted average, not a
   percentage.

    1. Who Will Be Surveyed?

      WIA Title I-B participants with a last expected service date are eligible to be chosen for
      inclusion in the random sample. WIA Title I-B participants include individuals who
      are either adults, dislocated workers, older youth (ages 19-21), and younger youth (ages
      14-18).

    2. How Many (Number to be Surveyed)?

      States may randomly sample individuals to obtain the desired number of completed
      surveys. States are required to determine the appropriate sample sizes and sampling
      percentages using the required response rates and the required number of interviews.
      500 completed participant surveys must be obtained each year for calculation of the
      measures. Participants should be contacted as soon as possible on or after the last
      expected service date and no later than 60 days after the closure date. Closure date is
      another term for the last expected service date.

      A completed survey is defined as a survey in which all three questions regarding
      overall satisfaction have been answered. The standard of 500 from a sample of the
      whole population of customers provides accuracy such that there is only a 5 in 100
      chance that the results would vary by more than ±5 points from the score obtained from
      surveying the whole population.

    3. How (methodology) and When to Conduct Surveys?

      To ensure ACSI results for individuals are collected in a consistent and uniform
      manner, procedures used by states to obtain participant customer satisfaction
      information must satisfy the following criteria:

          •   Participant customer satisfaction responses must be collected by way of
              telephone interviews. In-person interviews and mail questionnaires may be
              used only in situations where the individual does not have a telephone;

•   Participants should be contacted as soon as possible on or after the last expected
    service date and no later than 60 days after the closure date. Closure date is
    another term for the last expected service date;

•   States must complete a minimum of 500 surveys during the program year to
    accurately assess performance on the ACSI;

•   If sampling is used, states must randomly select participants to obtain the
    desired number of completed surveys. States are required to determine the
    appropriate sample sizes and sampling percentages using the required response
    rate and the required number of completed interviews;

•   The sampling methodology used to select potential respondents for the surveys
    must ensure the consistent random selection of a sample of participants eligible
    for the surveys. With this in mind, no individual in the participant group eligible
    for the survey may be arbitrarily excluded from a sample. Procedures used to
    draw a sample must conform to accepted statistical practices, such as using a
    table of random numbers. Participant samples must be drawn on at least a
    monthly basis;

•   The minimum response rate is 70 percent. The response rate is calculated as the
    number of respondents with complete customer satisfaction information divided
    by the total number in the sampling frame. A survey is considered complete
    where valid answers are provided by respondents for each of the core questions
    outlined by ETA;

•   It is very important that all records for eligible participants be entered in a
    timely fashion into the databases. The information in these databases will be
    used to develop lists of participants who received services that are the subject of
    the survey. Every precaution must be taken to prevent a response bias;

•   States must develop a process for creating an up-to-date customer list, capturing
    information on each customer’s address and telephone number. Local programs
    should collect alternate contact information from a person known by the
    customer who would know the whereabouts of the customer in the event the
    customer cannot be reached at the address and telephone number recorded for
    the customer;

•   A letter in advance of the survey should be sent out informing the customer that
    he or she can expect to receive a telephone call about his or her satisfaction with
    the services. States should place a phone number on the letter, suggesting that
    customers call if they need help or call if their phone numbers changed;

•   Attempts must be made to contact all eligible participants in the sample to avoid
    introducing bias. A minimum of five follow-up attempts is required, involving
    various times of the day before closing the record;

       •    Respondents must be told that responding to the survey is voluntary and that the
            information they provide will be kept confidential. However, states must not
            make any assurances regarding confidentiality without ensuring that they have
            the legal authority to make such assurances. Such authority can be found in
            state privacy laws, for example;

       •    States must use the minimum core customer satisfaction questions specified by
            ETA in collecting and reporting satisfaction outcomes. These mandatory items
            should be located at the beginning of the questionnaire. States may include
            other questions about the service experience; and

       •    The interview should be limited to 15 minutes or less. Shorter questionnaires
            produce better response rates when compared to longer questionnaires.

4. What are the Core Questions?

   The following introductory statement and core ACSI questions are to be read to the
   respondent at the beginning of the interview1. The introductory script sets the context
   for the interview by focusing the interview on the service experience, date(s) of service,
   and the firm(s) or organization(s) delivering the service(s). In addition to contact
   information, the survey administrators will need to know the service(s) provided to
   each customer, the date(s) of service, and the firm(s) providing the service(s) in order
   to complete the interviews. The service(s) should be worded in terms recognizable to
   the customer.

   [Introductory Script]

   My name is (interviewer) with (firm or agency) and I am conducting a survey for the
   (program name). I would like to speak to Ms./Mr. (name of participant).

   Are you the Ms./Mr. (name of participant) who received a (name of service(s)) from the
   on (date or date range)? [IF THE RESPONDENT CONFIRMS PARTICIPATION, PROCEED WITH
   THE INTERVIEW. IF NO, END THE INTERVIEW AND THANK THE RESPONDENT FOR HIS OR
   HER TIME.]

   I would like to ask you some questions about your recent experience with the services
   provided to you. Our purpose is to learn from you how to improve programs and
   services offered to people. The survey was approved by the Office of Management and
   Budget under OMB Control No. 1205-0420, expiring 02/28/2009 to collect information
   on your experience with services we provide to individuals. Without this approval, we


              1
                  Note: The first question can be modified to suit the individual needs of the State and the
    names for program services recognizable for their population. The lead-in question provided is a model
    to be used as guidance.

would not be able to conduct this survey. The questionnaire is voluntary and should
take no longer than 15 minutes to complete.

Please know the answers you provide to my questions will be kept confidential. If you
have any questions regarding this estimate or any other aspect of this data collection,
including suggestions for reducing this burden, send them to the U.S. Department of
Labor, Room S-4231, Washington, D.C. 20210 (Paperwork Reduction Project 1205-
0420). Do you have any questions for me before we start the interview? [IF NO,
CONTINUE. IF YES, PROBE TO UNDERSTAND THE QUESTIONS, PROVIDE ANSWERS AND
RESTATE THE IMPORTANCE OF THE RESPONDENT’S PARTICIPATION IN THE INTERVIEW.]

[The Questions]

First, I want you to rate your overall experience with the services. I will read to you
three statements and ask you to rate your experience.

Utilizing a scale of 1 to 10 where a 1 means “Very Dissatisfied” and 10 means “Very
Satisfied” what is your overall satisfaction with the services provided from     ?

Very                                                              Very    DK2         REF3
Dissatisfied                                                 Satisfied
1        2            3        4           5   6   7   8   9        10         11         12

Considering all of the expectations you may have had about the services, to what extent
have the services met your expectations? A 1 now means “Falls Short of Your
Expectations” and 10 means “Exceeds Your Expectations.”

Falls Short of                                                Exceeds     DK         REF
Expectations                                               Expectations
1      2       3               4           5   6   7   8    9        10   11         12

Now think of the ideal program for people in your circumstances. How well do you
think the services you received compare with the ideal set of services? A 1 now means
“Not very close to the Ideal” and 10 means “Very Close to the Ideal.”

Not Close                                                  Very Close     DK        REF
To Ideal                                                     To Ideal
1        2            3        4           5   6   7   8   9      10      11        12




             2
                 DK = Don’t Know

             3
                 REF = Refused to Answer

C. Measuring One-Stop Employer Customer Satisfaction

   The weighted average of employer ratings on each of the three questions regarding overall
   satisfaction is reported on a 0-100 point scale. The score is a weighted average, not a
   percentage.

    1. Who Will Be Surveyed?

      One-Stop Employers

      Those eligible for surveying include employers who received a substantial service
      where the service has been completed or, if it is an ongoing service, when a full
      segment of service has been provided. All employers who received a substantial
      service involving personal contact with WIA staff are eligible to be chosen for
      inclusion in the random sample. Examples of substantial services include staff
      facilitated job orders, customized job training, customized labor market information
      requests, and on-the-job training activities.

      When an employer has received multiple services, priority should be given to the
      service that required the greatest expenditure of financial assistance or staff time4, and
      the survey conducted regarding their satisfaction with that service. An employer
      should only be surveyed one time during the course of the program year.

    2. How Many (number obtained)?

      Except in small states, a sample will be taken from these employers. At least 500
      completed interviews are required for the employer survey for calculation of the
      indicator. States are required to determine the appropriate sample sizes and sampling
      percentages using the required response rate and the required number of completed
      interviews. The sampling percentage should be constant over time to avoid under-
      representing or over-representing time intervals. A completed employer survey is
      defined as a survey in which all three questions regarding overall satisfaction have been
      answered. The standard of 500 from a sample of the whole population of customers
      provides accuracy such that there is only a 5 in 100 chance that the results would vary
      by more than 5 points from the score obtained from surveying the whole population.




                  4
                      Where an employer has received multiple services in a given time period, and there is
        separate contact information for each service, the contact information for the priority service should be
        used for surveying.

3. How (methodology) and When to Conduct Surveys?

   To ensure ACSI results for individuals are collected in a consistent and uniform
   manner, procedures used by states to obtain employer customer satisfaction information
   must satisfy the following criteria:

      •   Employer customer satisfaction responses must be collected by way of
          telephone interviews. In-person interviews and mail questionnaires may be
          used only in situations where the individual does not have a telephone;

      •   Employers should be contacted as soon as possible after the completion of the
          service and no later than 60 days after the completion of the service. For
          employers who listed a job order where no referrals were made, contact should
          occur 30 to 60 days after a job order was listed;

      •   States must complete a minimum of 500 employer surveys during the program
          year to accurately assess performance on the ACSI;

      •   If sampling is used, states must randomly sample employers to obtain the
          desired number of completed surveys. States are required to determine the
          appropriate sample sizes and sampling percentages using the required response
          rate and the required number of completed interviews;

      •   The sampling methodology used to select potential respondents for the surveys
          must ensure the consistent random selection of a sample of employers eligible
          for the surveys. With this in mind, no individual in the employer group eligible
          for the survey may be arbitrarily excluded from a sample. Procedures used to
          draw a sample must conform to accepted statistical practices, such as using a
          table of random numbers;

      •   Employer samples must be drawn on at least a monthly basis. To obtain
          sufficient numbers, smaller states will need to survey on an ongoing basis;

      •   The minimum response rate for the employer survey is 70 percent. The response
          rate is calculated as the number of respondents with complete customer
          satisfaction information divided by the total number in the sampling frame. A
          survey is considered complete where valid answers are provided by respondents
          for each of the core questions by ETA;

      •   It is very important that all records for eligible employers be entered in a timely
          fashion into the databases. The information in these databases will be used to
          develop lists of employers who received services that are the subject of the
          surveys. Every precaution must be taken to prevent a response bias.

       •    States must develop a process for creating an up-to-date customer list, capturing
            information on each customer’s address and telephone number. Local programs
            should collect alternate contact information from a person known by the
            customer who would know the whereabouts of the customer in the event the
            customer cannot be reached at the address and telephone number recorded for
            the customer.

       •    A letter in advance of the survey should be sent out informing the customer that
            he or she can expect to receive a telephone call about his or her satisfaction with
            the services. States should place a phone number on the letter, suggesting that
            customers call if they need help or call if their phone numbers changed.

       •    Attempts must be made to contact all eligible employers in the samples to avoid
            introducing bias. A minimum of five follow-up attempts is required, involving
            various times of the day before closing the record.

       •    Respondents must be told that responding to the survey is voluntary and that the
            information they provide will be kept confidential. However, states must not
            make any assurances regarding confidentiality without ensuring that they have
            the legal authority to make such assurances. Such authority can be found in
            state privacy laws, for example.

       •    States must use the minimum core customer satisfaction questions specified by
            ETA in collecting and reporting satisfaction outcomes. These mandatory items
            should be located at the beginning of the questionnaire. States may include
            other questions about the service experience.

       •    The interview should be limited to 15 minutes or less. Shorter questionnaires
            produce better response rates when compared to longer questionnaires.

4. What are the Core Questions?

   The following introductory statement and core ACSI questions are to be read to the
   respondent at the beginning of the interview5. The introductory script sets the context
   for the interview by focusing the interview on the service experience, date(s) of service,
   and the firm(s) or organization(s) delivering the service(s). In addition to contact
   information, the survey administrators will need to know the service(s) provided to
   each customer, the date(s) of service, and the firm(s) providing the service(s) in order
   to complete the interviews. The service(s) should be worded in terms recognizable to
   the customer.


              5
                  Note: The first question can be modified to suit the individual needs of the State and the
    names for program services recognizable for their population. The lead-in question provided is a model
    to be used as guidance.

[Introductory Script]

My name is (interviewer) with (firm or agency) and I am conducting a survey for the
(program name). I would like to speak to Ms./Mr. (name of employer contact).

Are you the Ms./Mr. (name of contact) who received a (name of service(s)) from the
on (date or date range)? [IF THE RESPONDENT CONFIRMS PARTICIPATION, PROCEED WITH
THE INTERVIEW. IF NO, END THE INTERVIEW AND THANK THE RESPONDENT.]

I would like to ask you some questions about your recent experience with the services
provided to your business. Our purpose is to learn from you how to improve programs
and services offered to area businesses. The survey was approved by the Office of
Management and Budget under OMB Control No. 1205-0420, expiring 02/28/2009 to
collect information on your experience with services we provide to employers.
Without this approval, we would not be able to conduct this survey. The questionnaire
is voluntary and should take no longer than 15 minutes to complete.

Please know the answers you provide to my questions will be kept confidential. If you
have any questions regarding this estimate or any other aspects of this collection,
including suggestions for reducing this burden, send them to the U.S. Department of
Labor, Room S-4231, Washington, D.C. 20210 (Paperwork Reduction Project 1205-
0420). Do you have any questions for me before we start the interview? [IF NO,
CONTINUE. IF YES, PROBE TO UNDERSTAND THE QUESTIONS, PROVIDE ANSWERS AND
RESTATE THE IMPORTANCE OF THE RESPONDENT’S PARTICIPATION IN THE INTERVIEW.]

[The Questions]

First, I want you to rate your overall experience with the services. I will read to you
three statements and ask you to rate your experience.

Utilizing a scale of 1 to 10 where 1 means “Very Dissatisfied” and 10 means “Very
Satisfied” what is your overall satisfaction with the services provided from   ?

Very                                                             Very
Dissatisfied                                                   Satisfied   DK6    REF7
1        2          3        4           5   6   7   8     9        10       11    12




          6
               DK = Don t Know

          7
               REF = Refused to Answer

      Considering all of the expectations you may have had about the services, to what extent
      have the services met your expectations? A 1 now means “Falls Short of Your
      Expectations” and 10 means “Exceeds Your Expectations.”

       Falls Short of                                                    Exceeds      DK    REF
      Expectations                                                     Expectations

      1        2        3    4       5      6       7      8       9        10         11    12

      Now think of the ideal program for people in your circumstances. How well do you
      think the services you received compare with the ideal set of services? A 1 now means
      “Not very close to the Ideal” and 10 means “Very Close to the Ideal.”

      Not Close                                                        Very Close     DK
   REF
      To Ideal                                                           To Ideal
      1        2        3    4       5      6       7      8       9        10        11      12




D. Definition of Terms

   Sample - A group of cases selected from a population by a random process where everyone
   has an equal probability of being selected.

   Last Expected Service – Occurs when the participant completes the activities outlined in his
   or her service strategy or service plan and there are no additional services expected other
   than supportive or follow-up services. Last expected service may also occur in situations
   where the participant voluntarily or involuntarily discontinues his or her participation in
   services outlined in the service plan.

   Last Expected Service Date – This date is used to determine when a customer becomes a
   part of the sampling frame for the participant customer satisfaction survey. In many
   instances, this date will be the same as the exit date. In situations where a case was ended,
   reopened within 90 days of the original closure date, and then ended again, the date used to
   determine inclusion in the sampling frame is the initial last expected service date.

   Response rate - The percentage of people in the sample who are contacted and respond to
   the core questions on the survey.

E. Calculation of the ACSI

   The ACSI scores represent the weighted sum of the three ACSI questions’ values which are
   transformed into 0 to 100 scale values. The weights are applied to each of the three
   questions to account for differences in the characteristics of the State’s customer groups.

   For example, assume the mean values of three ACSI questions for a state are:

        1. Overall Satisfaction       = 8.3
        2. Met Expectations           = 7.9
        3. Compared to Ideal          = 7.0

   Then, these mean values from raw data must first be transformed to the value on a 0 to 100
   scale. This is done by subtracting 1 from these mean values, dividing the results by 9
   which is the value of range of a 1 to 10 raw data scale, and multiplying the whole by 100:

        1. Overall Satisfaction       = (8.3 - 1)/9 x 100 = 81.1
        2. Met Expectations           = (7.9 - 1)/9 x 100 = 76.7
        3. Compared to Ideal          = (7.0 - 1)/9 x 100 = 66.7

   The ACSI score is calculated as the weighted averages of these values. Assuming the
   weights for the example state are 0.3804, 0.3247 and 0.2949 for questions 1, 2 and 3,
   respectively, the ACSI score for the state would be calculated as follows:

              (0.3804 x 81.1) + (0.3247 x 76.7) + (0.2949 x 66.7) = 75.4

   Weights were calculated by a statistical algorithm to minimize measurement error or
   random survey noise that exists in all survey data. State-specific weights were calculated
   using the relative distribution of ACSI respondent data for non-regulatory Federal agencies
   previously collected and analyzed by ACSI, LLC.

   Specific weighting factors have been developed for each State. New weighting factors will
   be published annually.