The ASCQ-Me research team[1]
drafted questions for ASCQ-Me keeping several considerations in mind, including
content validity, comprehension, and whether the format for administering the
questions would be fixed or adaptive.
Content Validity
In this section we describe our approach to developing
Sickle Cell Disease (SCD)-specific content, developing items based on patient
input, complementing the content of the Patient-Reported
Measurement Information System (PROMIS) item banks, and determining the
appropriate recall period.
Attribution to Condition
Two basic approaches typically are taken to develop any condition-
or disease-specific[2]
patient-reported outcome (PRO) or health-related quality of life (HRQOL) measure.
These are methods based on:
- Condition
attribution, wherein the question is considered disease-specific because
it attributes the symptom or deficit in functioning to a particular
condition or disease.
- Content validity,
wherein the question is considered disease-specific because it was
generated from one-on-one or group interviews with people who have the
condition.
ASCQ-Me is a measure
of how adults with SCD feel and function, so we had to decide whether
and how to build attribution to SCD into the questions. We took the
second approach to generating questions for ASCQ-Me. That is, we did not
create SCD-specific questions by adding the phrase “because of my SCD”, rather,
we based the content of questions on numerous indepth interviews with adults
living with SCD. Read more about why here.
Patient Input
In the previous section (Critical Incident Interviews), we
described the critical incident technique (CIT)
and how it contributed to the conceptual framework that the ASCQ-Me research
team generated. This
section demonstrates how the critical incidents were used to write items for
ASCQ-Me.
More than 1,213 incidents were generated from the CIT. From
this collection, 830 questions were written. The example below illustrates how
questions were generated from the CIT.
Question for the emotional impact item bank:
- Critical incident
- I had 5 close friends..[pass].. from sickle cell..it is hard for me to enjoy life sometimes because I am constantly trying to plan when my life is going to end.”
- Question
- In the past 7 days, how often did you feel that SCD could take your life at any moment?
- Answer
- Never –to– Always
Question for the stiffness item bank:
- Critical incident
- I wake up in the morning and feel like I’m 80 yrs. old. I am stiff. My back and joints will feel out of place. I have to spend a lot more time in the morning just to get started
- Question
- In the past 7 days, how often were your joints very stiff when you woke up?
- Answer
- Never –to– Always
Complementing PROMIS Content
The 830 items written from the CIT were reviewed for
redundancy, and a reduced list was compared to the questions in PROMIS item
banks that targeted similar health concepts: PROMIS Anger/Aggression, Anxiety,
Depression, Fatigue Experience, Fatigue Impact, Pain Quality, Pain
Interference, Physical Functioning, Social Role Performance, and Sleep/Wake
Functioning. The goal of this process was to determine whether ASCQ-Me had
enough unique content to warrant developing an item bank. Items eliminated from
the ASCQ-Me item set were items that were nearly identical to PROMIS items. In
fact, the entire group of ASCQ-Me items targeting Fatigue and Pain
Interference were redundant with those in the PROMIS item banks.
Item banks. Five item banks were developed for
ASCQ-Me to cover emotional aspects of living with SCD (Emotional Impact), pain
particular to SCD (Pain Impact), degree of stiffness and interference caused by
stiffness (Stiffness Impact), interference with sleep characteristic of SCD
(Sleep Impact), and particular ways that SCD interferes with or shapes social
activities (Social Impact). At the time that we developed the ASCQ-Me
questions, there was no finished PROMIS measure for cognitive functioning so we
developed questions for that domain of health as well, but we did not find
enough evidence of validity for the ASCQ-Me cognitive functioning item bank to
recommend it for use. Read more about our work on assessing cognitive functioning here.
In the meantime, PROMIS had developed two item banks for
cognitive functioning. We recommend that these be tested in adults with SCD.
SCD severity items. We required a method of
identifying groups of patients that might differ in their degree of disease
severity so that we could evaluate that ability of ASCQ-Me measures to describe
the impact of variations in SCD severity on the feeling and functioning of
adults. The ASCQ-Me project team consulted with providers who treat patients
with SCD to generate a list of questions that providers might ask in evaluating
disease severity. These questions included six conditions (different types of
organ damage, bone damage, and leg ulcers) and two treatments (daily pain
medication and regular transfusions).
Pain episode items. Five questions asking about the
frequency, severity, and duration of pain due to vaso-occlusive events were
drafted based on how these events were described in the CII’s. These questions
are meant to be an indicator of severity in addition to those listed in the SCD
severity medical history checklist.
Quality of care items. ASCQ-Me includes a
fixed-format questionnaire, called the ASCQ-Me Quality of Care measure, to
understand the health care experience of adults with SCD. This measure is not
integrated into the ASCQ-Me domain framework for health because patient reports of the quality of their care are
theoretically distinct from PRO or HRQOL measures in that they are descriptions
of care processes, not outcomes. Donabedian’s often-cited model to
support health-services research includes three categories of variables: structure, process, and outcome.[3]
Process measures include patient interactions with providers and what was done
to the patient and when. Outcome measures include indicators of patient health
following the delivery of health care. Patient reports can provide data for
both health care process and outcome measures. The Consumer Assessments of Healthcare Providers
and Systems (CAHPS) family of surveys is an example of measures based on
patient reports of health care delivery process. PROMIS is an example of
measures based on patient reports of their health.
The ASCQ-Me QOC measure is based on patient reports of health care delivery process. PROMIS is an example of measures based on patient reports of their health. Find out more about the ASCQ-Me QOC measure.
Appropriate Recall Period
Traditionally, PRO or HRQOL questionnaires ask patients to
respond to a time frame that would capture the state of well-being or
functioning that is currently most characteristic of respondents and not
unduly influenced by rare, acute symptoms. For this reason, a recall period
that is as short as possible while being resistant to the influence of an
immediate, but fleeting and uncharacteristic health event is desired. For
example, the SF-36 contains many questions that ask the respondent to refer to
the “past 4 weeks” (20 of 36 questions). But other SF-36 questions ask the
respondent to refer to the unspecified present (15 of 36). And one question
asks respondents to compare their health now to their health one year ago. By
contrast, PROMIS has chosen the “past 7 days” as the most common reference
period for questions. However, some PROMIS domains use the unspecified present
as well or a period of 30 days.
In developing the time frame of questions for ASCQ-Me, we
had the following objectives:
1) ASCQ-Me should use the shortest time frame possible to
describe the person’s state of being that is currently characteristic of him or
her and that is appropriate to the response scale; 2) Once decided, ASCQ-Me
should not change the question reference period from one administration to
another because such a change would make comparisons among administrations
impossible (i.e., one would not know whether a difference in scores was caused
by difference in the time frame of the questions or to the difference of
interest); and 3) Finally, because ASCQ-Me was designed to complement PROMIS,
we sought to adopt a time frame that is consistent with the PROMIS, as long as
it is also appropriate to the ASCQ-Me content.
With input from the advisory committee of adults with SCD,
guidelines for recall periods were drafted for question writers to follow:
The unspecified present should be used for items that
refer to characteristic ways of thinking or behaving that are stable over time,
such as:
- Ways
of coping with SCD—coping mechanisms, disease management
- General
self-perceptions—self image
- General
perceptions of how others view the self—discrimination/stigmatization
The 7-day period should be used for experiences that
have the potential to occur several times a day or week:
- Pain—this
can occur several times a day or more than once in a 7-day period.
- Sleeping
problems—we sleep (or try to sleep) at least 7 days a week.
- Fatigue—we
can experience fatigue several times a day or more than once in a 7-day
period.
- Difficulty
thinking, reasoning, paying attention—we use these faculties several times
a day and certainly more than once in a 7-day period.
- Depression,
anxiety, anger—it is possible to have these emotions several times a day
or several days out of 7.
- Problems
performing at paid work, household work, or in school.
The 30-day time frame would be appropriate for social
activities that are not certain to occur once a week:
- Engaging
in recreational activities with friends
- Engaging
in recreational activities with family
- Sexual
behavior
A 12-month time frame would be appropriate for
activities that are not certain to occur once a month:
- Engaging
in community activities
- Career-related
activities
- Changes
in social networks
- Periods
of absence from school—like career related
- Getting behind in school or work (due to
absence)—like career related
- Hospitalizations,
transfusions, other health care–related events
Evaluation of Question Comprehensibility
The director of AIR’s Cognitive Testing Lab reviewed all of
the candidate ASCQ-Me items and identified those which had the slightest
potential to be misunderstood. These were included in the cognitive testing
protocol. Two rounds of cognitive testing using the highly sensitive
“think-aloud[4],[5] protocol were conducted.
The first round covered the entire set of items recommended for testing and
the second round covered the changes that were made following the first round.
In each round, the items to be tested were divided into two halves. One half
was tested with six respondents and the other with five respondents, for a
total of 11 respondents per round. Each round used a formal cognitive interview
protocol to probe comprehension and navigation. Four trained cognitive
interviewers conducted the 22 interviews on two coasts: North Carolina and
California. All interviews were audio recorded and the notes from the
interviews and audio-recordings were analyzed.
Results
Sickle cell disease severity items. The demographics
section of the questionnaire included disease-severity items, which
characterized the current status of disease progression among respondents. They
determined whether the respondent had ever been diagnosed with a particular
condition, in contrast to the HRQOL items, which assessed recent or current
symptoms and ability to function. Many participants had trouble understanding
the terms used to refer to different conditions or diagnoses (e.g., avascular
necrosis). In most instances, this issue was addressed by providing lay
descriptions of the different conditions followed by the technical name of the
condition (in parentheses). For example, “Has a doctor or nurse ever told you
that you have leg ulcers (open sores on your legs and feet)?” was rewritten as:
“Have you ever had open sores on your legs or feet (leg ulcers)?”
Pain episode. The term “pain episode” was
misunderstood by most respondents. When asked, “When was your last pain
episode?” some interpreted it too narrowly, as only referring to instances when
hospitalized, while others interpreted it too broadly, as referring to all
types of pain, including pain that was mild or chronic. We used the term “pain
episode” rather than “pain crisis” or “sickle cell crisis” in deference to many
advocates who feel that “crisis” should not be used because it connotes an
overwhelming and extremely dangerous situation, potentially reinforcing
feelings of helplessness and hopelessness and preventing people from developing
proactive coping strategies. The project advisory board was polled regarding its
preferences for an alternative term, and a compromise solution was developed in
which the phrase “…pain episode (crisis)” was used to refer to the sudden onset
of severely debilitating pain.
Willingness to respond. All respondents were
positively disposed toward completing the questionnaire and indicated that, if
asked by their physician, they would complete the instrument. Respondents
stated that a questionnaire like this might increase knowledge about SCD and
could help others with the disease. At least two respondents noted that the
questionnaire allowed them to tell people what it is like to live with SCD.
Health as the reason for limitations. For some items
about limitations, it was determined that it is necessary to specify health as
the reason for the limitation. For example, when asked about needing help doing
household chores, one respondent indicated that help was needed because the
house was too big for any one person to clean, not because of health or SCD. To
attribute a need for help to health rather than an environmental contingency,
the phrase “because of your health” was added to the end of items about role
limitation, including household chores, grocery shopping, and going to
appointments.
Types of Assessments
Two general types of assessments were developed: 1) item
banks, which enabled computer adaptive testing (CAT) and 2) short, fixed-format
questionnaires, which could be administered by computer or by paper and pencil.
Item Banks. The ASCQ-Me research team designed
questions to target different levels of the continuum underlying functioning or
severity of symptoms (e.g., from impossible to move to easy to move). This is consistent
with the use of the item response theory modeling approach taken by PROMIS. Using
this approach, items for emotional impact, physical impact (including
stiffness, sleep functioning, and pain), and social functioning were developed.
Short, Fixed-Format Questionnaires. AIR developed
fixed-format questionnaires for severity of SCD, pain episodes, and quality of care.
[1]
The ASCQ-Me research team for this work was led from American Institutes for
Research and included San Keller, Roger Levine, Maureen Maurer, and Corrina
Moucheraud of American Institutes for Research; Ellen Werner of the National
Heart, Lung, and Blood Institute; Marsha Treadwell of UCSF Benioff Children’s
Hospital of Oakland; and Kathryn Hassell of the University of Colorado School
of Medicine.
[2]
We are using the terms “condition-specific” and “disease-specific”
interchangeably here because they are interchangeable with regard to SCD.
However, we note that ”condition-specific” includes states of being in addition
to disease such as particular disabilities, allergies, pregnancy, menopause,
obesity, and many others.
[4]
Ericsson, K., & Simon, H. (1980). Verbal reports as data. Psychological
Review, 87(3), 215–251.
[5]
Willis, G. B., Royston, P., & Bercini, D. (1991). The use of verbal report
methods in the development and testing of survey questionnaires. Applied
Cognitive Psychology, 5, 251–267.