Item Generation

The ASCQ-Me research team^[1] drafted questions for ASCQ-Me keeping several considerations in mind, including content validity, comprehension, and whether the format for administering the questions would be fixed or adaptive.

Content Validity

In this section we describe our approach to developing Sickle Cell Disease (SCD)-specific content, developing items based on patient input, complementing the content of the Patient-Reported Measurement Information System (PROMIS) item banks, and determining the appropriate recall period.

Attribution to Condition

Two basic approaches typically are taken to develop any condition- or disease-specific^[2] patient-reported outcome (PRO) or health-related quality of life (HRQOL) measure. These are methods based on:

1. Condition attribution, wherein the question is considered disease-specific because it attributes the symptom or deficit in functioning to a particular condition or disease. 
2. Content validity, wherein the question is considered disease-specific because it was generated from one-on-one or group interviews with people who have the condition.

ASCQ-Me is a measure of how adults with SCD feel and function, so we had to decide whether and how to build attribution to SCD into the questions. We took the second approach to generating questions for ASCQ-Me.  That is, we did not create SCD-specific questions by adding the phrase “because of my SCD”, rather, we based the content of questions on numerous indepth interviews with adults living with SCD.  Read more about why here.

Patient Input

In the previous section (Critical Incident Interviews), we described the critical incident technique (CIT) and how it contributed to the conceptual framework that the ASCQ-Me research team generated. This section demonstrates how the critical incidents were used to write items for ASCQ-Me. 

More than 1,213 incidents were generated from the CIT. From this collection, 830 questions were written. The example below illustrates how questions were generated from the CIT.

Question for the emotional impact item bank:

Critical incident

I had 5 close friends..[pass].. from sickle cell..it is hard for me to enjoy life sometimes because I am constantly trying to plan when my life is going to end.”

Question

In the past 7 days, how often did you feel that SCD could take your life at any moment?

Answer

Never –to– Always

Question for the stiffness item bank:

Critical incident

I wake up in the morning and feel like I’m 80 yrs. old. I am stiff. My back and joints will feel out of place. I have to spend a lot more time in the morning just to get started

Question

In the past 7 days, how often were your joints very stiff when you woke up?

Answer

Never –to– Always

Complementing PROMIS Content

The 830 items written from the CIT were reviewed for redundancy, and a reduced list was compared to the questions in PROMIS item banks that targeted similar health concepts:  PROMIS Anger/Aggression, Anxiety, Depression, Fatigue Experience, Fatigue Impact, Pain Quality, Pain Interference, Physical Functioning, Social Role Performance, and Sleep/Wake Functioning. The goal of this process was to determine whether ASCQ-Me had enough unique content to warrant developing an item bank. Items eliminated from the ASCQ-Me item set were items that were nearly identical to PROMIS items.  In fact,  the entire group of ASCQ-Me items targeting Fatigue and Pain Interference were redundant with those in the PROMIS item banks.

Item banks. Five item banks were developed for ASCQ-Me to cover emotional aspects of living with SCD (Emotional Impact), pain particular to SCD (Pain Impact), degree of stiffness and interference caused by stiffness (Stiffness Impact), interference with sleep characteristic of SCD (Sleep Impact), and particular ways that SCD interferes with or shapes social activities (Social Impact). At the time that we developed the ASCQ-Me questions, there was no finished PROMIS measure for cognitive functioning so we developed questions for that domain of health as well, but we did not find enough evidence of validity for the ASCQ-Me cognitive functioning item bank to recommend it for use. Read more about our work on assessing cognitive functioning here. In the meantime, PROMIS had developed two item banks for cognitive functioning. We recommend that these be tested in adults with SCD.

SCD severity items. We required a method of identifying groups of patients that might differ in their degree of disease severity so that we could evaluate that ability of ASCQ-Me measures to describe the impact of variations in SCD severity on the feeling and functioning of adults. The ASCQ-Me project team consulted with providers who treat patients with SCD to generate a list of questions that providers might ask in evaluating disease severity. These questions included six conditions (different types of organ damage, bone damage, and leg ulcers) and two treatments (daily pain medication and regular transfusions).

Pain episode items. Five questions asking about the frequency, severity, and duration of pain due to vaso-occlusive events were drafted based on how these events were described in the CII’s. These questions are meant to be an indicator of severity in addition to those listed in the SCD severity medical history checklist.

Quality of care items.  ASCQ-Me includes a fixed-format questionnaire, called the ASCQ-Me Quality of Care measure, to understand the health care experience of adults with SCD.  This measure is not integrated into the ASCQ-Me domain framework for health because patient reports of the quality of their care are theoretically distinct from PRO or HRQOL measures in that they are descriptions of care processes, not outcomes.  Donabedian’s often-cited model to support health-services research includes three categories of variables: structure, process, and outcome.^[3] Process measures include patient interactions with providers and what was done to the patient and when. Outcome measures include indicators of patient health following the delivery of health care. Patient reports can provide data for both health care process and outcome measures. The Consumer Assessments of Healthcare Providers and Systems (CAHPS) family of surveys is an example of measures based on patient reports of health care delivery process. PROMIS is an example of measures based on patient reports of their health. The ASCQ-Me QOC measure is based on patient reports of health care delivery process. PROMIS is an example of measures based on patient reports of their health. Find out more about the ASCQ-Me QOC measure.

Appropriate Recall Period

Traditionally, PRO or HRQOL questionnaires ask patients to respond to a time frame that would capture the state of well-being or functioning that is currently most characteristic of respondents and not unduly influenced by rare, acute symptoms. For this reason, a recall period that is as short as possible while being resistant to the influence of an immediate, but fleeting and uncharacteristic health event is desired. For example, the SF-36 contains many questions that ask the respondent to refer to the “past 4 weeks” (20 of 36 questions). But other SF-36 questions ask the respondent to refer to the unspecified present (15 of 36). And one question asks respondents to compare their health now to their health one year ago. By contrast,  PROMIS has chosen the “past 7 days” as the most common reference period for questions. However, some PROMIS domains use the unspecified present as well or a period of 30 days. 

In developing the time frame of questions for ASCQ-Me, we had the following objectives:

1) ASCQ-Me should use the shortest time frame possible to describe the person’s state of being that is currently characteristic of him or her and that is appropriate to the response scale; 2) Once decided, ASCQ-Me should not change the question reference period from one administration to another because such a change would make comparisons among administrations impossible (i.e., one would not know whether a difference in scores was caused by difference in the time frame of the questions or to the difference of interest); and 3) Finally, because ASCQ-Me was designed to complement PROMIS, we sought to adopt a time frame that is consistent with the PROMIS, as long as it is also appropriate to the ASCQ-Me content.

With input from the advisory committee of adults with SCD, guidelines for recall periods were drafted for question writers to follow:

The unspecified present should be used for items that refer to characteristic ways of thinking or behaving that are stable over time, such as:

• Ways of coping with SCD—coping mechanisms, disease management
• General self-perceptions—self image
• General perceptions of how others view the self—discrimination/stigmatization

The 7-day period should be used for experiences that have the potential to occur several times a day or week:

• Pain—this can occur several times a day or more than once in a 7-day period.
• Sleeping problems—we sleep (or try to sleep) at least 7 days a week.
• Fatigue—we can experience fatigue several times a day or more than once in a 7-day period.
• Difficulty thinking, reasoning, paying attention—we use these faculties several times a day and certainly more than once in a 7-day period.
• Depression, anxiety, anger—it is possible to have these emotions several times a day or several days out of 7.
• Problems performing at paid work, household work, or in school.

The 30-day time frame would be appropriate for social activities that are not certain to occur once a week:

• Engaging in recreational activities with friends
• Engaging in recreational activities with family
• Sexual behavior

A 12-month time frame would be appropriate for activities that are not certain to occur once a month:

• Engaging in community activities
• Career-related activities
• Changes in social networks
• Periods of absence from school—like career related
• Getting behind in school or work (due to absence)—like career related
• Hospitalizations, transfusions, other health care–related events

Evaluation of Question Comprehensibility

The director of AIR’s Cognitive Testing Lab reviewed all of the candidate ASCQ-Me items and identified those which had the slightest potential to be misunderstood.  These were included in the cognitive testing protocol.  Two rounds of cognitive testing using the highly sensitive “think-aloud^[4],[5] protocol were conducted. The first round covered the entire set of items recommended for testing  and the second round covered the changes that were made following the first round. In each round, the items to be tested were divided into two halves. One half was tested with six respondents and the other with five respondents, for a total of 11 respondents per round. Each round used a formal cognitive interview protocol to probe comprehension and navigation. Four trained cognitive interviewers conducted the 22 interviews on two coasts: North Carolina and California. All interviews were audio recorded and the notes from the interviews and audio-recordings were analyzed.

Results

Sickle cell disease severity items. The demographics section of the questionnaire included disease-severity items, which characterized the current status of disease progression among respondents. They determined whether the respondent had ever been diagnosed with a particular condition, in contrast to the HRQOL items, which assessed recent or current symptoms and ability to function. Many participants had trouble understanding the terms used to refer to different conditions or diagnoses (e.g., avascular necrosis). In most instances, this issue was addressed by providing lay descriptions of the different conditions followed by the technical name of the condition (in parentheses). For example, “Has a doctor or nurse ever told you that you have leg ulcers (open sores on your legs and feet)?” was rewritten as: “Have you ever had open sores on your legs or feet (leg ulcers)?”

Pain episode. The term “pain episode” was misunderstood by most respondents. When asked, “When was your last pain episode?” some interpreted it too narrowly, as only referring to instances when hospitalized, while others interpreted it too broadly, as referring to all types of pain, including pain that was mild or chronic. We used the term “pain episode” rather than “pain crisis” or “sickle cell crisis” in deference to many advocates who feel that “crisis” should not be used because it connotes an overwhelming and extremely dangerous situation, potentially reinforcing feelings of helplessness and hopelessness and preventing people from developing proactive coping strategies. The project advisory board was polled regarding its preferences for an alternative term, and a compromise solution was developed in which the phrase “…pain episode (crisis)” was used to refer to the sudden onset of severely debilitating pain.

Willingness to respond. All respondents were positively disposed toward completing the questionnaire and indicated that, if asked by their physician, they would complete the instrument. Respondents stated that a questionnaire like this might increase knowledge about SCD and could help others with the disease. At least two respondents noted that the questionnaire allowed them to tell people what it is like to live with SCD.

Health as the reason for limitations. For some items about limitations, it was determined that it is necessary to specify health as the reason for the limitation. For example, when asked about needing help doing household chores, one respondent indicated that help was needed because the house was too big for any one person to clean, not because of health or SCD. To attribute a need for help to health rather than an environmental contingency, the phrase “because of your health” was added to the end of items about role limitation, including household chores, grocery shopping, and going to appointments.

Types of Assessments

Two general types of assessments were developed: 1) item banks, which enabled computer adaptive testing (CAT) and 2) short, fixed-format questionnaires, which could be administered by computer or by paper and pencil.

Item Banks. The ASCQ-Me research team designed questions to target different levels of the continuum underlying functioning or severity of symptoms (e.g., from impossible to move to easy to move). This is consistent with the use of the item response theory modeling approach taken by PROMIS.  Using this approach, items for emotional impact, physical impact (including stiffness, sleep functioning, and pain), and social functioning were developed.

Short, Fixed-Format Questionnaires. AIR developed fixed-format questionnaires for severity of SCD, pain episodes, and quality of care.