TECH-AID INSTITUTE RESEARCH REPORT
LIVE SMART LIVE SAFE: PREVENTING RESPIRATORY ILLNESS

The Study

We conducted an evaluation with 40 individuals with mild intellectual disabilities (ID). Our design consisted of a one-group Pre-Post-Follow-Up evaluation. The Posttest took place immediately after a participant used the program; the Follow-Up evaluation took place 2 weeks after the Posttest. Measures included structured interviews, assessor observations, and a comprehension test administered via computer. All evaluation activities took place in the participants’ homes.

Recruitment and Sample Demographics

Participants were recruited via flyers sent to local organizations serving people with mild ID (e.g. The ARC, local residential programs and service brokerages), through previous contacts and via word-of-mouth by participants. In the case of recruitment through organizations, we asked our agency contact to present the study possibility to a potential participant and to obtain their permission to pass on their name and phone number to Tech-Aid. Word-of-mouth referrals took the form of interested persons contacting Tech-Aid directly via phone. At the initial phone contact, details of the study were explained to callers; if they indicated they were interested, we obtained contact information and scheduled them for the baseline assessment. Informed consent was obtained at the initial home visit.

Participants were compensated after completion of the Baseline and Exit evaluations and at completion of the Follow-Up evaluation. In addition to cash payments, participants who did not have a thermometer at the time of the Follow-Up assessment were allowed to keep the thermometer they’d used at the initial assessment. At Baseline, we also provided each participant with a copy of the Health Map and 4 copies of pictorial shopping lists for Health Map recommended supplies.

Fifty subjects were initially recruited to take part in the evaluation. Five people dropped out before baseline, four missed two baseline evaluation appointments, and two voluntarily quit during baseline, representing an attrition rate of 22%. The two participants that quit during the baseline assessment were older, had a history of lengthy institutionalization, and were unable to use the computer program independently. Thirty-nine subjects completed the Baseline, Exit and Follow-Up evaluations. 54% of the participants were women, 46% men. They ranged in age from 21 to 61 (mean age 41.1, s.d. 10.1). 69% reported being non-Hispanic, but 31% did not know their ethnicity. 62% reported their race as Caucasian, 2.6% as African-American, 5% as Native American, 2.6% as mixed race Caucasian/African American, 5% as mixed race Caucasian/Native American; 23% did not know their race.

67% of the sample lived independently, 15% semi-independently, 8% in foster homes, 8% at home with their parents, and 2.6% had been living independently but had a housing crisis and were living with friends at the time of the evaluation. Only 39% were employed. Of those individuals who were employed, 1 (2.6%) worked in a sheltered employment setting, 5 (12.8%) worked at a supported community job, 7 (18%) worked at an independent (non-supported) community job and 1 (2.6%) worked at home making crafts to sell. 64% were single, 15% were married, and 21% had live-in partners. 28% were first time computer users.

Procedures

In-home evaluation appointments were made with participants via phone calls. Tech-Aid assessors (3 sub-contractors with extensive experience with the population) participated in a 3-hour training on conducting evaluations and carried out all the in-home evaluation activities. Assessors met weekly with Tech-Aid to turn in data and discuss any issues or notable observations. The first part of the initial home visit consisted of 1) obtaining informed consent. Two forms of consent were obtained as advised by our Institutional Review Board (IRB). The first consent was an “in-home” consent that clearly explained that Tech-Aid staff were mandatory reporters, and that the participants did not need to let us in their home, that they could say no thanks at that point, and the assessor would leave. The second consent was specifically related to being a research participant in this study. Subsequent to obtaining consent, the assessors 1) conducted the Baseline Interview, 2) presented the computer Pretest, 3) completed the Observation Questionnaire as the participant used the program, 4) administered the User Satisfaction and Exit Interviews and 5) presented the computer Posttest. At the Baseline evaluation, if a participant did not have a thermometer, a new still-in-box thermometer was provided so that we could establish the participant’s skills at taking their temperature, reading the thermometer and interpreting the results. That thermometer was then cleaned with alcohol wipes, labeled with the participant’s id number and returned to him/her at Follow-Up. These appointments took an average of 3 hours each.

Follow-Up evaluations consisted of conducting the Follow-Up Interview and presentation of the computer Follow-Up test. These appointments averaged less than an hour.

Measures

Six measures were developed by project staff: 1) a Baseline Interview, 2) a Computer Pre-, Post- and Follow-Up Test, 3) an Observation Questionnaire, 4) a User Satisfaction Interview, 5) an Exit Interview, and 6) a Follow-Up Interview. For ease of administration, the Baseline Interview, User Satisfaction Interview and Exit Interview were combined in one document.

Baseline Interview: This interview consists of eight sections: 1) Demographics, 2) Recall of in-home supplies needed, 3) Actual in-home supplies, 4) Taking ones temperature and interpreting the results, 5) Identifying symptoms on the Health Map and interpreting what to do, 6) Identifying whether the participant has a regular physician and if they know his/her name, 7) Assessing whether the participant is a first-time computer user and 8) Interview questions for the computer assessment. Five scores were computed from this interview, all as the sum of the number of correct responses within that section of the interview: 1) Recall of needed supplies, 2) Actual supplies on hand, 3) Temperature Taking skills, 4) Symptom Identification, and 5) Correct answers to the computer vignette questions.

Computer Assessment: This measure consists of 19 items presented via computer with three additional questions administered by assessors as part of the Interview. The first three questions are related to a vignette developed by project staff; participants watch an interaction and are then asked to identify a) whether any mistakes were made and if so, b) what those mistakes were. Of the remaining 16 items, 7 were taken directly from the program, although 3 of those 7, asking participants to match thermometer readings to a sample did not use the same examples as those in the program. The remaining 11 items were developed by project staff and consist of both knowledge items and situational items. This assessment was administered at Baseline, Exit and Follow-Up. One score was computed from this instrument, the sum of the number of correct answers. Since 4 of the questions had multiple correct answers, scores can range from 0 to 25.

Observation Questionnaire: This questionnaire, completed by assessors, consists of 12 questions measuring the participant’s experience during program use. Questions range from documenting whether the participant had difficulty using the mouse and/or the interface, to whether the participant was frustrated, engaged or impatient during program use. In addition, assessors noted any difficulties encountered during program use, where in the program such difficulties occurred, and made notes of any unusual occurrences.

User Satisfaction Interview: The interview consists of 14 items assessing the participant’s reaction to the program. It was administered directly after the participant completed the program but before she/he completed the Exit Interview and the computer Posttest. Participants were asked whether they liked the program, if there was anything they particularly liked or disliked, if they had learned anything and if so, what. They were also asked to rate whether the program moved too fast or slowly, program length and difficulty, whether they’d like to use more programs like this one and whether they thought their friends would like the program.

Exit Interview: This interview consists of repeating four sections of the Baseline Interview: 1) Recall of in-home supplies, 2) Taking one’s temperature and interpreting the results, 3) Identifying symptoms on the Health Map and interpreting what to do, and 4) Interview questions for the computer assessment. Four scores were computed from this interview, matching the same four scores from Baseline: 1) Recall of needed supplies, 2) Temperature Taking skills, 3) Symptom Identification, and 4) Correct answers to the computer vignette questions. Actual supplies on hand was not measured at Exit, since this interview took place at the end of the Baseline assessment (i.e., first home visit) and participants’ hadn’t had a chance to go shopping.

Follow-Up Interview: This interview consists of six sections: 1) Use of the Health Map and shopping lists, 2) Recall of in-home supplies needed, 3) Actual in-home supplies, 4) Taking one’s temperature and interpreting the results, 5) Identifying symptoms on the Health Map and interpreting what to do, and 6) Interview questions for the computer assessment. Scores computed from this instrument repeated the five scores computed at Baseline. In addition, at his interview, assessors also documented whether the participant had retained the Health Map, had used the Health Map and if so, which sections, and whether they had used the shopping lists.

Results

Interview Scores:

Four scores were computed from the Baseline and Exit Interviews: Recall of needed supplies, Temperature Taking skills, Symptom Identification, and number of correct answers to the computer vignette questions. Repeated measures ANOVAs were conducted to analyze changes on these scores. Significant improvements were observed on all four scores. Cohen’s d values (Cohen, 1988) for these changes range from a high of 2.11 to a low of .58; Cohen characterizes these effect sizes as moderate to large. See Table 1 below.

Table 1: Baseline to Exit Interview Scores

Thermometer Skills

A Chi Square test was conducted to determine whether the number of participants who could correctly use the thermometer to take their temperature changed from Baseline to Exit. At Baseline, 44% could correctly take their temperature; at Exit, 77% correctly took their temperature (Chi Square (1)=9.76, p<.002). From Exit to Follow-Up, this measure increased to 82%: (Chi Square (1)=1.66E, p=.000).

We also examined two other questions related to Temperature Taking: 1) correctly reading the thermometer and 2) correctly interpreting the thermometer reading. At Baseline, 77% could correctly read the thermometer; at Exit, 90% could correctly read the thermometer (Chi Square (1)=5.10, p=.08). From Exit to Follow-Up, this measure decreased to 82% (Chi Square (1)=5.1, p=.024). At Baseline, 72% could correctly interpret the thermometer reading; at Exit, 90% could correctly interpret the reading (Chi Square (1)=4.82, p=.06). From Exit to Follow-Up, this had decreased to 85% (Chi Square(1)=1.06E, p=.026).

Interpreting the Health Map

A Chi Square test was conducted to determine whether the number of participants who would correctly interpret the Health Map symptom instructions had increased from Baseline to Exit. At Baseline, 62% could correctly interpret the instructions, at Exit, this had increased to 74% (Chi Square (1)=9.80, p=,003). From Exit to Follow-Up, this percentage decreased to 69% (Chi Square (1)=5.39, p=.02).

Assessor Observation Data

90% of participants accessed the program using a mouse, 8% used a track pad, and only 2.6% (1 participant) needed constant assessor assistance to use the program. Assessor assistance consists of the subject pointing out what to click on, and the assessor manipulating the mouse to click where indicated. 82% never needed assessor assistance to click on items, 8% had assistance a few times, and 5% had assistance about half the time. Assessors reported that 92% were engaged with the program, though 28% appeared frustrated at some point, and 26% appeared impatient at some point. 63% used the Health Map during the program. Written comments by assessors indicated that while some participants may have had some difficulties using the program, they all made comments about how much they liked the program.

User Satisfaction

All 39 participants said they liked the program, 25% a little, 74% a lot. 80% said the program taught them something. When asked what they learned, sample comments included “about the flu”, "airborne germs, spreading germs”, “when to call the doctor”, “flu shots are important”, "didn’t know germs live in water”, “don’t share drinks” and “if someone is sick, wash your hands, stay away, don’t go to work sick”. When asked if there was anything in particular they liked about the program, comments included “it answered questions and explained things”, the Health Map and temperature chart”, “how to use the thermometer”, "what food to have when you’re sick”, "it shows what happens when you don’t do the right things” and “it tells you everything you need to know”. 80% reported the program was very easy to use, 85% that the program moved at the right speed, and 74% that it was the right length (25% thought it was too long). 97% said they’d like to use more programs like this and all (100%) thought their friends would like the program.

Computer Test Scores

One score was computed from this test, the sum of the number of correct answers. A repeated measures ANOVA was conducted to analyze changes on this score between Baseline and Exit. At Baseline the mean score on this measure was 17.0 (s.d. 3.7). At Exit, the mean score was 19.7 (s.d. 2.86). This change was significant (F(1,38)=34.8, p<.000). Cohen’s d for this change is .82, characterized as a large effect.

Retention at Follow-Up

Repeated measures ANOVAs were conducted on scores from the Interview and Computer Assessment to determine whether participants had retained the changes observed between Baseline and Exit. On one score, Recall of Supplies, we observed a significant decrease between Exit and Follow-Up, indicating a lack of retention. However on 1 score, the actual Supplies (including both Health Kit and Food supplies) on Hand, the significant difference represented a positive change; on the other four scores, no significant changes were observed, indicating participants had retained the gains they made between Baseline and Exit. (Note: For the analysis of outcomes on Supplies on Hand, participants living at home with family or in foster homes were dropped.) Results of these analyses can be seen in Table 2 on the following page.

Other results:

Only 28% had a thermometer at the Baseline assessment, 46% had less than half of the recommended Health Kit supplies and 31% had less than half of the recommended Food Supplies. At Follow-Up, though these percentages increased, 38% still had less than half the recommended Health Kit supplies, and 46% still had fewer than half the recommended Food Supplies. Paired samples T-Tests used to examine changes in Health Kit and Food Supplies were non-significant. All participants had kept the Health Map supplied at Baseline, and 30% reported using it in the two weeks since Baseline.

Table 2: Retention (Exit to Follow-Up Scores)

Discussion

Our feasibility evaluation demonstrated that the prototype program had significant immediate (Baseline to Exit) outcome effects. Not only were these effects significant, they were also meaningful; Cohen’s d for two measures was moderate (.58, and .63) and for the other three was large (2.11, .95 and .82). At the two-week Follow-Up, participants retained the gains they had made on four of the six measures and increased what supplies they had on hand. Only four measures (recall of what supplies are needed, reading thermometer temperatures, interpreting thermometer readings, and interpreting Health Map symptom instructions) showed decreases, though none of the scores on these variables dropped below Baseline levels. Assessor observations documented the fact that almost all participants were able to access the program using the mouse and were engaged with the program while using it. Measures of User Satisfaction confirmed these observations; participants liked the program, felt they had learned something, and rated the program easy to use. Almost all said they would like to use more programs like this and all felt their friends would also like it.

Follow Us on Twitter