Evaluating behavioral skills training with and without simulated in situ training for teaching safety skills to children.

Abstract

This study compared the effectiveness of behavioral skills training (BST) to BST plus simulated in situ training (SIT) for teaching safety skills to children to prevent gun play. The results were evaluated in a posttest only control group design. Following the first assessment, participants in both training groups and the control group who did not use the safety skills received in situ training. Following acquisition of the skills, dyad assessments were conducted in which a peer challenged the participant to play with the gun. The results showed that the training groups did not differ from the control group following training, but that the safety skill scores for the training groups and the control group increased significantly following in situ training. Furthermore, a high percentage of participants across groups used the skills during the dyad assessment.

**********

Hundreds of children are accidentally injured or killed by guns in this country each year (Eber, Annest, Mercy, & Ryan, 2004). Most often these injuries and deaths occur in the child's home or in a home of a close friend or relative where a child finds a handgun and plays with it (DiScala & Sege, 2004; Eber et al., 2004; Grossman, Reay, & Baker, 1999; Kellerman & Reay, 1986; Miller, Azrael, & Hemenway, 2001; Miller, Azrael, Hemenway, & Vriniotis, 2005; Ordog et al., 1988; Wintemute, Teret, Kraus, Wright, & Bradfield, 1987).

Recently, behavioral skills training procedures utilizing instructions, modeling, rehearsal, and feedback have been evaluated for teaching safety skills to children to prevent gun play (Miltenberger & Gross, in press). In these studies, children's safety skills are assessed with in situ assessments in which children find a gun without the knowledge that they are being assessed. The results from studies evaluating BST have been mixed, with some showing the success of BST and others showing that safety skills failed to generalize following BST (Himle, Miltenberger, Flessner, & Gatheridge, 2004; Himle, Miltenberger, Gatheridge, & Flessner, 2004; Gatheridge et al., 2004; Miltenberger et al., 2004).

In an attempt to promote generalization following BST, researchers have evaluated in situ training. In situ training is implemented when a child does not perform the skills correctly during an in situ assessment. The researcher catches the child in the presence of the gun, provides social disapproval at the child's failure to use the skills, and provides a brief training session in which the child must rehearse the correct skills five consecutive times while receiving feedback from the researcher. Following the implementation of in situ training in two studies, all children performed the safety skills correctly in subsequent in situ assessments (Himle, Miltenberger, Flessner, & Gather idge, 2004; Miltenberger et al., 2004)

Because this research showed that in situ training implemented after the use of BST enhanced the generalization of safety skills, Miltenberger et al. (2005) evaluated BST with in situ training integrated into the training protocol. All ten 4- and 5-year olds in this study learned the skills after a few training sessions and the skills generalized and maintained for three months. These results suggest that the use of in situ training can make BST more effective and efficient.

Thus far, research has shown that BST with in situ training is an effective method for training safety skills to prevent gun play to children. In addition, research shows that BST with in situ training is effective for teaching other safety skills (e.g., Johnson et al., 2005). However, implementing in situ training requires substantial time from researchers and teachers. In order to make this training more widely acceptable, training needs to be more efficient and presented in such a manner that parents or teachers are able to implement training on their own. One way in which training could become more efficient is by simulating in situ training so that each child does not have to receive training in naturalistic settings several times. Therefore the purpose of this study was to compare BST to BST with simulated in situ training (SIT).

Method

Participants and Setting

Participants were recruited from two after school childcare centers. All parents of 6- and 7-year-old children were asked if they would like their child to participate in the safety skills training program. Those who were interested signed consent forms for their children, and children over 7 years of age also signed assent forms. Children that had previously received behavioral skills training to prevent gun play from this research group were excluded from the current study; however if children who had received other training or no previous gun safety training were eligible. A total of 43 children participated in this study. A brief questionnaire was sent to parents of all participants to collect additional data on participant characteristics (e.g., number of guns in the home); however only 2 of 43 parents returned this questionnaire. Therefore, this information is not reported.

Peer confederates were same-aged peers who attended the same after school center and were in the same classroom as the participants. Peer confederates were chosen based on teacher recommendation and parent approval. They were trained, using BST, and assessed on the firearm safety skills. All confederates engaged in appropriate safety behaviors during an in situ assessment prior to participating in peer confederate training. Five peer confederates participated in this study.

All training and assessments occurred in classrooms at the childcare centers. Classroom locations were varied based upon availability.

Materials

Three real, but disabled firearms from the local police department were used for training and assessments. Each firearm had its trigger welded and ammunition magazine removed. All firearms were randomly chosen for training sessions and assessments, thus the presentation of each firearm was varied.

A training video that included four short scenes was used in the simulated in situ training (SIT) condition. The first scene depicted a 9-year-old child engaging in the correct safety skills upon finding an unattended firearm (not touching the gun, leaving the room, and telling a parent) and receiving praise from his father. Scene two showed the same scenario but the child touched the firearm and was caught by his father. The father provided a stern reprimand and told the child he would lose access to a favorite activity. Next, the same child was shown, but a 9-year-old friend was also present. The friend provided a verbal challenge for the child to play with the firearm (e.g., "Look, a gun. Do you want to play with it?"). The child engaged in the correct skills and was provided with praise from his father. Finally, in the last scene the child and friend were again shown, but the child did not engage in the safety skills, resulting in a reprimand and loss of activity from the father. The actors played the same roles in each video scene and all actors and actress were unknown to the study participants. Each of the videos was shot from the view of the child, in an attempt to make the safety skills exhibited by the child and praise and reprimands from the parent more salient for the viewers.

Target Behaviors and Data Collection

Participants were assessed using in situ assessments in which the child was unaware that an assessment was taking place. The researcher placed a real, but disabled, firearm in a location where the child was then asked to engage in a task (e.g., the gun was placed near toys that the child was asked to put away). The location for assessments and the placement of the disabled firearm were randomly chosen based upon availability and previous assessments with particular participants. Indeed, multiple assessments were not conducted in the same manner more than once for one participant. The child's response to the firearm was rated on a 4 point scale (i.e., 0 to 3). A score of 0 was given if the child touched the gun, regardless of any following behaviors. A score 1 was given if the child did not touch the gun, but did not get out of the area within 10 seconds and did not tell an adult about the presence of the firearm. A score of 2 was given if the child did not touch the firearm and left the area within 10 seconds, but did not tell an adult. Finally, a score of 3 was given if the child engaged in all safety behaviors of not touching, getting away within 10 seconds, and telling an adult.

Two types of in situ assessments were conducted, individual and dyad. In the individual assessment, the child found the gun while alone. In the dyad assessment, the child found the gun with a same age peer who was a confederate in the study. The confederate challenged the participant to play with the gun by picking up the disabled firearm and stating "Look, a gun. Do you want to play with it?" The peer confederates were chosen based upon availability for each assessment. When participants required more than one dyad assessment, a confederate who had not served as the confederate in previous assessments was chosen for the subsequent assessment.

Interobserver Agreement

Interobserver agreement was assessed by having a researcher independently view and score 59% of the individual assessments and 53% of the dyad assessments. The percentage of agreement was calculated for each observation by dividing the number of agreements for the 4 behaviors (did touch, did not touch, got away in 10 seconds, and told an adult) by agreements plus disagreements. Interobserver agreement was 100%.

Procedure

We used a posttest only control group design with two treatment groups, Behavior Skills Training (BST) and Simulated In situ Training (SIT). Participants were randomly assigned to each group resulting in 15 participants in the BST group, 12 participants in the SIT group, and 16 participants in the control group. Participants in the BST group and the SIT group received training based on the respective programs, while the control group did not receive any training prior to the initial assessment.

Participants from all three groups were assessed using in situ assessments. That is, they were assessed in a naturalistic setting, without knowledge of the assessment. If any participant did not receive a score of 3 during their first assessment, they received an in situ training session (see description below). Participants that required in situ training were assessed again at a later date. If they did not score a 3 during the assessment, they received another in situ training session and subsequent assessment. This process continued until participants received a score of 3 or until four in situ training sessions had been conducted for a participant. If a participant met the criterion score (i.e., 3) on an individual assessment, no further individual assessments were conducted. Dyad assessments were conducted for each participant who scored a 3 on their final individual assessment. For those participants not scoring 3 during the dyad assessment, up to two in situ training sessions and subsequent in situ assessments were conducted. Once a participant scored 3 during a dyad assessment, no further assessments were conducted.

BST. Participants were trained in small sub-groups (2-4 participants per sub-group). A BST program, employing instructions, modeling, rehearsal, and feedback, was used to teach participants skills to use if they ever encountered an unattended firearm. Similar to other firearm safety programs, the skills taught were to stop, don't touch, run away, and tell an adult upon finding a firearm (National Rifle Association Safety and Education Division, 2001). Researchers began each session discussing the dangers of firearms and telling the participants what to do if they ever encountered a firearm. Next the researchers modeled the correct behavior sequence of stopping, not touching, running away, and telling an adult. Then the participants were each given a chance to rehearse the safety skills in a role play that simulated finding a gun somewhere in their natural environment (home, garage, friends, relatives, school, daycare). After each role play, praise and corrective feedback was given to each child. Several role plays were conducted each session using a variety of simulated situations. Sessions were complete when each child had correctly rehearsed the safety skills 5 times. Three sessions were conducted with each sub-group and the session length ranged from 10-40 minutes.

SIT. Simulated in situ training was conducted in a similar format to that of BST, with three sessions lasting approximately 10-40 minutes. As in BST, instructions, modeling, rehearsal, and feedback were used to train the participants. However, there were three additional features in the SIT condition. One feature was telling participants about real life accidents involving injury and death that had occurred when other children had played with a firearm. These vignettes, pulled from actual situations reported in the media, were told to the participants during the instruction phase of sessions 2 and 3 in an attempt to enhance motivation to use the skills. The second difference in SIT compared to BST was the use of video clips that were shown to the participants during the 2nd and 3rd session. During session 2, the participants were shown the first two video scenes, in which the child was alone. In session 3, the second two clips in which the child was challenged by a friend were shown. Third, during session 3, the participants rehearsed the safety skills in role play scenarios that included a peer challenge.

Control. Participants in the control group did not receive any training prior to the first assessment.

In situ training. Following in situ assessments in which a child from any of the three groups did not engage in the safety skills (did not achieve a score of 3), in situ training was implemented. The researcher "caught" the child not engaging in the safety skills and expressed concern about the child being near the gun. The researcher then implemented a brief training session similar to a BST session. That is, the researcher provided instruction, although in less detail than in a standard BST session. Then, as in a standard BST session, the researcher required the child to rehearse the safety skills five consecutive times correctly while providing feedback.

Results

The cumulative mean individual assessment scores for each group (BST, SIT, control) in each assessment are shown in Figure 1. In addition, the cumulative percentage of individuals receiving scores of 0, 1, 2, and 3 in each assessment for each group is shown in Table 1. Although participants who had previously scored 3 on an assessment were not reassessed individually, their score was calculated as part of the cumulative group means and percentages for each subsequent assessment. A 3 X 2 mixed ANOVA was conducted for individual assessments, with (BST, SIT, and control) as the between subjects variable and time (assessments 1 and 2) as the with-in subject variable. A significant main effect was found for time, F(1, 24) = 43.90, p = .000. No significant main effect was found for group, however, a main effect for group approached significance F(1, 29) = 3.236, p = .082. No significant interaction between group and time was found.

[FIGURE 1 OMITTED]

Table 1
The cumulative percentage of children in each group scoring a 0, 1, 2,
and 3 in each individual assessment

            Assessment      1st           2nd           3rd
              Score     Assessment %  Assessment %  Assessment %

Control        0=          12.5           0.0            0.0
(n = 16) *     1=          81.25         31.25          12.5
               2=           0.0           6.25           0.0
               3=           6.25         62.25          87.5

BST            0=           6.67          0.0            0.0
(n = 15) *     1=          60.0           6.67           0.0
               2=           6.67         13.33           6.67
               3=          26.7          80.0           92.86

SIT            0=          16.67          0.0            0.0
(n = 12) *     1=          50.0           9.1            0.0
               2=          16.67          0.0            0.0
               3=          16.67         90.9          100.0

            Assessment Score  4th Assessment %  5th Assessment %

Control            0=               0.0                0.0
(n = 16) *         1=               6.25               0.0
                   2=               0.0                6.25
                   3=              87.5               87.5

BST                0=               0.0                0.0
(n = 15) *         1=               0.0                0.0
                   2=               0.0                0.0
                   3=             100.0              100.0

SIT                0=               0.0                0.0
(n = 12) *         1=               0.0                0.0
                   2=               0.0                0.0
                   3=             100.0              100.0

Note. BST = behavioral skills training and SIT = simulated in situ
training

* = Control group lost one participant prior to assessment 4; BST group
lost one participant prior to assessment 3; SIT group lost one
participant prior to assessment

Due to the significant main effect for time, additional group comparisons were conducted for individual assessments at posttest 1 and posttest 2. Scores for BST, SIT, and control did not significantly differ from one another at posttest 1 or posttest 2. However, scores for each group significantly differed from posttest 1 to posttest 2. BST F(1, 24) = 8.659), p = .007; SIT F(1, 20) = 14.335, p = .001; Control F(1, 31) = 21.270, p = .000. At the first posttest, only 27% of BST participants, 17% of SIT participants, and 6% of control participants achieved a score of 3. Following one in situ training session, the cumulative percentages were 80%, 82%, and 63% respectively. One hundred percent of participants in the SIT group and the BST group scored 3 after two in situ training sessions and 88% of the control group participants score 3 at this point.

The cumulative mean dyad assessment scores for each group (BST, SIT, control) in each assessment are shown in Figure 2. Additionally, the cumulative percent of individuals scoring 0, 1, 2, or 3 in each assessment for each group is shown in Table 2. As with individual assessments, a previous score of 3 for a participant was carried through to subsequent assessments for calculating the cumulative means and percentages for each group. A 3 X 2 mixed ANOVA was conducted for dyad assessments, with group (BST, SIT, and control) as the between subjects variable, and time (assessments 1 and 2) as the with-in subject variable. No significant main effects were found. During the first dyad assessment 85% of BST participants, 91% of SIT participants, and 79% of control participants scored a 3. The cumulative percentages were 92%, 100% and 93% respectively during the second dyad assessment, and 100% for all groups by the third dyad assessment.

[FIGURE 2 OMITTED]

Table 2
The cumulative percentage of children in each group scoring a 0, 1, 2,
and 3 during the dual assessments

         Assessment      1st           2nd           3rd
            Score    Assessment %  Assessment %  Assessment %

Control      0=          0.0           0.0           0.0
(n=14)       1=         14.29          7.14          0.0
             2=          7.14          0.0           0.0
             3=         78.57         92.86        100.0

BST          0=          0.0           0.0           0.0
(n=13)       1=         15.38          7.69          0.0
             2=          0.0           0.0           0.0
             3=         84.62         92.31        100.0

SIT          0=          0.0           0.0           0.0
(n=11)       1=         18.18          0.0           0.0
             2=          0.0           0.0           0.0
             3=         81.82        100.0         100.0

Note: BST = behavioral skills training and SIT = simulated in situ
training.

Discussion

This study compared the effectiveness of BST to BST with simulated in situ training (SIT) for teaching safety skills to prevent gun play. The results showed that following three training sessions of BST and SIT, there was no significant difference in demonstrated safety skills between group participants (BST, SIT, and control). Thus, nothing was gained by incorporating the additional variables (i.e., videos, vignettes, dyad role plays) for the SIT group, and neither training group was superior to the control group. Following one in situ training session for those participants who did not use the skills during the first assessment, the cumulative mean scores for all groups increased such that the scores were significantly higher than the first assessment. However the scores were not significantly different between groups, demonstrating that in situ training was as effective with the control group as well as with both BST and SIT despite the control group not receiving training prior to the initial assessment.

The findings from this study suggest that in situ training alone may be an effective method for training children safety skills to prevent gun play. In previous research, in situ training was effective when implemented following two or more BST sessions (e.g., Himle et al., 2004; Kelso, et al., 2007; Miltenberger et al., 2004; 2005). The effectiveness of in situ training with the control group in the present study suggests that prior instructions and modeling with rehearsal of the skills (BST & SIT) may not be necessary for in situ training to produce acquisition and generalized use of the skills. Although in situ training can require much time and resources, the time and resources required to conduct BST or SIT sessions could be avoided by using in situ training alone.

Once each child achieved a score of 3 during a post training assessment, the child participated in a dyad assessment in which the child was challenged by a peer to play with the found gun. The safety skills used during individual in situ assessments generalized to dyad in situ assessments whether or not the participants had rehearsed scenarios in which a peer challenged them. No significant differences were found between BST and either IST or control groups in dyad assessments. Participants in BST and control groups were not trained to use the safety skills if they found a gun with a peer; however after successfully using the skills in individual assessments, most participants were able to perform the safety skills when challenged by a peer during their first dyad assessment. This finding is important, as children are as likely to find a gun with a similar age peer or sibling as alone. As this is the first study to demonstrate successful use of the skills when challenged by a peer, further research is needed to replicate this finding.

This study clearly demonstrated the ineffectiveness of simulated in situ training & BST while demonstrating the effectiveness of actual in situ training. Consistent with Himle et al. (2004a; 2004b), Milten-berger et al. (2004), and Kelso, Miltenberger, Waters, Helm, and Bagne, 2007) participants receiving BST (with or without SIT) performed no better than control participants. It was not until in situ training was provided that scores increased relative to the first assessment. Although the results add to the growing body of research supporting the necessity of in situ training to promote generalized use of the safety skills (e.g., Himle et al., 2004; Kelso et al., 2007; Miltenberger et al., 2004; 2005), it is not clear what factors contributed to the effectiveness of in situ training and the failure of simulated in situ training. Future research should continue to evaluate the variables that are necessary for effective training.

There were limitations to this study. The first is that the control group only served as a control group for the initial assessment. Following the first assessment and subsequent in situ training, this group could be considered an additional experimental group, which evaluated the effectiveness of in situ training alone. However, there were at least two benefits to using the control group in this manner. First, all participants were able to benefit from some type of safety skills training. Not training all participants may have been considered unethical. Second, the benefit of gathering information regarding in situ training alone seemed to outweigh negative aspect of not having a control group at all assessment points. In future research, it may be useful to incorporate a standard control group that receives no training throughout the experiment. In this way the effects of each subsequent intervention can be compared to no intervention and the effects of multiple assessments can be assessed as a possible confounding variable.

A second limitation is that training integrity data were not collected. It is noteworthy that two trainers typically led training sessions and each trainer had been through training on how to lead each training program. However, with the similarity of BST and SIT, it would have been beneficial to have data on the integrity of training to ensure that the training programs differed.

This study evaluated simulated in situ training as a means for increasing the efficiency of training and found that simulated in situ training was not effective. Future research should continue to evaluate more efficient means of training safety skills to children. Increasing efficiency may be accomplished by evaluating the use of a training package for parents to use with their own children (e.g., Gross, Miltenberger, Knudson, Bosch, & Brower-Breitwieser, 2007) or teachers and other community leaders to use with groups of children.

Acknowledgment 

Our thanks to Andrew Groninger, Patrick Fylling, Jessica Lemer, Spicer Latu, Shannon Morlock, Ashley Randall, Mark Goetz, and Sarah Heimerdinger for their assistance in data collection and training. Send correspondence to Raymond Miltenberger at [e-mail].

References

DiScala, C., & Sege, R. (2004). Outcomes in children and young adults who are hospitalized for firearms-related injuries. Pediatrics, 113, 1306-1312.

Eber, G. B., Annest, J. L., Mercy, J. A., & Ryan, G. W. (2004). Nonfatal and fatal firearm-related injuries among children aged 14 years and younger: United States, 1993-2000. Pediatrics, 113, 1686-1692.

Gatheridge, B. J., Miltenberger, R. G., Huneke, D. F., Satterlund, M. J., Mattern, A. R., Johnson, B. M., & Flessner, C. A. (2004). Comparison of two programs to teach firearm injury prevention skills to 6- and 7- year-old children. Pediatrics, 114, 294-299.

Gross, A., Miltenberger, R., Knudson, P., Bosch, A., & Brower-Breit-wieser, C. (2007). Preliminary evaluation of a parent training program to prevent gun play. Journal of Applied Behavior Analysis, 40, 691-695.

Grossman, D. C, Reay, D. T., & Baker, S. A. (1999). Self-inflicted and unintentional firearm injuries among children and adolescents; the source of the firearm. Archives of Pediatrics & Adolescent Medicine, 153, 875-882.

Himle, M. B., Miltenberger, R. G., Flessner, C, & Gatheridge, B. (2004). Teaching safety skills to children to prevent gun play. Journal of Applied Behavior Analysis, 37, 1-9.

Himle, M. B., Miltenberger, R. G., Gatheridge, B., & Flessner, C. (2004). An evaluation of two procedures for training skills to prevent gun play in children. Pediatrics, 113, 70-77.

Johnson, B. M., Miltenberger, R. G., Egemo-Helm, K., Jostad, C. J., Flessner, C., & Gatheridge, B. (2005). Evaluation of behavioral skills training for teaching abduction-prevention skills to young children. Journal of Applied Behavior Analysis, 38, 67-78.

Kellerman, A. L., & Reay, D. T. (1986). Protection or peril? An analysis of firearm-related deaths in the home. The New England Journal of Medicine, 314, 1557-1560.

Kelso, P., Miltenberger, R., Waters, M., Egemo-Helm, K., & Bagne, A. (2007). Teaching skills to second and third grade children to prevent gun play: A comparison of procedures. Education and Treatment of Children, 30, 29-48.

Miller, M., Azrael, D., & Hemenway, D. (2001). Firearm availability and unintentional firearm deaths. Accident Analysis and Prevention, 33, 477-484.

Miller, M., Azrael, D., Hemenway, D., & Vriniotis, M. (2005). Firearm storage practices and rates of unintentional firearm deaths in the United States. Accident Analysis and Prevention, 37, 661-667.

Milterberger, R.G., Flessner, C., Gatheridge, B., Johnson, B., Satterlund, M., & Egemo, K. (2004). Evaluation of behavioral skills training procedures to prevent gun play in children. Journal of Applied Behavior Analysis, 37, 513-516.

Miletenberger, R. G., Gatheridge, B. J., Satterlund, M., Egemo-Helm, K. R., Johnson, B. M., Jostad, C., Kelso, P., & Flessner, C. A. (2005). Teaching safety skills to prevent gun play: an evaluation of in situ training. Journal of Applied Behavior Analysis, 38, 395-398.

Miltenberger, R. G., & Gross, A. (in press). Teaching safety skills to children. In W. Fischer, C. Piazza, & H. Roane (Eds.), Handbook of applied behavior analysis. New York: Guilford.

National Rifle Association Safety and Education Division. (2001). Eddie eagle gun safety program. Fairfax, VA: National Rifle Association.

Ordog, G. J., Wasserberger, J., Schatz, I., Owens-Collins, D., English, K., Balasubramanian, S., et al. (1988). Gunshot wounds in children under 10 years of age: A new epidemic. American Journal of Diseases of Children, 142, 618-622.

Wintemute, G. J., Teret, S. P., Kraus, J. F., Wright, M. A., & Bradfield, G. (1987). When children shoot children: 88 unintended deaths in California. Journal of the American Medical Association, 257, 3107-3109.

Raymond Miltenberger

University of South Florida 

Amy Gross

Western Michigan University 

Peter Knudson

North Dakota State University 

Amanda Bosch

University of Florida 

Candice Jostad

Western Michigan University

Carrie Brower Breitwieser

Idaho State University 

Correspondence to Raymond G. Miltenberger, Ph.D., University of South Florida, 13301 Bruce B. Downs Blvd., MHC 2113A, Tampa, FL 33612; e-mail: [e-mail]