Methodological Constructs of Tracking and Retaining Highly Mobile Participants

This study systematically  reviews the methodological constructs  of longitudinal  research, and explores the effectiveness of traditional  as well as new and innovative strategies in tracking and retaining participants for longitudinal research. Specifically, this study explores the extent to which it is possible to track and engage highly mobile participants  in a long-term  follow-up study five years after the end of a community-based after-school arts program. Data for the present study are from the National Arts and Youth Demonstration Project (NAYDP). Challenges pertaining  to participant  tracking  and retention  are  discussed  from the perspectives of the research staff, as well as the youth involved in the recruitment  (tracking process); some of the lessons the investigators learned from the follow-up study are also discussed.

attrition, subject retention, tracking, after-school community-based programs, longitudinal research

Over the last decade, researchers have shown that after-school arts programs have a positive influence on the psychosocial functioning of youth (Wright, Offord, Duku, Rowe, Ellenbogen, 2006). Common variables, regardless of research rigor associated with these outcomes, include: stable and trained personnel, frequent assessment, and positive social norms with opportunities for youth to engage (Bodilly, Beckett, 2005). To better determine effective outcomes and determine which program effects endure, follow-up studies (e.g., longitudinal research) are necessary. Through longitudinal research, accurate outcome variables can be evidenced, enabling researchers to then develop adaptive prevention and/or intervention programs for populations in need, or at risk.

One of the problems with longitudinal follow-up studies however, is that of subject loss. Even well-designed studies will have some loss to follow-up (Hunt, White, 1998). Not all youths who enroll in these programs actively participate and sustain their motivation to stay; many of them drop out. Several terms are used to refer to this phenomenon, including attrition rates (Weisman, Gottfredson, 2001), retention rates (Armistead, Clark, Barber, Dorsey, Hughley, Favors, Wyckoff, 2004), participation rates (Simpkins, Ripke, Huston, Eccles, 2005), and attendance (Chaput, Little, Weiss, 2004). Attrition is an issue that needs to be discussed as subject loss is a threat to the validity of longitudinal designs (Navratil, Green, Loeber, Lahey, 1994). There is evidence to suggest that significant attrition in a study can result in non-random sampling at follow-up, which in turn can affect the means and variances of variables of interest (Goodman, Blum, 1996); this does not necessarily mean that underlying relationships between variables will be skewed, since that would be quite dependent on the research design and the statistical methods employed to examine said data.

Unfortunately, studies provide little detail on their follow-up methodology (Cotter, Burke, Stouthamer-Loeber, Loeber, 2005; Cotter, Burke, Loeber, Mutchka, 2005b; Dunn, Pirie, Oakes, 2004), or have not conducted follow-up data analysis at all (Durlak, Weissberg, Pachan, 2010). Even where methodology has been well defined, recorded retention methods are seemingly stronger for follow- up studies within a six-month, to two-year timeframe; there is less substantial research on participant retention methodology for longer term follow-up longitudinal research (Navratil et al., 1994).

This article will review participant retention strategies in longitudinal research and will also discuss strategies that have successfully overcome retention barriers. Second, this article will describe the retention procedures and/or strategies implemented in a Canadian longitudinal study, the National Arts and Youth Demonstration Project (NAYDP) five years after completion of the program, which contributed to positive outcomes on youths’ school experience, and hope for the future (Wright, Alaggia, Krygsman, 2014) (a follow-up study conducted from 2007-2011); this will be used to inform researchers involved in longitudinal studies on the methods utilized to maintain high retention rates with youth in a position of low income, aboriginal and multicultural communities (Wright et al, 2014).

Although longitudinal research is ideal for determining the most accurate outcomes from implemented programs, this type of research is not without flaws. Two of the most common issues with longitudinal research are: loss of study participants over time, and the inability to locate participants whose contact information has changed (Lyons, Carter, Carter, Rush, Stewart, Archbold, 2004).

It has also been documented that longitudinal studies often do not address the effort involved in working with participants who are difficult schedule (Navratil et al.,1994), nor have scholars paid sufficient attention in determining how various outcomes are linked to specific cultures and ethnicities (Fredricks, Simpkins, 2012). Acquiring and tracking individual demographic information (e.g, gender, race, age and income), as well as community demographic information (e.g., family, school and neighborhood characteristics) for after-school programs and program participants are necessary to determine appropriate program features that will suit local need(s) (Durlak et al., 2010).

Strategies for tracking subjects have traditionally included communication via telephone, mail, and public documents (Pirie, Thomson, Mann, Peterson, Murray, Flay, Best, 1989; Sullivan, Rumptz, Campbell, Eby, Davidson, 1996). Pirie and colleagues (1989) described the experiences of tracking subjects of two longitudinal smoking prevention programs in the United States. The follow- up was conducted with participants in the 11^th and 12^th grades, three years after the program had ended. Overall, 83% of youths were located via directory searches, long-distance operators, mailings, drivers’ license bureaus, and school records (Pirie et al., 1989).

Sullivan et al., (1996) examined retention strategies in a longitudinal community-based population of abused women in the United States. Using all of the above-mentioned strategies, retention rates varied from 94-97% at six-month intervals over 2 years of follow-up. However, had the researchers relied solely on traditional methods of telephone, mail and alternative contact information given, 64% would have remained at 6 months, 31% at 12 months and 22% after four years. In addition to collecting participants’ contact information, ‘Release of Information’ forms were signed for contact persons who may have information on where the participant was living if contact information had changed. Contact persons were family, friends, neighbors, coworkers, or individuals in community organizations. If these attempts to reach participants were unsuccessful, the recruiters moved to other network and community-oriented retention strategies with the available alternative contacts while maintaining confidentiality.

Given the former, evidence suggests that tracking and retaining participants can be costly in terms of staff time; however the cost of retaining “difficult to contact individuals” has been found to be worth the effort in terms of study investment and maintaining validity (Ahern, Le Brocque, 2005; Cotter et al., 2005; Navratil et al., 1994).

In keeping with the former tracking and retention strategies, literature also discusses the importance of researcher training in relation to successful participant tracking and retention over time. Specifically, Navratil et al. (1994) was one of the few studies that detailed the importance of this training in longitudinal studies and recommended several strategies: (a) Interviewers were hired based on their confidence to work with a variety of people, the ability to remain relaxed with others, as well as flexibility in their work hours; (b) Staff members were trained thoroughly in accordance with the interview protocol and pilot interviews were conducted; (c) In order to prevent subject loss, a variety of contact information was collected at the initial assessment including participants’ contact information, parental consent to contact the participant’s school in the case of address change and contact information of two contact people who would always know how to find the participant. The participants’ contact information and two contact people were updated at each subsequent assessment. Participants were also informed of what was expected of them during the course of the study, including the length of the study (Hunt, White, 1998; Navratil et al., 1994). After each assessment phase, the participants were thanked for their participation and provided with a note with their payment (Navratil et al., 1994). Hand-written letters and routine birthday cards were mailed to the participants. Persistence, patience, flexibility, and empathy on the part of the research staff were described as the key to successfully ensuring continued participation. Building rapport with the family was especially important for reluctant participants and having a consistent staff member recruiting participants each year facilitated the retention procedures.

In recent years, researchers have started to utilize various forms of social media as a means of tracking participants over time; albeit there are difficulties surrounding privacy and maintaining confidentiality. One example of this comes from Nwadiuko, Isbell, Zolotor, and Kotch, (2011), who contacted participants via Facebook and MySpace, having taken very specific precautions to maintain privacy in the event that they were contacting someone other than those they intended to reach. Through this method of contact, Nwadiuko et al were able to achieve a 20% response rate from participants over both sites. These participants, it should be noted, were not successfully reached by traditional mail prior to social media outreach attempts (2011).

Given that it is well understood that subject loss can bias potential results in longitudinal research, it has been recommended that check-ins occur with participants who are taking part in research over a longer period of time. Specifically, it has been noted in a one-month follow-up study, that if participants are contacted regularly between the release of an initial study and the follow-up portion of the research, rates of participant attrition are less than if these participants had not been contacted; this is especially true for participants who fall under the “hard-to-reach” category (David, Alati, Ware, Kinner, 2013). While these results were true for a study with a shorter follow-up window than the current research, it can be postulated that this method of maintaining regular contact during the interim period of release to follow-up would still be helpful to reduce attrition, even in longer-term follow-up studies; the frequency of the regular contact would just need to be modified to accommodate the follow-up time line.

Apart from the aforementioned studies detailing methods for participant retention, attrition rates in longitudinal studies, which explore youth problem behaviors, tend to vary greatly; research that has provided evaluations have often not controlled well for self-selection biases that could nine longitudinal studies and found retention rates varied from 30% to 80% with a mean of 62% for follow-up studies with six-month to two- year intervals. Moreover, the retention rate decreased to 53% when follow-up intervals were extended four to ten years. Maintaining regular contact with participants, ideally every six to twelve months, increases the chance of locating participants and remaining aware of any residential changes. Staff characteristics were also very important in minimizing subject loss, such as informative training, enthusiasm, building rapport with participants and families, flexibility in scheduling interviews, persistence in recruiting at each data collection, consistency of recruiters and responding promptly to problems or questions (Coen, Patrick, Shern, 1996; Marcellus, 2004; Navratil et al., 1994; Sullivan et al., 1996).

Given the limitations of existing research on the methodological constructs of tracking and retaining participants, and the lack of follow-up evaluations (Durlak, Mahoney, Bohnert, Parente, 2010), the present study was intended to explore the extent to which it is possible to track and engage youths in a long-term follow-up study (five years after the end of a community-based arts programs) and to examine the effectiveness of traditional tracking strategies as well as new innovative strategies more relevant to youth and community after-school programs. To do this, this study describes tracking and retention strategies utilized in the NAYDP, the implications from this study, and recommendations for future longitudinal research studies.

Data for the present study are from the National Arts and Youth Demonstration Project (NAYDP) (Wright et al., 2006). The NAYDP and its predictors of attendance have thoroughly been described elsewhere (John, Wright, Sheel, Duku, 2007; Wright et al., 2006, Wright et al., 2014). This study explored the extent to which community-based organizations could engage and retain children from low-income and multicultural communities, in artistic endeavors (combination of theater, visual, and media arts). It also explored the relationship between involvement in arts programs and positive outcomes regarding child and youth psychosocial functioning (e.g., bonding with pro-social peers and adults). A total of five community-based organizations were selected as sites for the NAYDP. The five sites were (a) located in: (Montreal, QC; Toronto, ON; Winnipeg, MB; Vancouver, BC and Tillsonburg, ON) (b) targeted arts programs towards youths 9-15 years old, (c) located in low-income communities, and (d) culturally reflective and regionally diverse. The program ran from the end of September 2002 to the end of June 2003, and data were collected at the end of each of the three terms: December 2002, March 2003, and June 2003. A six-month follow-up assessment was carried out in December 2003. To facilitate the children’s ongoing contact with the arts organizations and their peers, and to maintain program effects over time, two booster sessions were conducted in the fall of 2003. The attrition rate was 37% for the 183 youth who participated at some point in the program, including late joiners. With respect to duration of attendance, 84% of participants completed at least one term, and 72% of those capable of completing at least two terms (i.e., excluding Term 3 joiners) did so. The two reasons most often cited for dropping out were loss of interest in the program (26%) and moving from the area (18%). Attendance rate was over 80% for all terms.

The study being explored is the National Arts & Youth Demonstration Project (NAYDP; 2001-2004), one of the first well-controlled quasi-experimental studies to evaluate the effects of a community-based after-school arts programs for youths residing in low-income, multicultural, and Aboriginal communities in five sites across Canada (Wright et al., 2006). The present study is a follow- up of the NAYDP participants four and a half years after the end of the after-school arts programs. Self-report questionnaires on psychosocial functioning were administered to all youths. In addition, forty youths were randomly selected to be interviewed regarding program benefits and key components (Wright et al., 2014).

The sample consisted of 183 participants, 64.5% girls and 35.5 % boys. Of the 183 youths, 59% were White, 26% Aboriginal, 11% Black, 6% Asian, 5% Latino, and 4% other, including youths who identified themselves with more than one ethnic group. The majority of youths were from low- income families where 40% earned an income below $20,000/year and 32% had received welfare in the last 12 months. The youths participated in a nine-month structured arts program delivered by community-based organizations in 2002-2003. The youths were 9 to 15 years of age when the program began, and were 14 to 20 years of age at the time of the follow-up.

Researcher Training. It was critical to train the research assistants to optimize youth recruitment and sustain their participation in the data collection process. Thus, the principal investigators conducted cross-site training prior to the recruitment of the youths. The research assistants who had collected data, or were involved in delivering the program in the original study, attended a full-day training session at McGill University. The purpose of the training session was to: (a) outline project objectives and discuss the roles and responsibilities of the project team; (b) detail communication strategies, data collection procedures and timelines with the investigative team; (c) develop an in- depth work plan for youth recruitment at each site. These cross-site training procedures ensured that optimal recruitment strategies would be employed and monitored by the investigative team.

The research protocol/policy manual proved to be an invaluable tool for training the research assistants; it focused on developmental needs of adolescents, social, economic, emotional, behavioral and cultural differences in participants. The written material allowed for context-specific question and discussion sessions, as well as clear and consistent policies for the emotional and behavioral management of youths. An unexpected function of the manual was that it helped alleviate staff anxiety prior to the recruitment and data collection process. Most importantly, it defined the communication process and accountability procedures between the investigative team and the staff (Wright, John, Sheel, 2007). Specifically, the research assistants were taught the importance of a high response rate in addition to the importance of adhering to the data collection protocol and documentation process while conducting the study. Data Collection. The data collection protocol included the following:

1. An introductory letter: The introductory letter was addressed to parents (for youths under the age of 18) and was sent to the last known address one week prior to contacting the families. Letters were personalized for each of the five sites, with the community organizations logo and program photos included in order to trigger memories of the program. The purpose of the letter was to prepare the participants for the research assistants’ call.

2. Tracking Participants: The participants were contacted by calling the last known number; searching for the parents through telephone directories; and contacting all other contacts in the research database (relatives, friends, employers). The process was then documented on a “Participant Tracking Sheet”.

3. The Interview: The research assistants contacted the parents and youths to invite them to participate in the follow-up study. If they agreed to participate, the research assistants scheduled a home visit for an interview. Interviews were conducted at a community centre or local coffee shop in the event that parents were uncomfortable with a home visit. Informed consent was obtained from both the parent and the youth prior to the questionnaire being administered.

4. Incentives and thank you note: Youths were offered $20 for participating in the study.

Following collection of the data, the research assistants were asked to respond to the following open-ended questions: a) how did you go about tracking the hard-to-reach responses/reactions from parents? d) What were the responses/reactions from the youths? e) What strategies were helpful in convincing reluctant parents to participate? f) What strategies were helpful in convincing reluctant youths to participate? g) In your opinion, what was the most challenging aspect of this data collection process? h) In your opinion, what was the most surprising aspect of this data collection process? i) What advice would you give another research assistant about to conduct a similar data collection procedure?

Overall, 91 of the original 183 participants completed the self-report questionnaire, thus, a response rate of 50% was achieved. However, the response rate increased to 79% when considering only the 115 youths who had completed the original arts program. As depicted in Table 1, 31% of the sample retained previous contact information and 69% of the sample required tracking strategies. Of the 69% of the sample which required tracking strategies, making successful contact occurred as follows: one additional source of information allowed 7% of the sample to be located, using two additional sources of information allowed 15% of the sample to be located, and using three sources of information allowed an additional 2% of the sample to be located (see Table 1.). As depicted in Table 1, sites one, two and three had the highest rates of participants where the original contact information remained the same. As such, these sites required fewer additional sources of contact information. The opposite was true for sites four and five.

As a whole, 45% of the sample could not be located at the four and a half-year follow-up time period. Two or more research assistants independently identified the following as effective strategies: persistence in calling to schedule an appropriate time; arranging contact and data collection through a community organization; and the $20 youth incentive.

Overall, the most effective strategy for tracking hard-to-reach youths was contacting relatives and friends through the original contact information database, as most people were willing to give a forwarding telephone number and address. Another successful strategy was working with the community center that offered the original arts programs. In one site in particular, the community center not only offered arts program but also offered a range of social services; therefore, families were most likely in consistent contact with the center. The research assistant and the family met at the center, as the community center was considered a safe and trusted place by the participants. At another site, most families were difficult to contact because many of them did not own telephones. However, participants were familiar with a research assistant from the community and in one case, the research assistant was approached in a public area and was therefore able to establish contact with the participant. Last, many youths had the same network of friends or went to the same school as participants from the program and these friends offered to give the contact information of the research assistant.

One strategy, which did not prove as effective for this population of participants was the use of social networking websites such as “Facebook”, “My Space”, and “Hi5”. Although many youths were using these websites, they often used aliases. In addition, although local directories were useful in finding a small number of parents, many families solely utilized unlisted cell phone numbers. This finding is comparable to the research of Bolanos, Herbeck, Christou, Lovinger, Pham, Raihan, Rodriguez, Sheaff, Bercht, (2012), whereby Facebook was used to re-contact study participants after an 8-year period of time; less than 50 of the remaining 551 participants were reached via this social networking site, and of this approximate 50 participants, only 11 completed the follow-up. This is a very low follow-up rate; in fact less than one percent successfully completed the follow-up study component. One recommendation from the work of Bolanos et al. (2012) that should be considered, is that social media is yet another means to track and reach participants; it has been recommended that prior to the initial study completion, researchers should access participants’ web address for whichever social media site they are using (e.g., a Facebook web address); these addresses are less likely to change if participants change their “username” or use an alias on these types of sites. In addition, it is possible that more frequent contact via social media between release from the NAYDP and the follow-up 4 ½ years later, might have increased this success rate.

Overall, the youths appeared to be happy to participate in the research study. This may have been due to previous acquaintance with the research assistants, and subsequently participants were happy to hear from them. At one site in particular, the research assistant was also a trained youth counselor who had a great rapport with the arts program participants. Five years later, she was greeted with comments like: “Sure I’ll participate in the study. I can’t wait to see you and tell you everything I’ve been doing”. The research assistant was surprised at the extent to which the youths and parents would ask her for advice and referrals on other social issues once they had completed the questionnaire. Additionally, research assistants were successful in engaging families with an honest and forthright manner, explaining their connection to the program and to known community leaders. Research assistants also mentioned patience, flexibility, and persistence as key attitudes for collecting data.

Last, monetary incentives were identified by most research assistants as being an effective form of encouragement to participate. A research assistant stated, “In some cases the money was a big motivator but even with some who clearly came for the money, there was a sense of nostalgia.” However, the monetary incentive was not seen as important for sites that consisted of older participants (18 to 20 years of age). Many youths even made a point of mentioning that they “would have done it without the 20 bucks”.

Tracking the youths may have been easier if the long-term follow-up study had been planned before the arts program had ended. This would have allowed the researchers to update the youth’s contact information on a yearly basis, or perhaps every six months, as evidenced in previous research discussing longitudinal designs. Last, as with the original study, email addresses were not collected, nor were social media web addresses; these would have been very useful for tracking parents and even youths who were using these means of communication. Further, obtaining “Release of Information” forms for schools to provide current addresses may have been helpful had this been anticipated at the beginning of the study.

Minimizing attrition is vital to the success of longitudinal research. In a review of attrition research, Marcellus (2004) describes attrition reduction in terms of an ecological model including: participant strategies, researcher strategies, strategies related to the study and strategies related to the environment; these are all interrelated with the participant at the center. In other words, to best prevent and plan for attrition, creating a design that participants can complete, is of importance.

As is the case in the present study, quasi-experimental evaluations were conducted without planning for long-term follow-up; this made tracking participants challenging. This matter is further complicated when there is a significant lapse, as in this case almost five years, after the end of the intervention. The longer the lapse, the higher the loss in tracking will be. For example, the present research examining data from the NAYDP at a 5-year follow-up was only able to maintain contact with 31% of participants with the original contact information that was gathered. Had this long-term follow-up been planned prior to the completion of the NAYDP, this study could have employed some of the other methods of retention and contact strategies to increase the number of participants in the follow-up.

Future research will need to differentiate between outcomes from studies with specific demographic information, socioeconomic status, or particular sample characteristics (Ahern, Le Brocque, 2005). For example, as previously mentioned, Pirie and colleagues (1989) tracked 83% of youths in a follow-up study 3 years post-program. The youths were in grades 11 and 12 at time of follow-up. Such a high tracking rate is not surprising as youths still in school are much easier to track. Conversely, in a cross-sectional population survey, like the Ontario Child Health Study, there was an attrition rate of 30% in the first follow-up with a major increase in subsequent follow-ups. (Boyle, Offord, Racine, Catliin, 1991)

The participants in the present study fit the criteria of the most difficult to track in a follow- up study. They were multicultural and aboriginal youths living in low-income neighborhoods; most importantly, they were also highly mobile. Epstein and Botvin (2002) noted that the most difficult participants to follow are those who are highly mobile and the follow-up period is long. However, follow-up with such populations may still be conducted with a respectable response rate. As demonstrated, engaging participants in the study may be based on a research assistants’ relationship with the participants’ families, friendliness, persistence, and flexibility; more specifically, through utilizing a research policy that defined the communication process and accountability procedures between the investigative team and the staff (Wright et al., 2007), the research assistants were taught the importance of a high response rate in addition to the importance of adhering to the data collection protocol and documentation process while conducting the study. Therefore, the qualities mentioned, along with further characteristics of a non-threatening program may have assisted in tracking youths who participated in the original arts program. The former discussion of the present research is particularly important and relevant to the field of social work, as some of the methods used in this study espouse one of the core values of the social work profession – meeting others where they are at. Specifically, this paper highlights the importance of planning for research, training researchers accordingly, and expecting some attrition, but this research also lends evidence to the notion of creativity and occasionally working outside the realm of typical methodological constructs.

Cette étude examine l’efficacité  des méthodes traditionnelles en matière d’études longitudinales ainsi que des stratégies innovantes qui permettent  de suivre et de  faciliter  l’engagement  de participants  très mobiles cinq ans après la fin de leur participation à un programme artistique extrascolaire communautaire. Plus précisément, cet article explore, à partir du point de vue du personnel de recherche, ainsi que de celui des jeunes impliqués dans le recrutement, les difficultés relatives au suivi des participants.  Certaines leçons apprises par les enquêteurs de l’étude sont également discutées. Les données proviennent du Programme national de démonstration sur les arts et les jeunes.

Recherche longitudinale,  rétention des participants,  programmes communautaires,  programme national de démonstration sur les arts et les jeunes