BMJ Open
BMJ Publishing Group
Investigating the critical elements and psychosocial outcomes of Youth Flexible Assertive Community Treatment: a study protocol for an observational prospective cohort study

DOI: 10.1136/bmjopen-2019-035146, Volume: 10, Issue: 4, Pages: 0-0
Article Type: research-article, Article History


When adolescents experience complex psychiatric and social problems, numerous healthcare services usually become involved. In these cases, fragmentation of care services is a risk that often results in both ineffective care and in patients disengaging from care services. To address these issues, Youth Flexible Assertive Community Treatment (Youth Flexible ACT) was developed in the Netherlands. This client-centred service delivery model aims to tackle the fragmented care system by providing psychiatric treatment and support in a flexible and integrated manner. While Youth Flexible ACT is gaining in popularity, the effectiveness of the care model remains largely unexamined.

Methods and analysis

Here, we present an observational prospective cohort (2017–2021) in which a broad range of treatment outcomes will be monitored. The primary aim of the study is to examine change in treatment outcomes over the course of the Flexible ACT care. The secondary aim is to examine the association between (elements of) Youth Flexible ACT model fidelity and treatment outcomes. An estimated total number of 200 adolescents who receive care from one of the 16 participating Youth Flexible ACT teams will be included in the study. Participants will be asked to complete assessments at four time points in 6-month intervals, resulting in a study duration of 18 months. Latent growth curve analysis will be conducted to examine change in psychosocial functioning over time and its relation to model fidelity.

Ethics and dissemination

This study received ethical approval from Trimbos Ethics Committee (201607_75-FACT2). This approval applies for all participating institutions. The results of the study will be reported in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology statement. Results will be disseminated via peer-reviewed academic journals and presentations at conferences. In addition, results will be made available for participating sites, funders and researchers.


Strengths and limitations of this study

  • This is the first multicentre study in which the effectiveness of Youth Flexible Assertive Community Treatment (Youth Flexible ACT) will be investigated.
  • This study will provide a complete overview of Youth Flexible ACT in which model fidelity and both psychiatric and social functioning will be assessed.
  • A strength of the study is its observational and naturalistic character which improves external validity.
  • We examine changes in treatment outcomes in a longitudinal study design, with follow-up assessments up to 18 months.
  • Primary limitation: as no variables are directly manipulated, causal inference is impeded.


About 5% of Dutch children and adolescents have a mental disorder that leads to functional impairment.1 2 Approximately 20%–30% of them have to deal with severe mental health issues that require a more intensive and integrated form of care.3 In addition to psychiatric problems, these adolescents experience various difficulties in everyday life, including problems with education, employment, peer relationships, family, housing, finances, health, substance abuse and issues with the criminal justice system. These difficulties hinder their development and limit their ability to function well in society.4 Adolescents and their relatives are often required to act as the central communicators and coordinators of the care they receive. This active role requires motivation and a fairly high level of knowledge about the healthcare system, which is often too challenging for adolescents with complex care needs. In addition, healthcare providers themselves often struggle to manage multiple healthcare issues of their patients due to limited communication and coordination between healthcare providers.5 6 As a result, treatment disengagement and dropout are common.7 8 Together, this calls for a model of care that provides longitudinal, comprehensive, flexible and assertive care. Youth Flexible Assertive Community Treatment (Youth Flexible ACT) is designed to meet those demands.

Youth Flexible ACT is a client-centred service delivery model for community mental healthcare that provides assertive outreach, psychiatric treatment and support with daily living, adapted to the individual needs. The primary focus is to set up a collaborative effort with adolescents, families and their (in)formal networks while working on shared goals aimed at improving their participation in the community and enhancing their quality of life.4 Youth Flexible ACT consists of a multidisciplinary team of professionals who deliver a complete range of services on a continuum of care.

Previous research into (Flexible) ACT and Youth (Flexible) ACT

Flexible ACT is a Dutch adaptation and elaboration of Assertive Community Treatment (ACT), which originated in the USA in the 1970s.9 10 ACT is a well-known approach for individuals with severe mental illness that has been studied extensively, spread widely throughout the world and became embedded in the Dutch Multidisciplinary Schizophrenia Guideline.11–13 Studies summarised in a Cochrane Database Systematic Review provide strong evidence that ACT can increase engagement with treatment, reduce hospitalisation and lead to improvements in social domains, including stable housing, employment and patient satisfaction compared with the care as usual.13 However, studies conducted after the initial Cochrane Review in 1998, mostly outside the USA, have shown mixed results.14 15 For example, the German ACCESS study revealed that ACT was associated with symptomatic and functional improvements and better service engagement in patients with schizophrenia spectrum disorders compared with standard care.16 A recent Chinese study showed positive results in terms of hospital readmission, symptoms and relapse, employment, social and occupational functioning, and quality of life of caregivers.17 However, the British Randomised Evaluation of Assertive Community Treatment (REACT) study found no effects on clinical and social outcomes and hospitalisation.18 Additionally, a Dutch study did not find a difference between ACT and standard care in reducing admission days and clinical outcomes.19 These inconsistent findings could be due to low ACT model fidelity or insufficient contrast between experimental and control conditions, since treatment as usual gradually incorporates elements of ACT.14 15 20

While ACT is indicated for the most vulnerable patients with severe mental illness (predominantly psychotic disorders) who have the greatest needs for care, Flexible ACT delivers care for a broader group of patients with severe mental illness.9 10 For stable patients, Flexible ACT provides multidisciplinary treatment and support through individual case management. For unstable patients, it provides intensive care offered by the same team.21 Flexible ACT allows for flexible delivery of different modes of care according to the stability of the patient, in turn enhancing continuity of care.10 The Flexible ACT model was developed in the Netherlands in 2003. Over the last 10 years, the model has been widely implemented in the Dutch mental healthcare system (roughly 300 teams21). Lately, the Flexible ACT model has gained considerable interest in England, Canada and Scandinavia.21–23 However, despite the enthusiasm of service providers, the evidence base for the effectiveness of the adult Flexible ACT model remains sparse.24–26 Preliminary results have shown increased symptomatic remission of psychotic symptoms in patients with severe mental illness compared with controls receiving standard treatment,27 higher levels of psychosocial functioning,28 fewer hospital admissions and reduction of inpatient bed use,23 29 and increased compliance with treatment, decrease in unmet needs and improved quality of life.30 Flexible ACT has been shown to be more cost-effective compared with assertive outreach teams in England due to reductions in bed use, face-to-face contacts and changes in staffing.23 Outcomes of Youth Flexible ACT, which was introduced in the Netherlands in 2011, are reported in two reports published in the Dutch literature. Both studies were uncontrolled pre-post studies, and they showed preliminary positive outcomes in that patients’ behavioural problems, problems with family life, hallucinations and delusions,8 attention problems, emotional symptoms, self-injury and peer problems improved.31 However, no improvement was found in the quality of life of the adolescents.31

The broader domain of Youth ACT is also limited to pre-post studies with small samples, which have reported similar positive outcomes, such as improvements in psychiatric condition,32–34 improved global functioning and increased life skills,35 and decreased number of days in hospital.35–37 Moreover, a recent Swiss study showed that Youth ACT results in improved daily functioning and clinical benefits.38

In the field of assertive outreach for adolescents, the largest controlled studies involved evaluations of early psychosis programmes. The Danish OPUS study demonstrated that patients receiving an assertive intervention for 2 years had fewer psychotic symptoms and decreased substance use, increased adherence to medication and increased treatment satisfaction.39 Furthermore, a randomised controlled trial conducted in England indicated that (a variant of) ACT was superior to standard care for maintaining contact with professionals and reducing readmissions.40 Finally, a Dutch quasiexperimental study showed positive effects of ACT on measures of psychopathology, psychosocial functioning and quality of life.41 Altogether, research of integrated outreach models for youth and adolescents supports the effectiveness of the Youth Flexible ACT model.

Research into (Flexible) ACT model fidelity

Studies on the effectiveness of (Flexible) ACT raised the issue of identifying essential elements of the model and investigated model fidelity, which reflects the degree to which different elements of the model are implemented in full accordance with the model.42 Studies have pointed out that higher ACT model fidelity is associated with improved outcomes for patients, such as level of daily functioning and less homeless days.43–46 Specifically, the typical team structure of ACT with shared case loads and daily team meetings has been found to be associated with better daily functioning45 and decreased hospitalisation rates.47 Furthermore, a positive association was found between the presence of consumer-providers, improvements in daily functioning and the number of homeless days.48 Nevertheless, no studies have examined critical elements related to effectiveness of Youth Flexible ACT.

Purpose of the present study

Although the Youth Flexible ACT model has been well received by mental health professionals, its popularity has been increasing and national Flexible ACT and (inter)national ACT studies have provided promising results, little evidence exists to support this particular model. Much research has focused only on adults, and the results have yielded coarse outcomes based merely on small study populations. Additionally, as the healthcare sector has been deinstitutionalised over the years, it is necessary to study Youth Flexible ACT in the current healthcare landscape.

The present study is an observational longitudinal study in which a broad range of psychosocial treatment outcomes will be monitored over time. The general objective of the present study is to give insight into outcomes of Youth Flexible ACT while exploring the Youth Flexible ACT population characteristics, content and process of care. The study addresses three research questions: (1) improvement in treatment outcomes over the course of the Flexible ACT care, (2) associations between Youth Flexible ACT model fidelity and treatment outcomes, and (3) associations between specific critical elements of the Youth Flexible ACT model and treatment outcomes. This study is of an explorative nature to the extent that no similar study has been conducted on the youth population. However, reasonable expectations based on the adult literature can be generated as follows: (1) psychosocial problems of adolescents will decrease when treated by a Youth Flexible ACT team and (2) high model fidelity will be associated with a decrease in psychosocial problems.

Methods and analysis


The Youth Flexible ACT Study is an 18-month observational prospective cohort study that examines change in treatment outcomes over the course of Youth Flexible ACT care and examines the association between (elements of) Youth Flexible ACT model fidelity and treatment outcomes. A total of 16 (non-specific) Youth Flexible ACT teams across seven mental healthcare institutes in the Netherlands participate in the study. An estimated number of 200 adolescents receiving care from a Youth Flexible ACT team will be included in the study. Participants and their mental health workers will be asked to complete assessments at four time points in 6-month intervals. Also parents/carers were asked to participate. The data collection started in 2017 and is scheduled to conclude in February 2021. The results of the study will be reported in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement.49

Patient involvement

This study is designed in collaboration with mental health workers. Before the start of the data collection, we have received input from all participating Youth Flexible ACT teams, a peer support worker and two Youth Flexible ACT patients. They have provided input about the recruitment process, information letter and (duration of) assessment battery. Results of the Youth Flexible ACT Study will be disseminated via scientific journals, presentations at conferences and will be made available at participating sites.

Study sample

The study sample comprises adolescents aged 12–24 years who receive Youth Flexible ACT care at one of the participating mental healthcare organisations throughout the Netherlands: Accare, GGZ Noord-Holland-Noord, GGZ Oost Brabant, Kenter Jeugdhulp, Lucertis, Mondriaan-Gastenhof and Triversum.

Participants receiving Youth Flexible ACT care will be included in the study if they meet the following research inclusion criteria:

  • Participants must be 12–24 years of age.
  • Participants must have sufficient knowledge of the Dutch language both spoken and written.
  • Participants and their parent/caregiver must provide a written informed consent.

Study population

Youth Flexible ACT provides treatment to young people with complex and severe mental health problems, who have difficulty engaging in regular mental healthcare. Practical experiences and literature indicate that these young people have a hard time accessing and remaining in regular outpatient mental healthcare.4 33 38 Several reasons for treatment disengagement in adolescents have been suggested, such as fragmented healthcare system,50 treatment discontinuity51 52 and difficulty to trust services.53 According the Youth Flexible ACT model description4 young people are eligible for Youth Flexible ACT care if:

  • They are diagnosed with a mental health disorder (or presumptive diagnosis).
  • They experience difficulties in multiple areas of daily life (eg, problems with education, employment, peer relationships, housing, finances, health, substance abuse and issues with the criminal justice system).
  • They face family system problems and/or parenting issues.
  • They live in the district of the Youth Flexible ACT team.

The pathway to Youth Flexible ACT care is straightforward and direct. Anyone (eg, patients’ parent, care workers) in the Netherlands can directly contact a Youth Flexible ACT team to suggest a potential referral when he/she thinks a patient is eligible. An intake coordinator of the Youth Flexible ACT team then determines if the inclusion criteria outlined above are met. If so, a referral from the general practitioner is requested before care can start.

Youth Flexible ACT

Youth Flexible ACT is a community-based mental health service in which integrated teams provide long-term assertive outreach care consisting of both treatment for psychiatric symptoms and practical assistance with daily living needs, rehabilitation and recovery support. Youth Flexible ACT teams consist of various professional disciplines, including psychiatrists, nurses, psychologists, employment specialists, psychiatric nurses, addiction specialists, peer support workers, social workers, and family and systemic therapists. Youth Flexible ACT encompasses a multiagency approach that coordinates collaboration with professionals from other services. Team members visit the patients at their home or at other preferred locations and provide assertive care when necessary. Time will be devoted to build and maintain trust between mental health workers and patients and to motivate patients to receive treatment and support. Mental health workers have a small case load (1:15) and deliver two modes of care: individual case management and intensive team care. Patients receiving individual care have a case manager and a head practitioner (psychiatrist, healthcare or clinical psychologist). Other team members can be added, as needed, for specific elements of treatment or support. A patient who is in need of extra care will receive intensive care from several team members. These patients are listed on a digital board, and the team discusses them every day to decide which form of care should be provided and by which team members. When the crisis or the need for intensive care is over, individual case management is resumed. Youth Flexible ACT, in which the family system plays a major part, has additional features beyond those that are part of Flexible ACT for adults. In particular, it is important to support the following four developmental tasks4 8: shaping changing relationships within the family (moving from dependence to autonomy), stimulating contact with peers (peers become more important as reference group as the influence of parents decreases), participating in education or work and filling leisure time. Furthermore, the possibilities for personal growth and the utilisation and cultivation of personal strength are emphasised instead of mental health disorder symptoms. Youth Flexible ACT is primarily a service delivery model. It describes the organisation of care for adolescents with complex care needs. The Flexible ACT model does not dictate the specific content of a treatment plan, although the use of evidence-based practices is advised. The degree to which teams implement these guidelines determines the level of model fidelity. With high Flexible ACT model fidelity, a complete multidisciplinary team provides the desirable treatment and support according to the guidelines. A more detailed portrayal of (Youth) Flexible ACT is outlined in the (Youth) Flexible ACT model description.4 21 54

Recruitment and assessments

During the intake process, team members of Youth Flexible ACT teams will ask eligible adolescents to participate in the study. After signing informed consent, participants will be asked to complete a baseline measurement within a 12-week margin. Participants will then be monitored every 6 months with questionnaires, up to four measurements. These questionnaires will be completed during a regular appointment by a familiar mental health worker or participants have the option to complete the questionnaires independently in their own time. It will take approximately 20–30 min for adolescents to complete the assessment battery. Adolescent participants receive a remuneration of €10. Both paper and online versions are available. Online versions are preferred to minimise missing data. Researchers are in close contact with mental health workers and ensure that participants complete the questionnaires timely. An online Dutch data system will be used to collect the data. Confidentiality of the data is guaranteed through a login procedure and each institution has its own digital environment. Participants who finish their treatment within 1.5 years will, with the adolescents’ permission, receive a link to the remaining questionnaires by email.


To increase the reliability of assessments, all mental health workers involved in data collection were trained in administering the questionnaires before baseline assessments. In particular, mental health workers received a Health of the National Outcome Scales for Children and Adolescents (HoNOSCA) training based on the official training.55 The HoNOSCA56 is a global scale measuring daily functioning and mental health symptoms. To examine the inter-rater agreement of the HoNOSCA, original case vignettes were used. The training consisted of a HoNOSCA information lesson, completing a training vignette and discussion. Subsequently, 82 mental health workers of 13 participating teams completed the actual vignette of which scores were analysed.

Intraclass correlations (ICC) were calculated as a measure of agreement between raters of a single vignette based on absolute agreement using two-way mixed-effects model. Statistical analyses were performed using IBM SPSS Statistics V.22.0. The resulting ICC between raters ranged from moderate to good agreement: ICC average measures=0.99 (95% CI 0.98 to 1.00) and ICC single measures=0.57 (95% CI 0.40 to 0.78). The findings indicate a moderate degree of agreement between ratings within an item and between items. Since a single vignette was used, the results reflect the agreement between raters and do not reveal the reliability of scale items.

Study outcome measures

Table 1 displays an overview of outcome measures that will be reported by clinicians, patients and parents/carers at each time point. The employed set of questionnaires together reflect the multiple life domains in which Youth Flexible ACT operates. The combination of questionnaires assesses general psychological functioning, specific diagnostic characteristics and daily functioning of the participants.

Table 1
Overview of outcome assessments
Variable Instrument Time of assessment (months)
Clinician reported outcomes
 Daily functioning HoNOSCA T0, T6, T12, T18
 Sociodemographics Questions concerning sociodemographics T0
 Content of care Seven questions concerning content of care T6, T12, T18
Patient reported outcomes
 Psychosocial well-being SDQ T0, T6, T12, T18
 Health-related quality of life Kidscreen-10 + additional questions T0, T6, T12, T18
 Depressive symptoms CDI-2 T0, T6, T12, T18
 Social support Scale ‘social support and peers’ from the Kidscreen-52 T0, T6, T12, T18
 Empowerment Subscale ‘interactional empowerment’ from the questionnaire EMPO 3.1 T0, T6, T12, T18
 Psychosis risk screening PQ-16 T0, T6, T12, T18
 Treatment satisfaction Four brief questions based on the Jeugdthermometer T0, T6, T12, T18
 Care utilisation One question concerning care utilisation T0
 Care utilisation and coordination Two questions concerning care utilisation and coordination T6, T12, T18
Parent reported outcomes
 Psychosocial well-being—child SDQ-P T0, T6, T12, T18
 Health-related quality of life—child Kidscreen-10 parent version T0, T6, T12, T18
 Psychological distress MHI-5 T0, T6, T12, T18
 Parenting stress PSQ-S T0, T6, T12, T18
 Treatment satisfaction Four questions based on the Jeugdthermometer parent version T0, T6, T12, T18

CDI-2, Child Depression Inventory-2; EMPO, Dutch Empowerment Questionnaire; HoNOSCA, Health of the National Outcome Scales for Children and Adolescents; MHI-5, Mental Health Inventory; PQ-16, Prodromal Questionnaire; PSQ-S, Short version of the Parenting Stress Questionnaire; SDQ, Strengths and Difficulties Questionnaire; SDQ-P, Strengths and Difficulties Questionnaire for Parents.

Clinician reported outcomes

Daily functioning

The Dutch version of the HoNOSCA56 will be used to measure mental health symptoms and daily functioning. Items are rated on a 5-point scale indicating the severity of problems ranging from no problem (0) to severe problem (4). Studies demonstrated sufficient reliability and validity for use in clinical samples.56–58

Sociodemographics and general case load information

Mental health workers provide information about the patients’ level of completed education, referrer, admission duration and whether IQ tests have been conducted. Mental healthcare workers also complete questions about the size and composition of the case load as part of the audit procedure.

Content of care

Content of care encompasses seven multiple-choice questions that provides insight into type of treatment and support, frequency of visits and frequency of provided intensive ‘ACT’ care in the past 6 months.

Patient reported outcomes

Psychosocial well-being

The Strengths and Difficulties Questionnaire (SDQ) is a brief screening questionnaire for children and adolescents which screens for psychosocial well-being.59 60 It includes 20 questions measuring the subscales emotional, conduct, and hyperactivity/inattention symptoms and peer problems. The subscale prosocial behaviour is excluded because scores of this subscale are not necessary to compute a total difficulties score. Each item is scored on a 3-point scale (0=not true, 1=somewhat true and 2=certainly true). The subscales are ranging from 0 to 10 for each scale and are added together to generate a total difficulties score, ranging from 0 to 40. Scores above the cut-off of 16 (>90th percentile) are considered ‘abnormal’.59 61 In addition, if an adolescent experiences difficulty, the impact scale can be used to indicate the extent to which any problems interfere with daily functioning. The five items are scored on a 3-point scale (0=not at all/only a little, 1=quite a lot, 2=a great deal) and can be added to compute an impact score that ranges from 0 to 10. Scores above the cut-off of 2 are considered ‘abnormal’.59 61 The SDQ was found to have sufficient reliability and validity for assessment in clinical samples.59 60 62–64

Health-related quality of life

The Kidscreen-10 is a 10-item questionnaire measuring health-related quality of life.65 66 Items are rated on a 5-point Likert scale ranging from 0 (never) to 5 (always). A total score can be generated by summing the 10 items. Total scores will be converted into Rasch scores67 and translated into T values (M=50; SD=10), with higher values indicating higher health-related quality of life. In addition to these 10 brief questions, the adolescents will be asked to provide additional information concerning important areas of daily living needs, such as finance, education or housing. Research has shown adequate psychometric properties for the Kidscreen-10.66

Depressive symptoms

Depressive symptoms will be measured using the Dutch version of the Child Depression Inventory-2 (CDI-2).68 69 The CDI-2 consists of 28 items measured on a 3-point scale ranging from 0 (depressive symptom is absent) to 2 (depressive symptom is always present). Sum scores can be computed by adding together scores of all 28 items. A higher total score means more depressive symptoms. A total score of 14 or higher indicates clinical levels of depressive symptoms.68 The internal consistency and validity of the CDI-2 have shown to be good.68

Social support

To measure social support, the scale ‘Social support and peers’ from the Kidscreen-5267 will be used. This scale examines the quality of the social interaction between adolescents using six items, for example, ‘did you have fun with friends?’ The items are measured on a 5-point Likert scale ranging from 1 (never) to 5 (always). A total subscale score can be computed by adding the six items. The total score will then be converted into Rasch scores and translated into T values (M=50; SD=10), with higher values indicating higher quality of social interaction with peers.67 We added two additional questions about satisfaction with social support. The Kidscreen-52 has shown acceptable levels of reliability and validity.70 71


Empowerment will be measured with the subscale ‘interactional empowerment’ from the questionnaire Dutch Empowerment Questionnaire (EMPO 3.1),72 73 which assesses the willingness to change undesired situations, to look for solutions, to take control and to know how to access resources.73 The subscale comprises six questions measured on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). A total score can be computed by summing the six items and can then be translated into T values, with higher values indicating higher level of empowerment. According to the authors of the EMPO 3.172 the questionnaire shows good internal consistency in a large clinical sample (total scale α=0.89; subscale intrapersonal empowerment α=0.89; subscale interactional empowerment α=0.79).

Psychosis risk screening

The Prodromal Questionnaire (PQ-16)74 is a self-report questionnaire used to screen for subclinical psychotic symptoms that may indicate an increased risk of psychotic disorder in the future. It consists of 16 items that can be rated as true (1) or false (0), based on subjective experiences during the previous month. If true, the distress score will be measured on a 4-point Likert scale ranging from 0 (no distress) to 3 (severe distress). The total score on the PQ-16 can be calculated by adding up all items, ranging from 0 to 16. A cut-off score of ≥6 predicts a high-risk status with high sensitivity (87%) and specificity (87%).74 Research has shown good psychometric properties for the PQ-16.74–76

Treatment satisfaction

Satisfaction with treatment will be measured with four brief questions based on the Jeugdthermometer.77 Treatment satisfaction is rated on a scale from 1 (very bad) to 10 (very good).

Care utilisation

At baseline assessment, patients will report their previous care utilisation via a single multiple-choice question.

Care utilisation and coordination

At follow-up, patients will report which forms of care they receive in addition to Youth Flexible ACT. In addition, they will report their satisfaction between the cooperation with the Youth Flexible ACT team and other healthcare facilities on a scale from 1 (unsatisfactory) to 10 (very satisfactory).

Parent reported outcomes

Psychosocial well-being

The Strengths and Difficulties Questionnaire for Parents (SDQ-P) is the parent version of the SDQ, as described above.59 The 20-item total difficulties subscale is allocated to four domains: emotional symptoms, conduct problems, hyperactivity inattention and peer problems. Parents rate the items on a 3-point Likert scale, ranging from 0 (not true) to 2 (certainly true). Scale scores can be computed by summing the scores on the scale items (range 0–10). A total difficulty score can be computed by adding the four scale scores (range 0–40). Scores above the cut-off of 14 (>90th percentile) are considered as a raised level of psychosocial problems.59 78 Studies showed good psychometric properties of the SDQ-P.60 78 79

Health-related quality of life

The Kidscreen-10 parent report is the parent version of the Kidscreen-10, a 10-item questionnaire on quality of life, as described above.66 67

Psychological distress

The Mental Health Inventory (MHI-5)80 is a brief questionnaire and will be used to assess parental psychological distress. The MHI-5 consists of five items scored on a 6-point Likert scale ranging from 0 (all of the time) to 5 (none of the time).

The total score will be obtained by summing up recoded scores, and transforming the scores to a scale ranging from 0 to 100. A higher score indicates better mental health and lower psychological stress levels. An MHI-5 cut-off score of ≤60 will be used to indicate psychological distress.81 The MHI-5 has shown good psychometric properties.82–85

Parenting stress

Parenting stress will be measured using the Short version of the Parenting Stress Questionnaire.86 The questionnaire focuses on the quality of the parent–child relationship measured using 10 items rated on a 4-point scale ranging from 1 (is completely applicable to me) to 4 (is not applicable to me). A total score will be calculated by summing all recoded items (ranging from 10 to 40) with higher total scores indicating higher levels of parental stress. Scores higher than 22 reflect raised level of parenting stress. Research has shown sufficient psychometric properties.73 86 87

Treatment satisfaction

Satisfaction with treatment will be measured using four brief questions based on the Jeugdthermometer parent version.77 The items are rated on a scale from 1 (very bad) to 10 (very good).

Youth Flexible ACT model fidelity

The association between psychosocial functioning of adolescents and (elements of) model fidelity will be studied. The level of implementation of the model will be measured with the Youth Flexible ACT scale (Youth FACTs) developed by the Centre for Certification ACT and Flexible ACT (CCAF) in 2014.42 The CCAF was established in the Netherlands in 2008 by Dutch mental healthcare professionals and researchers to ensure the quality of ACT and Flexible ACT. During the Youth Flexible ACT Study each team will be subjected to a single and official audit performed by the CCAF. These audits determine the degree to which each team complies with the Youth Flexible ACT model. All teams will be audited within a period of 1.5 years. The Youth FACTs consists of 62 items measuring seven main elements: team structure (15 items), team process (12 items), diagnostics, treatment and interventions (12 items), organisation of services (11 items), community care (5 items), monitoring (3 items) and level of professionalisation (4 items). Two independent raters from CCAF will score the Youth FACTs on a 5-point rating scale ranging from 1 (minimum implementation) to 5 (maximum implementation). The CCAF defines a total score on the Youth FACTs of 3.0 and lower as insufficient implementation while scores between 3.1 and 3.3 indicate a temporary certificate for 1 year, with improvements to be made to obtain a final certificate. Scores 3.4–4.0 are sufficient to receive the certificate while scores of 4.1 and higher are regarded as excellent. If a team has received a certificate, it remains valid for 3 years. During this period, the team is expected to report any major changes, including reorganisations and divisions, to the CCAF.

Administrative data

Sociodemographic and clinical characteristics (sex, age, psychiatric diagnose, treatment duration) will be collected via electronic patient records.

Statistical analysis

Latent growth curve analysis will be used to examine change in psychosocial functioning over time (T0, T6, T12, T18 in months) and its relation to model fidelity. We hypothesise that (1) psychosocial problems will decrease while treated by a Youth Flexible ACT team and (2) higher Youth Flexible ACT model fidelity will lead to greater improvements in psychosocial functioning over time. This change over time will be calculated with growth parameters (intercept, linear slope and possible quadratic factor) for each patient. For each of the study outcomes, the best fit function (model) will be assessed to measure change in outcomes. Control variables (age, gender and psychiatric diagnoses of patients) will be included as covariates in the growth model. In addition, the multiple elements of model fidelity will be used as predictors of the growth parameters. Regression coefficients will indicate the extent to which these elements contribute to these parameters. To deal with missing values, the full information maximum likelihood estimator, or the using-all-available information method, will be used. Furthermore, the data will be clustered within subjects and teams. Seven mental health institutions will participate in the study, with a total of 16 teams and an expected average of 20 patients per team. If the results of this study demonstrate substantial variation between the participating teams, analysis will be performed to account for team effects.

Sample size calculation

Given the practical challenges in conducting a multicentre study with a complex intervention, the aim of this study is to achieve maximum participation.88 A power analysis for a paired t-test with G*power indicated that a minimum of 156 patients will have to be recruited to achieve a power of 0.80, an alpha of 0.05 and an effect size of 0.20 (one tailed). Slightly more patients will be required (n=165) for a small slope effect of a linear latent growth model.89 When accounting for missing values, a target sample size of 200 participants should be sufficient. The past inflow data of all participating Youth Flexible ACT teams show that approximately 500 patients are included in care every year. This indicates that a sample size of 200 is attainable. Concerning follow-up measurements, the included teams indicated that most patients finish their Youth Flexible ACT treatment in 1–2 years. This suggests that most of the participants will remain in Flexible ACT care for the duration of the study (ie, 1.5 years). We expect being able to continue collecting data on most of these participants through follow-up assessments. The participating teams estimated that less than 5% of patients drop-out of Youth Flexible ACT care entirely. To ensure that the sample size is reached, a minimum number of inclusions are determined for each team. In addition, each team strives for maximum inclusion beyond this minimum bound.


Preliminary reflection on the limitations and strengths of the design

The observational and naturalistic character of the study design is both its weakness and its strength. Conclusions are restricted to associations between Youth Flexible ACT care and treatment outcome with the obtained data. No causal relationships can be implied because a control group is absent. Realising a resembling control group and providing them treatment as usual is practically impossible to achieve, due to the complexity of the psychosocial problems, vulnerability to mental health crisis and an extensive avoidance of mental health services. Additionally, as it is difficult to find a resembling study population, it is not possible to match the research results with a data set of another comparable cohort study. Furthermore, as a consequence of a naturalistic study design, the data obtained in the practical field of mental healthcare are subject to transitions and developments during data collection (eg, changes in team structure). Nevertheless, an important strength of the current study is its strong external validity, as Youth Flexible ACT will be studied as it is used in daily practice.90 Another strength of the study is its longitudinal design with follow-up assessments up to 18 months, providing the opportunity to evaluate long-term effects. Furthermore, examining a broad set of outcomes (both psychiatric and social functioning) allows for a more complete view of Youth Flexible ACT, since the data on participant and service characteristics will be collected from a large sample of patients and mental health workers.

Implications for clinical practice

When adolescents experience complex problems affecting various aspects of their lives, fragmentation of care services can lead to inconsistent and ineffective care. Youth Flexible ACT actively engages adolescents in treatment and provides a flexible response to the needs in different stages of care that enhances continuity of care. By providing assertive and integrated treatment, Youth Flexible ACT aims to tackle this fragmented mental healthcare system. The present study will contribute to clinical practice by providing insights into the effectiveness of Youth Flexible ACT and the essential elements responsible for the effect. This will provide valuable information for mental healthcare organisations, funding organisations and policymakers on how to maximise the quality of care for a vulnerable group of adolescents for whom the existing regular outpatient mental healthcare is hardly suitable.

Ethics and dissemination

The medical ethics committee CMO Region Arnhem-Nijmegen in the Netherlands concluded that the present study (NL57443.091.16) does not require medical ethical approval. In addition, this study received ethical approval from Trimbos Ethics Committee (201607_75-FACT2). The Trimbos Institute is the Netherlands Institute of Mental Health and Addiction, a non-profit research and knowledge centre that reviews non-WMO (Medical Research Involving Human Subjects Act) research projects. This committee ruled that our protocol complies with all applicable regulations. This approval applies for all participating institutions. Written informed consent from adolescents and parents or legal guardians will be obtained. The results of the study will be reported in accordance with the STROBE statement. Results will be disseminated via peer-reviewed academic journals and presentations at conferences. In addition, results will be made available for participating sites, funders and researchers.

Contributors:: MB will be responsible for data collection and data analysis as well as for reporting the study results. AAV will support the data analysis. DHMC, NF and HK read the manuscript and provided suggestions for improvement. DHMC, NF and HK also serve as supervisors. All authors read and approved the final manuscript.
Funding:: This study received external funding from Olim Foundation, a non-profit mental health foundation aimed at improving the quality of mental healthcare in the east part of North Brabant, the Netherlands. The Olim Foundation provided the funding after reviewing and approving the present protocol. Additionally, each of the six participating institutions will pay a small contribution of €7600 to cover research material expenses including online data managing systems. Finally, the Trimbos Institute will provide the necessary research expertise.
Disclaimer:: The Olim Foundation has no role in the collection, analysis and interpretation of data, or in the writing of manuscripts.
Competing interests:: None declared.
Patient consent for publication:: Not required.
Provenance and peer review:: Not commissioned; externally peer reviewed.


GGZ Nederland Jeugd-ggz: Investeren in de toekomst. Ambities voor 2011-2014, 2011 Available: [Accessed 16 Jul 2019 ].

GGZ Nederland Jeugd-ggz: beschikbaar, bereikbaar, betrouwbaar en in beweging. Een visie OP jeugd-ggz, 2014 Available:,%20beschikbaar,%20bereikbaar%20en%20in%20beweging.pdf [Accessed 16 Jul 2019 ].

Delespaul   P (Flexible) ACT is voor ernstige psychiatrische aandoening (EPA) : Van Veldhuizen   R , Polhuis   D , Bähler   M , et al , Handboek (flexible) act. Utrecht: De Tijdstroom, 2015: 2030.

Hendriksen-favier   A Modelbeschrijving fact Jeugd. Herziene versie. Utrecht: Trimbosinstituut, 2013.

Jones   A , Hannigan   B , Coffey   M , et al Traditions of research in community mental health care planning and care coordination: a systematic meta-narrative review of the literature. PLoS One 2018;13:e0198427 doi: 10.1371/journal.pone.0198427

McDaid   D , Oliveira   MD , Jurczak   K , et al Moving beyond the mental health care system: an exploration of the interfaces between health and non-health sectors. J Ment Health 2007;16:18194. doi: 10.1080/09638230701279873

Den Otter   J , Verschuur   F , Schell   P Assertive community treatment ook voor jeugd? Tijdschrift Sociale Psychiatrie 2009;89:1521.

Storm   A , Frieswijk   N , Hendriksen-Favier   A Fact ALS organisatiemodel voor langdurig zorgafhankelijke kinderen en jongeren. Kind Adolesc. Prakt. 2013;12:5261. doi: 10.1007/s12454-013-0018-1

Van Veldhuizen   R EPA-patienten, hun noden en eisen aan zorg : Van Veldhuizen   R , Polhuis   D , Bähler   M , et al. , Handboek (Flexible) ACT. Utrecht: De Tijdstroom, 2015: 329.


van Veldhuizen   JR Fact: a Dutch version of act. Community Ment Health J 2007;43:42133. doi: 10.1007/s10597-007-9089-4


Sood   L , Owen   A A 10-year service evaluation of an assertive community treatment team: trends in hospital bed use. J Ment Health 2014;23:3237. doi: 10.3109/09638237.2014.954694


Dixon   L , Treatment   AC Assertive community treatment: twenty-five years of gold. Psychiatr Serv 2000;51:75965. doi: 10.1176/


Marshall   M , Lockwood   A Assertive community treatment for people with severe mental disorders. Cochrane database Syst Rev 2000;2 doi: 10.1002/14651858


Burns   T The rise and fall of assertive community treatment? Int Rev Psychiatry 2010;22:1307. doi: 10.3109/09540261003661841


Van Vugt   MD Assertive community treatment in the Netherlands. Rotterdam: Erasmus University, 2015: 920.


Lambert   M , Bock   T , Schöttle   D , et al Assertive community treatment as part of integrated care versus standard care: a 12-month trial in patients with first- and multiple-episode schizophrenia spectrum disorders treated with quetiapine immediate release (access trial). J Clin Psychiatry 2010;71:131323. doi: 10.4088/JCP.09m05113yel


Luo   X , Law   SF , Wang   X , et al Effectiveness of an Assertive Community Treatment program for people with severe schizophrenia in mainland China - a 12-month randomized controlled trial. Psychol Med 2019;49:96979. doi: 10.1017/S0033291718001629


Killaspy   H , Bebbington   P , Blizard   R , et al The REACT study: randomised evaluation of assertive community treatment in North London. BMJ 2006;332:81520. doi: 10.1136/bmj.38773.518322.7C


Sytema   S , Wunderink   L , Bloemers   W , et al Assertive community treatment in the Netherlands: a randomized controlled trial. Acta Psychiatr Scand 2007;116:10512. doi: 10.1111/j.1600-0447.2007.01021.x


Marshall   M , Creed   F Assertive community treatment - is it the future of community care in the UK? International Review of Psychiatry 2000;12:1916. doi: 10.1080/irp.


Van Veldhuizen   JR , Bähler   M Flexible Assertive community treatment: vision, model, practice and organization (manual). The Netherlands: Groningen, 2013


Lexén   A , Svensson   B Mental health professional experiences of the flexible assertive community treatment model: a grounded theory study. J Ment Health 2016;25:37984. doi: 10.1080/09638237.2016.1207236


Firn   M , Hindhaugh   K , Hubbeling   D , et al A dismantling study of assertive outreach services: comparing activity and outcomes following replacement with the fact model. Soc Psychiatry Psychiatr Epidemiol 2013;48:9971003. doi: 10.1007/s00127-012-0602-x


Nordén   T , Norlander   T Absence of positive results for flexible Assertive community treatment. What is the next approach? Clin Pract Epidemiol Ment Health 2014;10:8791. doi: 10.2174/1745017901410010087


van Veldhuizen   R , Delespaul   P , Kroon   H , et al Flexible ACT & Resource-group ACT: different working procedures which can supplement and strengthen each other. a response. Clin Pract Epidemiol Ment Health 2015;11:1215. doi: 10.2174/1745017901511010012


Norlander   T , Nordén   T The hourglass model: are there structural problems with the scarcity of positive results for flexible act? Clin Pract Epidemiol Ment Health 2015;11:1557. doi: 10.2174/1745017901511010155


Drukker   M , Maarschalkerweerd   M , Bak   M , et al A real-life observational study of the effectiveness of FACT in a Dutch mental health region. BMC Psychiatry 2008;8:93 doi: 10.1186/1471-244X-8-93


Drukker   M , Visser   E , Sytema   S , et al Flexible assertive community treatment, severity of symptoms and psychiatric health service use, a real life observational study. Clin Pract Epidemiol Ment Health 2013;9:2029. doi: 10.2174/1745017901309010202


Sood   L , Owen   A , Onyon   R , et al Flexible assertive community treatment (FACT) model in specialist psychosis teams: an evaluation. BJPsych Bull 2017;41:1926. doi: 10.1192/pb.bp.116.053967


Nugter   MA , Engelsbel   F , Bähler   M , et al Outcomes of flexible Assertive community treatment (fact) implementation: a prospective real life study. Community Ment Health J 2016;52:898907. doi: 10.1007/s10597-015-9831-2


Van Houtem-Solberg   D , Chatrou   E , Werrij   M , et al Jeugd-F-ACT: Problematiek van Een bijzondere populatie in kaart gebracht. Tijdschr Psychiatr 2015;57:8926.


Van der Kwaak   A , Kramer   A , Jeugd   EACT Onderzoek naar cliëntkenmerken, aanbod, werkwijze en effecten. Rotterdam: Bavo Europoort, 2009.


Baier   V , Favrod   J , Ferrari   P , et al Early tailored assertive community case management for hard-to-engage adolescents suffering from psychiatric disorders: an exploratory pilot study. Early Interv Psychiatry 2013;7:949. doi: 10.1111/j.1751-7893.2012.00380.x


Roosenschoon   B , Schell   P Zorg voor jongeren in de grote stad; evaluatie van ACT-Jeugd Rotterdam. Tijdschrift voor Rehabilitatie en Herstel 2013;1:416.


McGREW   JH , Danner   M Evaluation of an intensive case management program for transition age youth and its transition to Assertive community treatment. Am J Psychiatr Rehabil 2009;12:27894. doi: 10.1080/15487760903066503


Ahrens   C , Frey   J , Knoedler   WH , et al Effect of PACT on inpatient psychiatric treatment for adolescents with severe mental illness: a preliminary analysis. Psychiatr Serv 2007;58:14868. doi: 10.1176/ps.2007.58.11.1486


Vijverberg   R , Ferdinand   R , Beekman   A , et al The effect of youth assertive community treatment: a systematic PRISMA review. BMC Psychiatry 2017;17:284 doi: 10.1186/s12888-017-1446-4


Mantzouranis   G , Baier   V , Holzer   L , et al Clinical significance of assertive community treatment among adolescents. Soc Psychiatry Psychiatr Epidemiol 2019;54:44553. doi: 10.1007/s00127-018-1613-z


Petersen   L , Jeppesen   P , Thorup   A , et al A randomised multicentre trial of integrated versus standard treatment for patients with a first episode of psychotic illness. BMJ 2005;331:6025. doi: 10.1136/bmj.38565.415000.E01


Craig   TKJ , Garety   P , Power   P , et al The Lambeth early onset (Leo) team: randomised controlled trial of the effectiveness of specialised care for early psychosis. BMJ 2004;329:1067 doi: 10.1136/bmj.38246.594873.7C


Verhaegh   MJM Assertive Community Treatment and early psychosis recovery outcome after a two-year treatment period : Effectiveness of assertive community treatment in early psychosis. The Netherlands: Tilburg University, 2009: 10623.


Centre for Certification ACT and Flexible ACT. Available: [Accessed 16 Jul 2019 ].


McGrew   JH , Bond   GR , Dietzen   L , et al Measuring the fidelity of implementation of a mental health program model. J Consult Clin Psychol 1994;62:6708. doi: 10.1037/0022-006X.62.4.670


McHugo   GJ , Drake   RE , Teague   GB , et al Fidelity to assertive community treatment and client outcomes in the new Hampshire dual disorders study. Psychiatr Serv 1999;50:81824. doi: 10.1176/ps.50.6.818


van Vugt   MD , Kroon   H , Delespaul   PAEG , et al Assertive community treatment in the Netherlands: outcome and model fidelity. Can J Psychiatry 2011;56:15460. doi: 10.1177/070674371105600305


Latimer   EA Economic impacts of assertive community treatment: a review of the literature. Can J Psychiatry 1999;44:44354. doi: 10.1177/070674379904400504


Burns   T , Catty   J , Dash   M , et al Use of intensive case management to reduce time in hospital in people with severe mental illness: systematic review and meta-regression. BMJ 2007;335:33642. doi: 10.1136/bmj.39251.599259.55


van Vugt   MD , Kroon   H , Delespaul   PAEG , et al Consumer-providers in assertive community treatment programs: associations with client outcomes. Psychiatr Serv 2012;63:47781. doi: 10.1176/


von Elm   E , Altman   DG , Egger   M , et al The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol 2008;61:3449. doi: 10.1016/j.jclinepi.2007.11.008


Settipani   CA , Hawke   LD , Cleverley   K , et al Key attributes of integrated community-based youth service hubs for mental health: a scoping review. Int J Ment Health Syst 2019;13:52 doi: 10.1186/s13033-019-0306-7


Block   AM , Greeno   CG Examining outpatient treatment dropout in adolescents: a literature review. Child Adolesc Soc Work J 2011;28:393420. doi: 10.1007/s10560-011-0237-x


de Haan   AM , Boon   AE , de Jong   JTVM , et al A meta-analytic review on treatment dropout in child and adolescent outpatient mental health care. Clin Psychol Rev 2013;33:698711. doi: 10.1016/j.cpr.2013.04.005


Lavik   KO , Veseth   M , Frøysa   H , et al ‘Nobody else can lead your life’: What adolescents need from psychotherapists in change processes. Couns Psychother Res 2018;18:26273. doi: 10.1002/capr.12166


Van Veldhuizen   R , Polhuis   D , Bähler   M , et al Handboek (flexible) act. De Tijdstroom: Utrecht, 2015.


Gowers   SG , Harrington   RC , Whitton   A , et al HoNOSCA Health of the Nation Outcome Scales (Child & Adolescent Mental Health):Trainer's Guide. Chester: University of Liverpool, 1998.


Gowers   SG , Harrington   RC , Whitton   A , et al Brief scale for measuring the outcomes of emotional and behavioural disorders in children. health of the nation outcome scales for children and adolescents (HoNOSCA). Br J Psychiatry 1999;174:4136. doi: 10.1192/bjp.174.5.413


Bilenberg   N Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA)--results of a Danish field trial. Eur Child Adolesc Psychiatry 2003;12:298302. doi: 10.1007/s00787-003-0343-1


Garralda   ME , Yates   P , Higginson   I Child and adolescent mental health service use. HoNOSCA as an outcome measure. Br J Psychiatry 2000;177:528. doi: 10.1192/bjp.177.1.52


Goodman   R Psychometric properties of the strengths and difficulties questionnaire. J Am Acad Child Adolesc Psychiatry 2001;40:133745. doi: 10.1097/00004583-200111000-00015


van Widenfelt   BM , Goedhart   AW , Treffers   PDA , et al Dutch version of the strengths and difficulties questionnaire (SDQ). Eur Child Adolesc Psychiatry 2003;12:2819. doi: 10.1007/s00787-003-0341-3


Theunissen   MHC , de Wolff   M , Vugteveen   J Handleiding voor Het gebruik van de SDQ bij adolescenten (12-17 jaar) binnen de Jeugdgezondheidszorg. Vragenlijst voor Het signaleren van psychosociale problemen. Leiden: TNO, 2019.


Theunissen   MHC , de Wolff   MS , Reijneveld   SA The strengths and difficulties questionnaire self-report: a valid instrument for the identification of emotional and behavioral problems. Acad Pediatr 2019;19:4716. doi: 10.1016/j.acap.2018.12.008


Vugteveen   J , De Bildt   A , Hartman   CA , et al Using the Dutch multi-informant strengths and difficulties questionnaire (SDQ) to predict adolescent psychiatric diagnoses. Eur Child Adolesc Psychiatry 2018;27:134759. doi: 10.1007/s00787-018-1127-y


Vugteveen   J , de Bildt   A , Theunissen   M , et al Validity aspects of the strengths and difficulties questionnaire (SDQ) adolescent self-report and Parent-Report versions among Dutch adolescents. Assessment 2019:10731911198584116. doi: 10.1177/1073191119858416


Erhart   M , Ottova   V , Gaspar   T , et al Measuring mental health and well-being of school-children in 15 European countries using the KIDSCREEN-10 index. Int J Public Health 2009;54 Suppl 2:1606. doi: 10.1007/s00038-009-5407-7


Ravens-Sieberer   U , Erhart   M , Rajmil   L , et al Reliability, construct and criterion validity of the KIDSCREEN-10 score: a short measure for children and adolescents' well-being and health-related quality of life. Qual Life Res 2010;19:1487500. doi: 10.1007/s11136-010-9706-5


Ravens-Sieberer   U The KIDSCREEN questionnaires - Quality of life questionnaires for children and adolescents - Handbook. Lengerich: Pabst Science Publisher, 2006.


Bodden   D , Stikkelbroek   Y , Braet   C CDI-2: Screeningsvragenlijst voor depressie bij kinderen en jongeren (Nederlandse bewerking). Amsterdam: Hogrefe Uitgevers B.V, 2016.


Kovacs   M The child depression inventory 2 (CDI-2). Toronto, Canada: Multi Health Systems, 2011.


Ravens-Sieberer   U , Gosch   A , Rajmil   L , et al The KIDSCREEN-52 quality of life measure for children and adolescents: psychometric results from a cross-cultural survey in 13 European countries. Value Health 2008;11:64558. doi: 10.1111/j.1524-4733.2007.00291.x


Ravens-Sieberer   U , Herdman   M , Devine   J , et al The European KIDSCREEN approach to measure quality of life and well-being in children: development, current application, and future advances. Qual Life Res 2014;23:791803. doi: 10.1007/s11136-013-0428-3


Damen   H , Veerman   JW , Vermulst   AA , et al Empowerment Vragenlijst voor jongeren (youth Empowerment questionnaire; versie 3.1.). Available: [Accessed 4 Jun 2019 ].


Damen   H , Veerman   JW , Vermulst   AA , et al Parental Empowerment: construct validity and reliability of a Dutch Empowerment questionnaire (EMPO). J Child Fam Stud 2017;26:42436. doi: 10.1007/s10826-016-0568-5


Ising   HK , Veling   W , Loewy   RL , et al The validity of the 16-item version of the prodromal questionnaire (PQ-16) to screen for ultra high risk of developing psychosis in the general help-seeking population. Schizophr Bull 2012;38:128896. doi: 10.1093/schbul/sbs068


de Jong   Y , Mulder   CL , Boon   AE , et al Screening for psychosis risk among adolescents in child and adolescent mental health services: a description of the first step with the 16-item version of the prodromal questionnaire (PQ-16). Early Interv Psychiatry 2018;12:66976. doi: 10.1111/eip.12362


Lorenzo   P , Silvia   A , Federica   P , et al The Italian version of the 16-item prodromal questionnaire (iPQ-16): Field-test and psychometric features. Schizophr Res 2018;199:35360. doi: 10.1016/j.schres.2018.03.023


Kok   I , Mulder   E , Cliëntwaardering in de   GGZ Handleiding bij diverse thermometers. Trimbos instituut: Utrecht, 2005.


Maurice-Stam   H , Haverman   L , Splinter   A , et al Dutch norms for the Strengths and Difficulties Questionnaire (SDQ) - parent form for children aged 2-18 years. Health Qual Life Outcomes 2018;16:123 doi: 10.1186/s12955-018-0948-1


Becker   A , Woerner   W , Hasselhorn   M , et al Validation of the parent and teacher SDQ in a clinical sample. Eur Child Adolesc Psychiatry 2004;13:1116. doi: 10.1007/s00787-004-2003-5


Berwick   DM , Murphy   JM , Goldman   PA , et al Performance of a five-item mental health screening test. Med Care 1991;29:16976. doi: 10.1097/00005650-199102000-00008


Driessen   M Geestelijke ongezondheid in Nederland in kaart gebracht. Een beschrijving van de MHI-5 in de gezondheidsmodule van Het permanent Onderzoek Leefsituatie. Den Haag, The Netherlands: Centraal Bureau voor de Statistiek, 2011.


Rumpf   HJ , Meyer   C , Hapke   U , et al Screening for mental health: validity of the MHI-5 using DSM-IV axis I psychiatric disorders as gold standard. Psychiatry Res 2001;105:24353. doi: 10.1016/S0165-1781(01)00329-8


Means-Christensen   AJ , Arnau   RC , Tonidandel   AM , et al An efficient method of identifying major depression and panic disorder in primary care. J Behav Med 2005;28:56572. doi: 10.1007/s10865-005-9023-6


Cuijpers   P , Smits   N , Donker   T , et al Screening for mood and anxiety disorders with the five-item, the three-item, and the two-item mental health inventory. Psychiatry Res 2009;168:2505. doi: 10.1016/j.psychres.2008.05.012


Hoeymans   N , Garssen   AA , Westert   GP , et al Measuring mental health of the Dutch population: a comparison of the GHQ-12 and the MHI-5. Health Qual Life Outcomes 2004;2:23 doi: 10.1186/1477-7525-2-23


Vermulst   A , Kroes   G , de Meyer   R , et al Handleiding OBVL. Nijmegen: Praktikon, 2015


de Greef   M , van Hattum   MJC , Granger   KL , et al Predictors of parent-professional alliance in home-based parenting support. Child Youth Serv Rev 2018;89:1718. doi: 10.1016/j.childyouth.2018.04.028


Barnard   KD , Dent   L , Cook   A A systematic review of models to predict recruitment to multicentre clinical trials. BMC Med Res Methodol 2010;10:63 doi: 10.1186/1471-2288-10-63


Muthén   LK , Muthén   BO How to use a Monte Carlo study to decide on sample size and determine power. Struct Equ Modeling 2002;9:599620. doi: 10.1207/S15328007SEM0904_8


Kortrijk   HE Use of routine outcome monitoring data for evaluating Assertive Com- munity treatment. Rotterdam: Erasmus University, 2013: 1826. is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: the critical elements and psychosocial outcomes of Youth Flexible Assertive Community Treatment: a study protocol for an observational prospective cohort study&author=Marieke Broersen,Daan H M Creemers,Nynke Frieswijk,Ad A Vermulst,Hans Kroon,&keyword=child & adolescent psychiatry,community child health,public health,&subject=Public Health,1506,1724,