CRISIS AFAR: an international collaborative study of the impact of the COVID-19 pandemic on mental health and service access in youth with autism and neurodevelopmental conditions
Molecular Autism volume 14, Article number: 7 (2023)
Heterogeneous mental health outcomes during the COVID-19 pandemic are documented in the general population. Such heterogeneity has not been systematically assessed in youth with autism spectrum disorder (ASD) and related neurodevelopmental disorders (NDD). To identify distinct patterns of the pandemic impact and their predictors in ASD/NDD youth, we focused on pandemic-related changes in symptoms and access to services.
Using a naturalistic observational design, we assessed parent responses on the Coronavirus Health and Impact Survey Initiative (CRISIS) Adapted For Autism and Related neurodevelopmental conditions (AFAR). Cross-sectional AFAR data were aggregated across 14 European and North American sites yielding a clinically well-characterized sample of N = 1275 individuals with ASD/NDD (age = 11.0 ± 3.6 years; n females = 277). To identify subgroups with differential outcomes, we applied hierarchical clustering across eleven variables measuring changes in symptoms and access to services. Then, random forest classification assessed the importance of socio-demographics, pre-pandemic service rates, clinical severity of ASD-associated symptoms, and COVID-19 pandemic experiences/environments in predicting the outcome subgroups.
Clustering revealed four subgroups. One subgroup—broad symptom worsening only (20%)—included youth with worsening across a range of symptoms but with service disruptions similar to the average of the aggregate sample. The other three subgroups were, relatively, clinically stable but differed in service access: primarily modified services (23%), primarily lost services (6%), and average services/symptom changes (53%). Distinct combinations of a set of pre-pandemic services, pandemic environment (e.g., COVID-19 new cases, restrictions), experiences (e.g., COVID-19 Worries), and age predicted each outcome subgroup.
Notable limitations of the study are its cross-sectional nature and focus on the first six months of the pandemic.
Concomitantly assessing variation in changes of symptoms and service access during the first phase of the pandemic revealed differential outcome profiles in ASD/NDD youth. Subgroups were characterized by distinct prediction patterns across a set of pre- and pandemic-related experiences/contexts. Results may inform recovery efforts and preparedness in future crises; they also underscore the critical value of international data-sharing and collaborations to address the needs of those most vulnerable in times of crisis.
It is widely recognized that pediatric populations are vulnerable to sudden and pervasive disruptions in their daily life, such as those brought by the COVID-19 pandemic [1,2,3,4]. Those with neurodevelopmental disorders (NDD), broadly defined as childhood-onset chronic conditions characterized by atypical brain development [5,6,7], have been identified by parents, educators, clinicians, and policy makers alike, as requiring specific attention given their preexisting behavioral, emotional, and learning difficulties. The gamut of such difficulties is often observed in individuals with autism spectrum disorder (ASD). A neurodevelopmental disorder, ASD, is defined by impairments in social communication, and restricted and repetitive behaviors/interests that are often accompanied by comorbid neurological and psychiatric conditions, as well as varying degrees of speech-language and intellectual abilities [8,9,10,11,12,13,14]. Accordingly, a number of narrative articles and reviews have highlighted the burden on mental health that the disruptions in educational and treatment services, routine changes, and social isolation have posed for youth with ASD and related NDD [4, 15,16,17,18,19,20,21,22]. Here, we report on an international empirical effort aimed at assessing the heterogeneity of the impact of the COVID-19 pandemic in a large international sample of previously well-characterized youth with ASD and/or other NDD.
Our decision to focus on the heterogeneity of the pandemic outcomes was originally based on prior disaster research in the general population, showing that the degree of severity of prior mental illness, disaster exposure, as well as perceived risk, are predictors of negative outcomes [23,24,25]. These findings have been echoed, to some extent, in initial studies of the COVID-19 pandemic in ASD/NDD youth; these have documented care service disruptions [26,27,28,29] and negative mental health impact [4, 27, 29,30,31,32,33,34,35,36,37,38,39,40]. Most of them emphasized main group effects. Yet, findings suggest significant heterogeneity in outcomes. For example, 70% to 92% of those diagnosed with ASD/NDD have lost at least one special education or therapeutic service [26,27,28, 30], but the number of services lost during the pandemic has varied across individuals. Similarly, transitions to online service provision have occurred, but have only involved relatively smaller groups [26, 27]. In one study, approximately two-thirds of parents indicated that service disruptions negatively impacted their children’s functioning to varying degrees . In parallel, studies documenting mental health difficulties varied in the symptom domain(s) examined, ranging from behavioral and/or emotional problems [30, 31, 36, 38, 39], ASD symptoms [28, 30, 39], living skills [32, 35], difficulties, and/or sleep disruptions [30, 32, 36, 40]. For example, the first study of the pandemic in children with ASD  reported more intense and frequent behavioral problems (in 36% and 42% of the sample, respectively). An additional study reported the onset of disruptive behaviors, anxiety, sleep problems, and irritability in varying proportions of children, ranging from 11 to 28% of the sample . Overall, the scientific literature suggests that, for ASD/NDD youth, different profiles of changes in mental and service access from pre- to pandemic time occur. However, the concomitant pattern of variability in these two domains remains unclarified.
Understanding outcome heterogeneity in both service and clinical symptom changes is a necessary step to identify which individuals have greater needs. This may also facilitate the identification of protective and risk factors. Toward these goals, we adapted the Coronavirus Health and Impact Survey Initiative (CRISIS)  for Autism and Related Neurodevelopmental Conditions (AFAR). CRISIS was originally designed to capture the multifaceted nature of risk in the general population during the COVID-19 pandemic by quantitatively assessing life changes and perceived risk about COVID-19, as well as mental health before and during the pandemic. Previous work in the general population established the psychometrics of CRISIS as well as its feasibility in delineating distinct life stress profiles and their predictive role in mental health outcomes [25, 41]. The CRISIS adaptation for autism and related neurodevelopmental conditions aimed to quantify both changes in symptom domains affected by or known to impact daily life, along with changes in therapeutic services. Preserving the original structure of CRISIS also allowed a comprehensive assessment of a range of disaster- and clinically related potential predictors of pandemic-related outcomes—a topic that is emerging in the literature, likely becoming a critical focus in years to come [28, 32, 34].
To enhance the scope of AFAR assessments, we formed an international, collaborative network of investigators to collect AFAR data in clinically well-characterized youth with ASD/NDD diagnoses. Unlike most prior ASD/NDD pandemic-related work which samples relatively homogeneous geographical regions, our aggregating data from multiple, international sites offered a naturalistic observational framework to quantify the degree of disaster exposure (i.e., new COVID-19 case rates and local containment measures) as a potential predictor of outcomes. This complemented the AFAR assessment of pandemic-related experiences, pre-pandemic mental health and child characteristics, thus yielding a comprehensive investigation of putative predictors of the pandemic’s outcomes.
Additionally, the aggregation of data across the AFAR network allowed us to rapidly generate a sample of individuals with ASD/NDD richly characterized by clinicians, at a scale not readily achievable otherwise. Given the multidimensional nature of the outcomes and predictors examined, we leveraged multivariate data-driven approaches in a large sample (n = 1275) of cross-sectional AFAR data collected over the initial phase of the COVID-19 pandemic. Against this background, the present study aimed to identify potential subgroups of outcome and their predictors to inform recovery efforts and prepare for future crises.
Survey development and structure
A workgroup of ASD/NDD experts (ADM, LG, SG, PP, AT, BV) led the adaptation from the CRISIS Parent/Caregiver Baseline Form. Adapting the caregiver survey was prioritized in order to immediately capture the acute phase of the pandemic across the widest possible range of ASD/NDD youth using the same method. The adaptation aimed to include assessments of clinical domains relevant to ASD/NDD and services, while maintaining the existing structure of CRISIS [25, 42]. At the time, empirical evidence on the impact of disasters on ASD/NDD was limited to one study reporting worsening in adaptive functioning . Therefore, along with adaptive skills, we prioritized the assessment of symptoms known to be affected by or to impact adjustment needs. These encompassed restricted and repetitive behaviors/interests (RRB) [44,45,46], externalizing, internalizing symptoms, and sleep problems that often co-occur in ASD/NDD [8, 12]. Parent/caregiver questions were developed to target observable behaviors rather than attempting to seek reports on internal states. In keeping with this goal and to contain the survey’s length, new questions on a range of co-occurring psychiatric symptoms replaced the Mood State domain. The Substance Use domain was also removed given that it is more accurately measured by self-report, which is beyond the scope of the present study [47, 48]. Like CRISIS, symptoms were rated on a Likert-scale based on (1) the three months prior to the start of COVID-19 pandemic in the respondent’s geographical area and (2) the two weeks prior to completion (Prior and Current time points, respectively).
To evaluate changes in service access, we derived items from a survey developed during the pandemic and piloted with people with syndromic intellectual disabilities and their caregivers . Questions queried changes in therapeutic services typically received both within and outside school settings in the respondents geographical area, following the start of the pandemic. The remainder of the original CRISIS was unchanged, except for some rewordings or additional response options (e.g., sleep problems) as summarized in Table 1 and detailed in Additional file 1: Methods.
Like CRISIS, questions of the parent/caregiver survey were developed for individuals aged five to 21 years; a later review identified a subset of questions developmentally applicable for children as young as three years. The initial adaptation was developed in English and then translated into five other languages and updated with rewordings in consultation with the larger AFAR network. The final version 0.5.1 of AFAR Parent/Caregiver Baseline Form (3–21) included 96 independent items, with 34 questions asked twice for Prior and Current time points. Clinical contacts with caregivers of youth with ASD/NDD during the acute phase of the pandemic informed the adaptation process; at its completion, several sites reached out to caregivers of ASD/NDD youth to review the survey; they favored its distribution for data collection (see Additional file 1: Methods for more details). AFAR is freely available for use by other investigators .
Following the AFAR data collection, we used exploratory and confirmatory factor analyses to identify the underlying structure of each of the domains developed a priori and to yield summary scores for quantitative investigations (see Additional file 1: Methods for more details). As detailed in Additional file 1: Results and in Tables S5 and S6, factor analyses yielded a stable and replicable factor structure, including a single factor for Adaptive Living Skills, two factors RRB largely reflecting lower- and higher-order RRB [46, 49], and four for Co-occurring Problem Behavior (i.e., Anxiety/Affect, Oppositional Behavior, Sleep Problems, Activity/Attention), among the AFAR-specific domains. Further, consistent with the general population , results yielded a single COVID-19 Worries factor, and multiple factors for the Life Changes and Behavior/Media domains (two and five, respectively).
Data collection and selection
AFAR Parent/Caregiver Baseline surveys were collected in 15 samples at 14 research and/or clinical institutions in Europe and North America. Data were collected cross-sectionally over the first six months of the pandemic (April–October 2020; Fig. 1, Additional file 1: Methods and Table S1). Parents of children aged three to 21 years with previously established clinician-based DSM-IV/5 [6, 50] or ICD-10  diagnosis of ASD and/or other NDD were invited to complete AFAR. As discussed below, only data from individuals aged five years and older are included in the present analyses.
Along with diagnostics, when feasible, previously collected information on intellectual functioning and symptoms was shared. IRB approval for collection and sharing of de-identified AFAR and related data was obtained at each institution. For the present analyses, only data from AFAR surveys with available AFAR responses for the variables used in clustering analyses and completed within the time interval of 90% of a given sample were included. This served to minimize outliers regarding to COVID-19 infection rates and related responses.
As shown in Table 2, to assess the heterogeneity of outcomes in symptom and service changes between prior and pandemic times, primary analyses leveraged data-driven hierarchical agglomerative clustering [51, 52] Clustering of the pandemic symptom changes was complemented by analyses of symptom and service changes across the whole aggregate sample (i.e., the sample derived from aggregating all eligible data) intended to further aid interpretation of results and comparisons with the literature, as well as to underscore the relevance of assessing heterogeneity. These whole-group-level analyses included repeated measures MANCOVA for symptom severity, measures of central tendency were used for access to services (Additional File 1; Methods). Following the identification of distinct outcome subgroups, we assessed and ranked a range of variables indexing pre-pandemic characteristics, as well as pandemic-related experiences and environments as their predictors. Analyses were corrected for multiple comparisons using false discovery rate (FDR) at q < 0.05, as applicable. The codes used in analyses are available in GitHub .
Outcome heterogeneity was assessed across 11 features indexing symptom and service changes pre-pandemic to pandemic time using agglomerative hierarchical clustering [51, 52]. Among these features, seven features reflected changes in clinically relevant symptoms for ASD/NDD. They indexed differences between Current and Prior scores on the symptom domains that were empirically identified with factor analyses (see Additional file 1: Methods and Tables S5–S6). The remaining four features included the total number of services that were either lost or continued within and outside school. All scores were converted to standard z scores prior to clustering; thus, clusters (i.e., subgroups) were characterized by their profile of deviation from the aggregate sample average. The optimal cluster solution was determined using NbClust , according to a majority rule among multiple goodness-of-fit measures.
To assess the relative contribution of multiple variables (i.e., features) as predictors of the outcome subgroups identified in hierarchical clustering, we used random forest classification. As depicted in Fig. 2, we assessed 20 features including family and child variables measuring pre- and pandemic-related experiences derived from AFAR, prior clinical characterization at each contributing sample (e.g., ASD vs. non-ASD NDD, number of psychiatric comorbidities), as well as COVID-19 pandemic environmental markers, empirically derived, for each individual, from open-data sources providing information by geographical area over time [55,56,57]. The variables derived from parent responses in AFAR included: socio-demographics, the child’s pandemic experiences, pre-pandemic services received, and a summary measure of clinical severity Prior (baseline) to the pandemic. This global severity summary score, computed across the seven AFAR symptom domains, was significantly correlated with previously collected standardized symptom measures (see Additional file 1: Fig. S4). The COVID-19 pandemic environmental markers included government responses, and new COVID-19 infection rates for each child in their geographical area at the time of the data collection. Briefly, to quantify new infection rates in a given child’s geographical area at the time of the AFAR data collection, we used the publicly available from Our World in Data’s (OWID) COVID-19 tracker  and COVID-19 European regional tracker . To quantify government responses to the COVID-19 pandemic, for each child at the time of the AFAR data collection in their geographical area, we used the government stringency (GS) index computed by the Oxford Coronavirus Government Response Tracker . The GS index combines metrics of infection containment and public information campaigns. It ranges from 0 to 100, higher scores reflect stricter government policies, and it is provided by day, in each territory. See Additional file 1: Methods for more details.
We quantified the relative predictive role of the 20 features examined following a previously established analytical framework [58,59,60]. Briefly, using a permutation importance method , we indexed importance as the average out-of-bag-error (OOBE), computed across 4000 bootstrapped samples (2/3 training, 1/3 testing). The larger the OOBE value, the greater the importance for a given feature. These features were ranked in decreasing order of importance according to the obtained OOBE values (see Additional file 1: Methods for details).
Aggregate sample characteristics
Data from 1275 youth, 5- to 21-year-old, aggregated across the 15 contributing samples met inclusion criteria for subsequent analyses. As shown in Figs. 1 and 3, contributing samples were from Europe (49%) and North America (51%). They included seven samples collected across research and clinical institutions in Italy, one each from Greece and Ireland, four samples collected in the USA (two in New York, and one each in California, and Missouri), and one sample collected across five centers in Ontario, Canada, as part of a COVID-19 multi-network collaboration in pediatrics, including the Province of Ontario Neurodevelopmental Disorders COVID Mental Health collaboration (POND-CMH) Network [61,62,63]. Key demographics (age and sex a birth), diagnosis, and intellectual functioning are summarized in Fig. 3 for the aggregate and each contributing sample. Most individuals (79%, n = 1004) met diagnostic criteria for ASD; 17% (n = 214) had ADHD without ASD, and 4% (n = 57) had other NDDs, without ASD. Reflecting the known male bias of ASD/NDD [64, 65], the sample predominantly included males (88%). Among 938 (80%) youth with available intelligence estimates, 62% (n = 624) were in the Average/Above Average range, 14% (n = 141) in the Borderline range, and 24% (n = 173) had mild to profound intellectual disability. Further, over half (64%, n = 811) of the caregivers had at least a college degree; the remaining had either a high school degree (30%), or elementary education (6%). Fifty-six percent of the aggregate dataset was of European/British ancestry (see Additional file 1: Table S4). Information on prior (i.e., baseline) AFAR symptom factor severity and number of services received within and outside school are in Fig. 5; complementary characteristics are in Additional file 1: Results, Figures S1, S2 and Tables S2–S4.
Pandemic outcome subgroups
Based on the NbClust approach, the four-cluster solution resulting from hierarchical clustering was optimal (see Figs. 4 and 5 and Additional file 1: Table S7). In other words, clustering analysis yielded four subgroups of ASD/NDD youth capturing distinct outcome profiles of symptom and service changes (Fig. 4). Based on their profile of deviations from the aggregate’s average of symptoms and/or service changes, the four outcome subgroups were defined as: broad symptom worsening only (20%), primarily modified services (23%), primarily lost services (6%), and average symptom/service changes (53%). The subgroup defined as broad symptom worsening only was characterized by worsening across all symptom domains as indexed by z scores ranging between 1 and 3 (i.e., 1 to 3 standard deviations above the aggregate’s mean) but by marginal service changes relative to the aggregate’s average (i.e., within 0.5 standard deviation from the aggregate’s average). The three remaining subgroups, totaling n = 1024 (80% of the aggregate), showed symptom changes within the aggregate sample’s average (z scores < 0.5) but differed in service access. As indicated by their name, one subgroup had the most services modified, another had the most services lost, and the third subgroup had the number of services lost and continued like those of the aggregate. These differential profiles of changes in symptom and services were confirmed by one-way ANOVAs and Tukey pairwise group mean comparisons (FDR q < 0.05; Additional file 1: Table S7). Since, as expected by our naturalistic study design, differences in subgroup distributions existed between contributing samples (Additional file 1: Fig. S3), we conducted follow-up analyses including “contributing sample” as covariate. Statistical group differences were unchanged. Similarly, secondary cluster analyses on data subsets distinct by survey completion rates yielded similar results (Additional file 1: Results).
Finally, Additional file 1: Table S8 provides descriptive statistics and subgroup comparisons regarding key demographics (age and sex at birth) and intellectual functioning. As detailed in Supplementary Results in Additional file 1, the four subgroups did not significantly differ in sex distribution; there was a main effect of subgroup for age with the subgroup average service and symptom change being significantly older (11.6 ± 3.5 years) than the three other subgroups which included children in mid-childhood (mean and SD range across subgroups: 9.7–10.8 ± 3.1–3.8 years). Most youth fell in the average/above average intelligence category across subgroups except for the primarily lost services subgroup, which was characterized by significantly greater intellectual impairment.
Whole-sample average impact
In the aggregate sample, one-way repeated measures MANCOVA yielded a significant main effect of time across the seven AFAR symptom factor scores (F(7,2548) = 77.40, FDR corrected p < 0.001). Follow-up comparisons for each symptom factor revealed statistical significance worsening of sleep problems (prior = 4.9 ± 2.1, current = 5.2 ± 2.3; F(1,2548) = 8.158, FDR corrected p = 0.03; see Fig. 5). Over the initial phase of the pandemic, on average the aggregate sample lost 1.2 ± 1.7; 1.3 ± 2 services, within and outside school. One service, on average, continued either within or outside school (1.1 ± 1.4; 0.8 ± 1.3, respectively), mostly continuing via telehealth/email (88% in school and 69% outside of school). More details on service changes are in Additional file 1: Table S7 and Supplementary results.
Predictors of outcome subgroup
The RF classification model predicted subgroup membership with 81% accuracy (precision/sensitivity = 82%, recall/specificity = 75%). The top-ranked predictor (OOBE: 12%) was the number of services received at school before the pandemic (Fig. 6a). Six predictors followed, with OOBE ranging from 6 to 1%. In rank order, they included the number of services received in and outside of school, GS index, Lifestyle Stress, COVID-19 Worries, new COVID-19 infections, and child’s age. The remaining features had OOBE < 1% and, thus, were considered negligible predictors.
Across the top seven predictors, each outcome subgroup was characterized by unique combinations of increases or decreases relative to the other subgroups. For example, the primarily modified services subgroup had a greater number of services before the pandemic relative to the broad symptom worsening only and to the average symptom/service changes subgroups. This latter subgroup, however, had lower ratings of COVID-19 Worries and Lifestyle Stress and included older youth in comparison with the other three subgroups. The primarily lost services subgroup had a similarly higher number of pre-pandemic services received but had the lowest GS index and COVID-19 new infection rates, as well as lower COVID-19 Worries and Lifestyle Stress relative to the other subgroups. Notably, the broad symptom worsening only subgroup had the highest COVID-19 Worries and Lifestyle Stress ratings, even with lower infection rates and GS index relative to the primarily modified services subgroup. These subgroups’ differences across predictors were confirmed by one-way ANOVAs and Tukey pairwise group mean comparisons (FDR q < 0.05; see Table 3). Since differences in subgroup distributions existed between contributing samples (Additional file 1: Fig. S3), we conducted follow-up analyses covarying for “contributing sample” in comparisons for the total baseline services received, Lifestyle Stress, COVID-19 Worries, and child age, but not for GS index and COVID-19 rates, as they reflect known differences across contributing samples per study design; statistical subgroup differences remained unchanged.
Notably, the predictor ranked eighth—i.e., pre-pandemic global severity—negligibly contributed to the classification model. Nevertheless, for illustrative and interpretation purposes, it is plotted in Fig. 6b and its descriptive statistics for each of the subgroups are summarized in Table 3. Similar to intellectual functioning, the four outcome subgroups did not differ with respect to pre-pandemic global severity except for the primarily lost services subgroup which was significantly more severe than the other three subgroups. Given that the primarily lost services subgroup was characterized by significantly greater intellectual impairment than the other three, we repeated supplementary RF analyses including intellectual categories in the subset of youth with intellectual functioning category available (n = 926). As detailed in Supplementary Results in Additional file 1, results were virtually unchanged relative to those from primary analyses; the feature indexing intellectual functioning had negligible predictive value for subgroup membership.
Prior disaster research in the general population, including studies of the COVID-19 pandemic [25, 66, 67], has consistently highlighted heterogeneity of mental health outcomes. The present study extends this insight into a multinational large sample of ASD/NDD youth by concomitantly assessing variability of changes in symptoms and therapeutic service access over the early stage of the pandemic. Across contributing samples, data-driven analyses identified four ASD/NDD subgroups: broad symptom worsening only (20%), primarily modified services (23%), primarily lost services (6%), and average symptom/service changes (53%). Their profiles revealed that symptom and service changes have distinct patterns of covariation among youth with ASD/NDD. The subgroup with notable clinical worsening neither had the greatest number of services lost nor the greatest number of continued services, suggesting that other contributors to clinical worsening also exist. Conversely, youth with relatively stable symptoms were parsed into three subgroups that differed in service access changes going from pre-pandemic to pandemic times. Recognizing these subgroups elucidated unique effects of a set of predictors and highlighted different pathways to either stable or worsening clinical presentations. Indeed, the varying pandemic impact on symptoms in ASD/NDD youth was predicted by unique combinations of universal and ASD/NDD-related pre- and pandemic contexts in which service changes occur, rather than any one characteristic of the child, their family, or their environment alone.
Our results underscore that solely focusing on group-level effects leads to an incomplete picture of the COVID-19 pandemic impact on ASD/NDD youth. At the whole-sample level, multivariate analyses revealed that only sleep problems significantly worsened going from pre-pandemic to pandemic times. While results for this group-level approach confirm earlier pandemic reports of increased sleep problems in ASD/NDD [30, 36, 68], they failed to recognize a more vulnerable subgroup (i.e., broad symptom worsening only outcome subgroup). Indeed, cluster analysis revealed that 20% of the children worsened even more than their ASD/NDD peers. Significant worsening affected a broad range of symptoms, including sleep problems, and beyond—i.e., externalizing and internalizing symptoms, RRBs, and daily living skills. For the remaining participants, symptom changes from pre-pandemic to pandemic times were within the aggregate average picture—i.e., statistically significant increases in sleep problems with other symptom severity being relatively stable. Consistent with prior literature [26,27,28], most youth in our ASD/NDD sample experienced a variety of service disruptions across settings, resulting in loss or modification of services (e.g., telehealth). However, youth in our aggregate sample experienced notable variability in service access. The present study found that the association between service and symptom changes is complex. In fact, clustering individuals across both services and symptom changes allowed us to further parse the relatively clinically stable youth into three homogeneous subgroups differing by service access changes. This enabled a fine-grained identification of predictors of differing pandemic outcomes—i.e., distinct profiles of symptom and service access changes across the four subgroups.
In the present study, the most relevant predictors of outcome subgroup membership included factors shared with the general population [23, 69], as well as others. Specifically, elevated perception of COVID-related risk (indexed by the COVID-19 Worries factor) and pandemic-related lifestyle stressors (i.e., restrictions on leaving home, cancellations of important events) predicted broad and more severe symptom changes in youth with NDD/ASD (i.e., the broad symptom worsening only outcome subgroup). Beyond these universal stressors, the present work identified age and a set of pandemic-related and NDD/ASD-specific experiences and contexts that must be considered collectively to provide a meaningful picture of pandemic impact in NDD/ASD youth (i.e., number of services received prior to the pandemic, COVID-19 containment measures, and COVID-19 new rates). These findings indicate that, in middle childhood, a greater number of baseline services may foster resilience and be protective to broad symptom worsening during a disaster, at least over its initial phase. Additionally, for those in middle childhood and living in areas with greater restrictions at times of high COVID-19 rates, continuing services, even if in modified format, may lead to a relatively more stable clinical profile.
Of note, pre-pandemic global clinical severity and other features related to the youth clinical presentation, including intellectual skills, negligibly contributed to prediction. Most children across subgroups had equivalent baseline symptom severity and average intelligence, except for the primarily lost services subgroup which was characterized by greater pre-pandemic impairment. These findings are in contrast with earlier ASD studies [28, 32] suggesting that preexisting challenges or symptom severity are associated with greater impairment during the pandemic. In considering possible explanations, we note that unlike the present efforts, prior NDD/ASD work focused on samples from relatively narrow geographical areas with largely similar COVID-19 rates and institutional containment responses. Additionally, inter-study differences in sample severity may exist; future studies focusing on a wider range of severity ratings are needed to build upon the present findings. Finally, neither specific NDD diagnosis nor psychiatric comorbidity’s burden robustly contributed to prediction of outcome subgroups. Thus, although our sample included predominantly autistic youth, considering the breadth of clinical impairment typically observed in ASD, our results suggest that the insights gained from the present effort can inform other NDD more broadly.
Although this is the most comprehensive and systematic study of the covariation of symptom and service changes in youth with NDD/ASD during the pandemic, results should be interpreted in light of several limitations. First, considering time constraints on questionnaire completion, albeit comprehensive, AFAR could not assess all possible domains of impact and/or prediction. For example, symptoms least expected to change over a short period of time, were given lower priority, most notably, social communication impairments . Given the protracted nature of the pandemic, future studies should include assessment of longer-term changes in social communication skills. Similarly, although family demographics, parent education, and parent being an essential worker were assessed and included in our predictive model, parent’s mental health, recently reported to relate to children’s outcomes during the pandemic [71,72,73,74], was not assessed. Second, because of the urgency of the pandemic, we did not systematically involve other stakeholders in the survey adaptation process. However, the input of caregivers of children with ASD was accounted for in a number of informal ways during and at completion of the survey. Third, although the aggregate sample included youths with clinician-based diagnoses, previously collected measures of severity varied by contributing sample, and assessments of the role of prior severity were based on parent responses in the AFAR survey. Nevertheless, we found that the AFAR baseline global severity scores correlated with available standardized measures. Fourth, although relatively large, our aggregate convenience sample could not address all demographics. For example, females were underrepresented, and preschoolers were not included. Although consistent with a systematic review of pandemic studies in youth  sex at birth was not a robust predictor of impact, given recent reports of a male to female ratio of 3:1 in ASD , future studies should oversample females. Fifth, the present study focused on impact over the first six months of the pandemic using a cross-sectional design. Longitudinal coordinated study designs and infrastructures (e.g., common assessment measures) are needed to capture long-term outcomes and define stability of the subgroups over time. Finally, this first study relied exclusively on parent reports in order to capture the acute phase of the pandemic across a widest possible range of ASD/NDD youth and abilities using the same approach. Future studies should include self-report and/or direct observation to complement the parent’s perspective.
As in the general population, the COVID-19 pandemic impact varies across ASD/NDD youth. Risk and resilience are rooted in the pre- and pandemic contexts in which service disruptions occur. Provision of mental healthcare in preparation for and during disasters are critical for ASD/NDD youth, further motivating efforts assessing effectiveness for telehealth and/or hybrid treatment programs. As heightened perception of risk was among the predictors for broad symptom worsening, during disasters, special attention should be paid to how much youth are concerned about a current crisis, and guided access to clear and appropriately dosed information is needed. Finally, this study demonstrates the value of international data sharing and collaborations. It also underscores the need for increased global coordination to include common assessment protocols and data structures to facilitate data sharing and analysis aimed to more readily assess and address the needs of those most vulnerable.
Availability of data and materials
Data were subjected to third-party restrictions. The codes used for EFA/CFA, RF, and HC are available at github.com/ChildMindInstitute/CRISIS-AFAR-analyses.
- Adaptive Liv.:
Adaptive Living skills
Adapted For Autism and Related neurodevelopmental conditions
Analysis of variance
Autism spectrum disorder
Center for Autism and Developing Brain, Weill Cornell Medical College/New York Presbyterian Hospital
Confirmatory factor analysis
Child Mind Institute-Autism Center
CMI Healthy Brain Network
Coronavirus disease 2019
Coronavirus Health and Impact Survey Initiative
Diagnostic statistical manual of mental disorders
Exploratory factor analysis
False discovery rate
Full intelligence quotient
International Classification of Diseases
Multivariate analysis of covariance
Province of Ontario neurodevelopmental network, COVID mental health collaboration
Restricted and repetitive behaviors
- Sleep Prob.:
Stella Maris Foundation, University of Pisa
Trinity College Dublin
University of Athens, National & Kapodistrian University of Athens, School of Medicine, First Department of Pediatrics, Unit of Developmental and Behavioral Pediatrics. “Aghia Sophia” Children’s Hospital
University Bari, Child Neuropsychiatry Unit, Policlinic of Bari
University of Cagliari, Child & Adolescent Neuropsychiatry Unit, A.Cao Paediatric Hospital
University of California San Francisco
University of Naples Federico II, Child and Adolescent Neuropsychiatry Unit
Unita' Operativa di Neuropsichiatria dell’ Infanzia e dell' adolescenza, Lodi
University of Sassari, Child Neuropsychiatry Unit, Azienda Ospedaliero-Universitaria
University Tor Vergata
Unite Nations. Policy Brief: The Impact of COVID-19 on Children. UN; 2020.
Nearchou F, Flinn C, Niland R, Subramaniam SS, Hennessy E. Exploring the impact of COVID-19 on mental health outcomes in children and adolescents: a systematic review. Int J Environ Res Public Health. 2020. https://0-doi-org.brum.beds.ac.uk/10.3390/ijerph17228479.
Child Protection and COVID-19. Accessed November 3, 2022. https://alliancecpha.org/en/child-protection-and-covid-19.
Panda PK, Gupta J, Chowdhury SR, et al. Psychological and behavioral impact of lockdown and quarantine measures for COVID-19 pandemic on children, adolescents and caregivers: a systematic review and meta-analysis. J Trop Pediatr. 2021. https://0-doi-org.brum.beds.ac.uk/10.1093/tropej/fmaa122.
World Health Organization. The international statistical classification of diseases and health related problems ICD-10: tenth revision. Volume 2: Instruction Manual. World Health Organization; 2004.
American Psychiatric Association. Diagnostic and statistical manual of mental disorders, Fifth Edition. American Psychiatric Association; 2013.
Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime Prevalence and age-of-onset distributions of DSM-IV disorders in the national comorbidity survey replication. Arch Gen Psychiatry. 2005;62(6):593. https://0-doi-org.brum.beds.ac.uk/10.1001/archpsyc.62.6.593.
Simonoff E, Pickles A, Charman T, Chandler S, Loucas T, Baird G. Psychiatric disorders in children with autism spectrum disorders: prevalence, comorbidity, and associated factors in a population-derived sample. J Am Acad Child Adolesc Psychiatry. 2008;47(8):921–9.
Kim SH, Macari S, Koller J, Chawarska K. Examining the phenotypic heterogeneity of early autism spectrum disorder: subtypes and short-term outcomes. J Child Psychol Psychiatry. 2016;57(1):93–102.
Lord C, Brugha TS, Charman T, et al. Autism spectrum disorder. Nat Rev Dis Primers. 2020;6(1):5.
Grzadzinski R, Huerta M, Lord C. DSM-5 and autism spectrum disorders (ASDs): an opportunity for identifying ASD subtypes. Mol Autism. 2013;4(1):12.
Lai MC, Kassee C, Besney R, et al. Prevalence of co-occurring mental health diagnoses in the autism population: a systematic review and meta-analysis. Lancet Psychiatry. 2019;6(10):819–29.
Pickles A, Anderson DK, Lord C. Heterogeneity and plasticity in the development of language: a 17-year follow-up of children referred early for possible autism. J Child Psychol Psychiatry. 2014;55(12):1354–62.
Charman T, Pickles A, Simonoff E, Chandler S, Loucas T, Baird G. IQ in children with autism spectrum disorders: data from the special needs and autism project (SNAP). Psychol Med. 2011;41(3):619–27.
Constantino JN, Sahin M, Piven J, Rodgers R, Tschida J. The impact of COVID-19 on individuals with intellectual and developmental disabilities: clinical and scientific priorities. Am J Psychiatry. 2020;177(11):1091–3.
Bellomo TR, Prasad S, Munzer T, Laventhal N. The impact of the COVID-19 pandemic on children with autism spectrum disorders. J Pediatr Rehabil Med. 2020;13(3):349–54.
Eshraghi AA, Li C, Alessandri M, et al. COVID-19: overcoming the challenges faced by individuals with autism and their families. Lancet Psychiatry. 2020;7(6):481–3.
Pellicano E, Stears M. The hidden inequalities of COVID-19. Autism. 2020;24(6):1309–10.
Courtenay K, Perera B. COVID-19 and people with intellectual disability: impacts of a pandemic. Ir J Psychol Med. 2020;37(3):231–6.
Ameis SH, Lai MC, Mulsant BH, Szatmari P. Coping, fostering resilience, and driving care innovation for autistic people and their families during the COVID-19 pandemic and beyond. Mol Autism. 2020;11(1):61.
Fegert JM, Vitiello B, Plener PL, Clemens V. Challenges and burden of the Coronavirus 2019 (COVID-19) pandemic for child and adolescent mental health: a narrative review to highlight clinical and research needs in the acute phase and the long return to normality. Child Adolesc Psychiatry Ment Health. 2020;14(1):20.
O’Sullivan K, Clark S, McGrane A, et al. A qualitative study of child and adolescent mental health during the COVID-19 pandemic in Ireland. Int J Environ Res Public Health. 2021. https://0-doi-org.brum.beds.ac.uk/10.3390/ijerph18031062.
Bromet EJ, Havenaar JM. Psychological and perceived health effects of the Chernobyl disaster: a 20-year review. Health Phys. 2007;93(5):516–21.
Bromet EJ, Atwoli L, Kawakami N, et al. Post-traumatic stress disorder associated with natural and human-made disasters in the World Mental Health Surveys. Psychol Med. 2017;47(2):227–41.
Nikolaidis A, Paksarian D, Alexander L, et al. The coronavirus health and impact survey (CRISIS) reveals reproducible correlates of pandemic-related mood states across the Atlantic. Sci Rep. 2021;11(1):8139.
Jeste S, Hyde C, Distefano C, et al. Changes in access to educational and healthcare services for individuals with intellectual and developmental disabilities during COVID-19 restrictions. J Intellect Disabil Res. 2020. https://0-doi-org.brum.beds.ac.uk/10.1111/jir.12776.
White LC, Law JK, Daniels AM, et al. Brief report: impact of COVID-19 on individuals with ASD and their caregivers: a perspective from the SPARK cohort. J Autism Dev Disord. 2021. https://0-doi-org.brum.beds.ac.uk/10.1007/s10803-020-04816-6.
Bhat A. Analysis of the SPARK study COVID-19 parent survey: early impact of the pandemic on access to services, child/parent mental health, and benefits of online services. Autism Res. 2021;14(11):2454–70.
Masi A, Mendoza Diaz A, Tully L, et al. Impact of the COVID-19 pandemic on the well-being of children with neurodevelopmental disabilities and their parents. J Paediatr Child Health. 2021;57(5):631–6.
Mutluer T, Doenyas C, Aslan GH. Behavioral implications of the Covid-19 process for autism spectrum disorder, and individuals’ comprehension of and reactions to the pandemic conditions. Front Psychiatry. 2020;11: 561882.
Asbury K, Fox L, Deniz E, Code A, Toseeb U. How is COVID-19 affecting the mental health of children with special educational needs and disabilities and their families? J Autism Dev Disord. 2021;51(5):1772–80.
Colizzi M, Sironi E, Antonini F, Ciceri ML, Bovo C, Zoccante L. Psychosocial and behavioral impact of COVID-19 in autism spectrum disorder: an online parent survey. Brain Sci. 2020. https://0-doi-org.brum.beds.ac.uk/10.3390/brainsci10060341.
Nonweiler J, Rattray F, Baulcomb J, Happé F, Absoud M. Prevalence and associated factors of emotional and behavioural difficulties during COVID-19 pandemic in children with neurodevelopmental disorders. Children. 2020. https://0-doi-org.brum.beds.ac.uk/10.3390/children7090128.
Vasa RA, Singh V, Holingue C, Kalb LG, Jang Y, Keefer A. Psychiatric problems during the COVID-19 pandemic in children with autism spectrum disorder. Autism Res. 2021. https://0-doi-org.brum.beds.ac.uk/10.1002/aur.2574.
Valenti M, Pino MC, Le Donne I, et al. Adaptive response of Italian young adults with autism to the COVID-19 pandemic: a longitudinal study. Res Dev Disabil. 2022;131: 104333.
Berard M, Rattaz C, Peries M, Loubersac J, Munir K, Baghdadli A. Impact of containment and mitigation measures on children and youth with ASD during the COVID-19 pandemic: report from the ELENA cohort. J Psychiatr Res. 2021;137:73–80.
Alonso-Esteban Y, López-Ramón MF, Moreno-Campos V, Navarro-Pardo E, Alcantud-Marín F. A systematic review on the impact of the social confinement on people with autism spectrum disorder and their caregivers during the COVID-19 pandemic. Brain Sci. 2021. https://0-doi-org.brum.beds.ac.uk/10.3390/brainsci11111389.
Amorim R, Catarino S, Miragaia P, Ferreras C, Viana V, Guardiano M. The impact of COVID-19 on children with autism spectrum disorder. Rev Neurol. 2020;71(8):285–91.
Martínez-González AE, Moreno-Amador B, Piqueras JA. Differences in emotional state and autistic symptoms before and during confinement due to the COVID-19 pandemic. Res Dev Disabil. 2021;116: 104038.
Bruni O, Melegari MG, Breda M, et al. Impact of COVID-19 lockdown on sleep in children with autism spectrum disorders. J Clin Sleep Med. 2022;18(1):137–43.
Nikolaidis A, DeRosa J, Kass M, et al. Heterogeneity in COVID-19 pandemic-induced lifestyle stressors predicts future mental health in adults and children in the US and UK. J Psychiatr Res. 2022;147:291–300.
CRISIS AFAR, The CoRonavIruS Health Impact Survey (CRISIS). The CoRonavIruS Health Impact Survey (CRISIS). Accessed March 30, 2022. http://www.crisissurvey.org/crisis-afar/.
Valenti M, Ciprietti T, Egidio CD, et al. Adaptive response of children and adolescents with autism to the 2009 earthquake in L’Aquila. Italy J Autism Dev Disord. 2012;42(6):954–60.
Peters-Scheffer N, Didden R, Sigafoos J, Green VA, Korzilius H. Behavioral flexibility in children with autism spectrum disorder and intellectual disability. Res Autism Spectr Disord. 2013;7(6):699–709. https://0-doi-org.brum.beds.ac.uk/10.1016/j.rasd.2013.02.016.
D’Cruz AM, Ragozzino ME, Mosconi MW, Shrestha S, Cook EH, Sweeney JA. Reduced behavioral flexibility in autism spectrum disorders. Neuropsychology. 2013;27(2):152–60.
Bishop SL, Hus V, Duncan A, et al. Subcategories of restricted and repetitive behaviors in children with autism spectrum disorders. J Autism Dev Disord. 2013;43(6):1287–97.
Ressel M, Thompson B, Poulin MH, et al. Systematic review of risk and protective factors associated with substance use and abuse in individuals with autism spectrum disorders. Autism. 2020;24(4):899–918.
Piehler TF, Lee SK, Stockness A, Winters KC. The correspondence of parent-reported measures of adolescent alcohol and cannabis use with adolescent-reported measures: a systematic review. Subst Abus. 2020;41(4):437–50.
Leekam SR, Prior MR, Uljarevic M. Restricted and repetitive behaviors in autism spectrum disorders: a review of research in the last decade. Psychol Bull. 2011;137(4):562–93.
American Psychiatric Association, American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition: DSM-IV-TR®. American Psychiatric Association; 2000.
Maechler M, Rousseeuw P, Struyf A, Hubert M, Hornik K. Cluster: Cluster Analysis Basics and Extensions. R package version 2.1. 2—For new features, see the ‘Changelog’file (in the package source). (2021).
Murtagh F, Legendre P. Ward’s hierarchical agglomerative clustering method: which algorithms implement ward’s criterion? J Classification. 2014;31(3):274–95.
CRISIS-AFAR-analyses: this repository includes the code accompanying the CRISIS-AFAR Paper. Github Accessed November 7, 2022. https://github.com/ChildMindInstitute/CRISIS-AFAR-analyses.
Charrad M, Ghazzali N, Boiteau V, Niknafs A. NbClust: an R package for determining the relevant number of clusters in a data set. J Stat Softw Artic. 2014;61(6):1–36.
Hale T, Petherick A, Phillips T, Webster S. Variation in government responses to COVID-19. Blavatnik school of government working paper. 2020;31:2020–2011.
Roser M, Ritchie H, Ortiz-Ospina E, Hasell J. Coronavirus pandemic (COVID-19). Our world in data. Published online 2020. https://ourworldindata.org/coronavirus/country/sweden?country=~SWE&fbclid=IwAR3JDc2cGwgfSuYly7pBg9hdMqlhwN--Z5UR1qjVWV_NB44KcPwwMexY9qM.
Naqvi A. COVID-19 European regional tracker. Sci Data. 2021. https://0-doi-org.brum.beds.ac.uk/10.1101/2021.02.15.21251788.
Genuer R, Poggi JM, Tuleau-Malot C. Variable selection using random forests. Pattern Recognit Lett. 2010;31(14):2225–36.
Alvarez-Fernandez S, Brown HR, Zhao Y, et al. Perceived social support in adults with autism spectrum disorder and attention-deficit/hyperactivity disorder. Autism Res. 2017;10(5):866–77.
Breiman L. Random forests. Mach Learn. 2001;45(1):5–32.
Baribeau DA, Dupuis A, Paton TA, et al. Structural neuroimaging correlates of social deficits are similar in autism spectrum disorder and attention-deficit/hyperactivity disorder: analysis from the POND Network. Transl Psychiatry. 2019. https://0-doi-org.brum.beds.ac.uk/10.1038/s41398-019-0382-0.
Kushki A, Anagnostou E, Hammill C, et al. Examining overlap and homogeneity in ASD, ADHD, and OCD: a data-driven, diagnosis-agnostic approach. Transl Psychiatry. 2019;9(1):318.
Cost KT, Crosbie J, Anagnostou E, et al. Mostly worse, occasionally better: impact of COVID-19 pandemic on the mental health of Canadian children and adolescents. Eur Child Adolesc Psychiatry. 2021. https://0-doi-org.brum.beds.ac.uk/10.1007/s00787-021-01744-3.
Loomes R, Hull L, Mandy WPL. What is the male-to-female ratio in autism spectrum disorder? A systematic review and meta-analysis. J Am Acad Child Adolesc Psychiatry. 2017;56(6):466–74.
Rucklidge JJ. Gender differences in attention-deficit/hyperactivity disorder. Psychiatr Clin North Am. 2010;33(2):357–73.
Ellwardt L, Präg P. Heterogeneous mental health development during the COVID-19 pandemic in the United Kingdom. Sci Rep. 2021;11(1):15958.
Ma Z, Idris S, Zhang Y, et al. The impact of COVID-19 pandemic outbreak on education and mental health of Chinese children aged 7–15 years: an online survey. BMC Pediatr. 2021;21(1):1–8.
Türkoğlu S, Uçar HN, Çetin FH, Güler HA, Tezcan ME. The relationship between chronotype, sleep, and autism symptom severity in children with ASD in COVID-19 home confinement period. Chronobiol Int. 2020;37(8):1207–13.
Miura I, Nagai M, Maeda M, et al. Perception of Radiation risk as a predictor of mid-term mental health after a nuclear disaster: the Fukushima health management survey. Int J Environ Res Public Health. 2017. https://0-doi-org.brum.beds.ac.uk/10.3390/ijerph14091067.
Anagnostou E, Jones N, Huerta M, et al. Measuring social communication behaviors as a treatment endpoint in individuals with autism spectrum disorder. Autism. 2015;19(5):622–36.
Fosco GM, Sloan CJ, Fang S, Feinberg ME. Family vulnerability and disruption during the COVID-19 pandemic: prospective pathways to child maladjustment. J Child Psychol Psychiatry. 2022;63(1):47–57.
Russell BS, Hutchison M, Tambling R, Tomkunas AJ, Horton AL. Initial challenges of caregiving during COVID-19: caregiver burden, mental health, and the parent-child relationship. Child Psychiatry Hum Dev. 2020;51(5):671–82.
Ueda R, Okada T, Kita Y, et al. Quality of life of children with neurodevelopmental disorders and their parents during the COVID-19 pandemic: a 1-year follow-up study. Sci Rep. 2022;12(1):4298.
Kalb LG, Badillo-Goicoechea E, Holingue C, et al. Psychological distress among caregivers raising a child with autism spectrum disorder during the COVID-19 pandemic. Autism Res. 2021;14(10):2183–8.
First and foremost, we are immensely grateful to all parents/caregivers and their children who generously contributed their time during the pandemic, one of the most challenging times for humankind in this century. We are also grateful to the research and clinical staff members at each contributing site supporting different aspects of collection of the AFAR survey data and the pre-pandemic clinical phenotypic data. We thank Irene Droney at CMI for support in the development and curation of the AFAR website, and RedCap data structure, Dr. Greg Kiar at CMI for helpful discussions about random forest classification, and Dr. Marco Pagani for helpful discussions on the use of NbClust in R. We are also thankful to Dr. Shafali Jeste for sharing the preprint version of the service survey developed during the pandemic for individuals with syndromic intellectual disabilities and their caregivers that we adopted to develop the AFAR service-related questions. Finally, it is with profound sadness that we wish to remember the clinical and scientific acumen, as well as unwavering dedication to child psychiatry of our coauthor, Professor Alessandro Zuddas, MD, who prematurely passed away after completion of the originally submitted manuscript.
This work was partially supported by grants from the National Institute of Mental Health (R01MH105506, R01MH115363) and gifts to the Child Mind Institute (CMI) from Dr. John and Consuela Phelan for support of ADM; and from Phyllis Green and Randolph Cowen for MPM. We would also like to thank the many individuals who have provided financial support to the CMI Healthy Brain Network to make the creation and sharing of this resource possible; the Italian Ministry of Health, Ricerca Corrente 2021/2.06 to A.N., R.T. and G.M; The Ontario COVID & Kids Mental Health Study is funded by the Canadian Institutes for Health Research (#173092); the Ontario Ministry of Health (#700); Centre of Brain and Mental Health, SickKids; Leong Centre for Healthy Children, SickKids; and the Miner’s Lamp Innovation Fund in Prevention and Early Detection of Severe Mental Illness, University of Toronto. In-kind support was provided by the Ontario Brain Institute for all POND-CMH data. Spit for Science was funded by the Canadian Institutes of Health Research (PJT-159462). The views of the Ontario COVID & Kids Mental Health Study do not necessarily represent those of the Province of Ontario and the Ontario Ministry of Health.
Ethics approval and consent to participate
All contributing data from each institution were approved by the local Institutional Review Boards (IRB) for data collection and sharing. All AFAR and related data were de-identified prior to data sharing.
Consent for publication
Dr. Bishop receives royalties for the sale of the ADOS-2 that she has co-authored. Royalties generated from any of their own research or clinical activities are donated to charity. Dr. Tancredi receives royalties for the sale of the Italian version of the ADOS-2. The other authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Vibert, B., Segura, P., Gallagher, L. et al. CRISIS AFAR: an international collaborative study of the impact of the COVID-19 pandemic on mental health and service access in youth with autism and neurodevelopmental conditions. Molecular Autism 14, 7 (2023). https://0-doi-org.brum.beds.ac.uk/10.1186/s13229-022-00536-z