I have also written a formal reply which has been published as a response on Pediatrics’ site which you can access and read (free) by clicking here.
Source: Unknown
When articles get published, they go through what’s called the peer-review process. During this process, two or three individuals, hopefully in your field and hopefully aware of the topic at hand (though I can tell you from experience that is not always the case) read the manuscript, make comments for editing, and give a final say as to whether or not it should be published. Recently, an article by Anna Price and colleagues
So here it goes…
—–
Overview.
The article presents findings from the Kids Sleep Study, a follow-up from the Infant Sleep Study, which took place in 2003 in Australia. The current findings pertain to child and parent outcomes for children who demonstrated sleep problems and who were either placed in an intervention group or control group when the children were 6 years of age. The topic itself is very interesting and much research is needed in this field. The authors rightly present that there is debate around infant sleep interventions with no clear research to date demonstrating long-term harms or benefits. However, the current study suffers from several flaws which make the conclusions drawn by the authors quite premature. Given the data collected and the analyses performed, the conclusions are unfounded.
Introduction.
The introduction does a decent job of covering the literature but fails to mention the very real shortcomings of the preexisting reviews (e.g., Mindell et al., 2006) – namely, that many studies (which comprise the review in question) do not have an appropriate control group. The failure to adequately control for natural changes in infant sleep, maternal depression, and the subsequent changes in parent-child relationships invalidates many of the conclusions reached in previous research. In fact, one of the strengths of the current study and the previous Hiscock and colleague articles (2003; 2005) is the randomized trial nature of the study. I suggest the authors look at the Infant Sleep Information Source run by Dr. Helen Ball and colleagues for more information.
A second concern is the statement that “no studies… have reported detrimental effects”. This is a rather disingenuous statement given the paucity of research looking at the outcomes that are hypothesized to be related to detrimental effects. In fact, in the Mindell et al. review cited only three articles examined secondary outcomes to sleep training and only one outcome is relevant to the concerns regarding sleep training – attachment. However, even attachment status was not measured appropriately, but rather using a non-validated self-report version of a measure that is supposed to be given by a trained professional in all three cases. None of the measures considered measuring objective child-parent attachment, but focused solely on parent-reported attachment. (Of note, the other main sleep review by Owens, France, & Wiggs, 1999, suffers the same problems.)
Third, the statement that “teaching parents to regulate their children’s sleep behaviour is a form of limit setting that… constitutes the optimal, authoritative, parenting style for child outcomes” requires much more backing than articles simply arguing that authoritative parenting styles are optimal (which is not in debate here). How does regulating a child’s sleep pertain to limit setting? How does a parent who does not “regulate” sleep fail in this regard? You fail to define “regulate” and “limit setting” and fail to cite research backing this assertion raising questions about its veracity.
Methods.
First I must commend the developers of the Sleep Study for actually utilizing a control group and utilizing randomized trials. This is rarely seen in sleep research and, as mentioned previously, raises concerns about the conclusions that have been drawn to date. The inclusion of a comparable control group allows for comparisons when methods and other controls are appropriate. However, I have strong concerns over the methodology utilized herein. Below are specific comments.
- First, the presence or absence of a “sleep problem” in infancy was parent-reported and not, so far as can be told, verified objectively. It is quite possible that parents’ feelings of self-efficacy from doing something (intervention) improved their own perception of their child early and later. The intervention and results do not seem to speak to fixing sleep problems so much as fixing the parental belief about a sleep problem (which can sometimes be fixed when parents are only given information about normative sleep patterns).
- There is no discussion of the training and adherence to study protocol by the intervention nurses. Can you provide how many followed the script during random checks? Especially during the later months of the study when standardization tends to suffer most?
- You mention that the control nurses were free to give advice, but were not trained to give the standardized response. Did you control for what information they gave? Given that infant behavioural interventions are quite common, is it not possible they provided some of the same information as the intervention nurses?
- Of the intervention group, individuals attended an average of 1.52 visits (presumably out of 2 given the data on mean length of visit). What were the reasons for not attending a second visit? Are there qualitative differences between the two groups on any of the variables of interest? What material was covered at each meeting and therefore what material was missed by those who only attended one session?
- The authors have written that “each family chose which (if any) type or mix of strategies they would use” and that only 100 of 174 families selected a strategy and attended the meetings. This is crucial and poses a large problem for the later analyses. First, there is no check that there are differences between the two strategies included in the intervention and yet no prior research to suggest they are equal in possible long-term effects. Second, there is no mention of how those that selected an intervention were different from those families that did not. What was the reason the other families did not select in? Did they have problems with the interventions? Did they believe the child’s sleep problems were not great enough to warrant intervention? It is also unclear if all intervention families received the information about positive bedtime routines or just those who also selected to use a behavioural management strategy.
- The saliva testing is a great addition to the measures; however, it is questionable if providing only one day’s worth of saliva (two measures) is an accurate assessment of chronic stress. A pattern over a few days would have been preferable. However, I realize it would be near impossible to obtain this now, so it should be discussed as a limitation to the current results.
- With the exception of health-related quality of life and stress, all child measures are parent-report. This poses a large problem, particularly when parents selected whether or not to utilize the interventions. As mentioned previously with respect to fixing sleep problems, parents’ perceptions may be coloured by their choice and feelings of having done something rather than the intervention itself. Without objective measures, it is impossible to rule this out, especially when they know the follow-up is to do with their earlier action (or inaction). This is particularly problematic for the child and child-parent measures. For example, the article cannot claim to have measured the parent-child relationship, but rather the parental perception of their relationship with their child. And if the effects of sleep training are presumed to be on the child, the measures are failing to capture this construct.
Analyses.
Overall, the analyses are well handled. Clustering and including confounds are to be recognized as being recommended and solid statistics. I am pleased to see the inclusion of research-based controls including gender, temperament, depression, and SES; however, you mention that there are analyses for which these were not included to avoid instability. Have you considered running a Structural Equation Model with all of your variables included in order to avoid this potential problem?
The largest problem is that this is one of the cases in which utilizing the intention-to-treat principle is unwarranted. Yes, generally this is something we want to consider, especially when groups self-select (e.g., in the case of home birth versus a hospital birth); however, you are looking at and making conclusions about outcomes in which nearly half of the intervention group declined the intervention for which they were randomly assigned. Notably, you fail to report how many of the follow-up individuals were from this group that did not utilizing any intervention (though clearly some of them belong to this group as your final intervention n=122 when only n=100 agreed to take part in one of the behavioural intervention strategties, and perhaps more from the latter group dropped out – we have no way of knowing). When making conclusions about the long-term effects of a particular strategy in a randomized trial, the intention-to-treat principle muddies the results because it is not providing outcomes pertaining to the strategies at hand and may be influenced (positively or negatively) by those who did not choose to take part in the intervention they were randomly assigned to. Please redo your analyses including only the group that did agree to treatment. Additionally, why did you expect intracluster correlations to fade over 5 years? If families remain in the same area, there are many variables expected to remain correlated. Acknowledging that you did the analysis with correlations allowed and results were similar would be warranted.
Results.
The results are presented in a cohesive and clear manner. A couple comments:
- How do the underrepresented groups tend to score on these types of outcome measures? This type of information would be useful to frame the current results.
- You mention that there are no real differences between the two groups (rightfully based on statistical analyses, but not based on the methods used); however, looking at the confidence intervals given, some of the results seem due to high variability within the actual measures. Notably both % of sleep problems and % of abnormal cortisol show very large 95% CIs. Given the n’s these seem abnormally large. I am particularly concerned because while the CI includes 1 in each case, the lower end is much closer to 1 than the upper, suggesting a potential trend towards long-term effects that is not being addressed in the current research (both for sleep problems and chronic stress in the intervention group). What were the effects of the control variables? What were the results without them? (I realize that some of the other large CIs are due to control variables even though the means are nearly identical, so I acknowledge these large CIs, but am not concerned by them.)
Discussion.
Overall, this is the weakest section. Your results, given the methods and analyses, yield no such conclusions while the limitations are overlooked in the discussion. The authors did not provide evidence that there were no long-term effects. In fact, given the use of intention-to-treat for a treatment that had a 58% agreement rate and randomized allotment, the authors can make very few conclusions at all. Add to that the use of measures that do not objectively measure the constructs of interest – sleep problems, child well-being, parent-child relationships, etc. – and the results become even less conclusive.
While I appreciate the authors’ attempt to address a question that is much needed, the fact remains that the current methodology and analyses do not address said question. If the authors are willing to completely redo their analyses, they can report on the effects of these interventions on the parental perception of certain variables as well as chronic stress (if this current method is a valid way to assess stress levels).
Recommendation.
I would not recommend this manuscript for publication in its current form. In fact, I question if the changes highlighted above would result in findings that have any real meaning given the methodology employed; however, I would be willing to entertain another draft with the appropriate analyses and results in place.
[1] Price AMH, Wake M, Okoumunne OC, Hiscock H. Five-year follow-up of harms and benefits of behavioral infant sleep intervention: randomized trial. Pediatrics 2012; DOI: 10.1542/peds.2011-3467.
Good review Tracy. One thing I found fascinating about this study was that, although it also found no long term benefits of sleep training – the sleep trained kids had just as many sleep problems as untrained kids at age six, the authors deliberately chose to underplay this finding and, as they usually do, media outlets followed suit.
One of the common statements about sleep-training, made in introductions to many journal articles, is that unresolved sleep problems are associated with problems later in childhood, including behaviour problems and sleep problems. Well, if you take the findings of this study as read, it (and another study they mention) disproves the notion that untreated nightwaking causes sleep problems later in childhood – something thousands of parents already know from personal experience.
Even the short term benefits of sleep training, which they cite from one of their earlier papers from this study, were unimpressive. Three months after the intervention 39 percent of the kids from the intervention group still had a sleep problem (as reported by parents) vs. 55% of the control group. In other words, the didn’t work for 2 out of 5 parents, and almost have the parents saw an improvement just by doing whatever they thought was best (which, may or may not have included some form of sleep training).
I blogged about this yesterday http://uncommonjohn.wordpress.com/
I read your analysis of the study with great interest — your statistical abilities are far above mine. Yet the first thing I thought when I read the headlines, was the fact that the control group seemed to be meaningless, as it was not clearly stated what those mothers were doing. Were they co-sleeping, skin-to-skin with their babies or were their babies placed in a cot (crib) in another room? The former would give meaning to the study, the latter makes it meaningless, as it would simply mean that both groups of babies were equally traumatised.
We do know from previous research (cited by Dr Nils Bergman of Kangaroo Mother Care — unfortunately I don’t have the reference) that if the baby is sleeping apart from the mother at night, though it may appear to be sleeping peacefully, the baby’s stress hormones remain elevated. He makes the point that babies are aware even during sleep whether they are safe near their mothers or separated from her.
Babies who were separated from their mothers in hospital following a traumatic birth, may never cry again after that first terrible separation, which has resulted in “one trial learning”, and the condition of “learned helplessness”. This can lead to the mother being unaware of the baby having any night-time needs for physical closeness, or the fact that the baby has concluded that it has been abandoned, causing it to dissociate and consider it futile to later signal its distress by crying. (I am describing my own personal story, which has caused me to spend decades of my adult life in psychotherapy, recovering from this trauma).
Synchronously, the day the study you analyse came out, an article went up on my website…
http://www.parenting-with-love.com/sleep-training-torturing-baby/
This article, which is anecdotal, but of great interest, was written by a friend of mine in 2008. The baby who she “ferberised” at nine months of age (a critical age for separation anxiety) is now twelve years old, and still suffers from serious sleep problems. Time has brought no improvement, and I doubt if it will unless the child undergoes the kind of therapy where the trauma is relived exactly as it happened, but with the containment of an enlightened witness (therapist) present to make the experience integrative, rather than retraumatising.
The current trend in psychotherapy that deals with the phenomena of the “hard wiring” of the brain in infancy is best represented by researchers like Allan Schore of UCLA, and Jaak Panksepp of Chicago — I expect you are familiar with their work, especially Panksepp’s latest conclusions regarding therapy and whether or not these very early traumas can be healed, and if so, to what degree.
Thank you for a wonderful article, which I greatly enjoyed reading, and which gives me hope that sanity may yet prevail if enough people keep speaking up.
You bring up some wonderful points Patricia! The concern over the control group is certainly warranted however in this type of research we presume that we’re comparing interventions with standard behaviour. So asking them to do nothing isn’t fair, etc. That’s why I didn’t bring it up; however, it would be a concern if the nurses were giving information to the control group that was similar to the intervention group. But it’s a flaw in research designs more generally that should be discussed in the limitations section (part of the discussion)!
The idea of sleep problems from training is gaining traction though there doesn’t seem to be much evidence for it. It really seems that some kids have persistent problems regardless of what parents do. Sleep training may exacerbate these problems, a legitimate concern, but by and large, what we know is that most kids by 2 will grow out of any sleep disturbances. Those that don’t may require some help (not sleep training), but earlier work won’t have helped them either 🙂
Hi Tracy, Thanks for your warm response. What does concern me is that in the control group, the mothers did “whatever”. Around the world, most of this is pretty traumatizing for babies. The statistical norm (average point) is not necessarily the desirable norm, in that it does not tell us what should be normal behaviour for mothers and babies, and therefore non traumatic for babies.
Nils Bergman (Kangaroo Mother Care doctor here in South Africa) who is doing all the number crunching says this:
http://home.mweb.co.za/to/torngren/eng-berg.html
I think (hope) that when these things are better understood, we will see the end of CIO and sleep-training.
You can find some of the studies Nils Bergman bases his conclusions on at http://www.kangaroomothercare.com
I completely agree – I’m just pointing out that for the sake of the study, the concern is with interventions versus “whatever”. I’d personally like to see research on the benefits of utilizing caring or attachment methods. That’s the real question for me 🙂
Btw, I’m well aware of Nils Bergman’s work 🙂 I have a piece covering a bit of Kangaroo Care somewhere on here!!!
Thanks for responding Tracy, It’s nice to meet someone who understands what I am talking about (and unusual!) Kind regards.
FYI these children were all 8-10 months of age. The stress hormone that Bergman cites should have regulated by then and thus shouldn’t be elevated in sleep. They would have “adapted” to sleeping alone if that were the case.
Hi Tracy, Thanks for creating a place to discuss! My understanding is that sleeping close to the mother is a biological need, and when children leave the parental bed of their own accord, it is somewhere around four to six years of age. At eight to ten months of age the baby’s brain has not finished growing (growth slows down, though it doesn’t stop, at about twelves months, If the baby has not had a positive night-time relationship with the parent, then as the pruning of the brain cells occurs (thousands of neuron die off) in the baby has been made to sleep alone, the synapses dealing with pleasure will not have connected up around that issue, and the pathways carrying the experience of loneliness, anxiety and emotional pain will connect up and become hard wired. Thus the potential pleasure pathways in the brain will have died off, and the pathways carrying the experience of loneliness, anxiety and emotional pain will be the ones that become hard wired.
I recently discovered this interview with Naomi Altort (she covers a number of topics) but night-time parenting is covered in some detail, with fresh and new insights that I had not considered before — i.e. the child sleeps with the parent, and no sleeping place or room is created elsewhere in the house until the child says, “I want to move out and sleep alone now.” (A lot of other good stuff on this video about how children learn — I think you might find it of interest). She has produced three children who are highly gifted and one child prodigy. http://vimeo.com/48792031
One problem is the many nuisance variables in studying something like this, as there are numerous issues that determine how a baby will grow up with a healthy brain, and sleep is just one of them. A warn and nurturing mother, whose baby sleeps separately, but is warmly nurtured in the day time and is responded to in the night, is difficult to compare with a baby where there is a difficult birth, a cold, unfeeling mother, and conflict between the parents which existed through the pregnancy and beyond, even if the baby is allowed into the parental bad. We are comparing applies and pears. I would like to see a study where Attachment Parenting or Aware Parenting (which is what I am trained in) are compared with babies who are sleep-trained, because that is where the big difference are going to be seen. Co-sleeping and avoiding sleep training are just one of the elements in Attachment Parenting”, and as we know it is the “securely attached” baby who is doing to do well, rather than the anxiously, or ambivalently attached baby, though sleep training can contribute to the latter, especially the anxiety.
I believe it is a biological need, though to be fair, there’s no evidence suggesting that after infancy (again though, it’s like the paucity of research on longer-term breastfeeding, just because it isn’t there doesn’t mean it’s not a benefit!). Honestly though, we have no idea what areas of the brain (with respect to specific events) are developing at what point. We know generalized areas (like the prefrontal cortex that continues development until mid-twenties), but specific neural connections? Only with language do we have some inkling of how those are developing. But we do know the brain is highly plastic at that age so it would not be surprising.
But it is a very difficult thing to study for the precise reasons you state. There are too many variables to control and consider and it will take years of good research to tease these apart.
Because I have spent about four decades in Primal Therapy (both receiving and giving it in what is called the “primal buddy” situation) I have dealt with so many of my own childhood traumas. Also being with countless other people who are spontaneously dealing with theirs, gives us the ability to find out in retrospect what issues were traumatic to us as babies and children, since in primal therapy, nothing is ever suggested to the patient by the therapist or buddy. From this pace we know after the fact, what has traumatised a huge number of individuals — at least those who have been motivated to seek help (a self-selecting population).
However how to quantify this in a study is something I would not know how to do — do you have any ideas? Whenever we talk about it we are dismissed as being “anecdotal”. But the truth is that if you are walking along a path, and you meet 50 people coming the other way, all of whom say there is an injured dog that they passed about 100 metres behind, you start to believe that there probably is one, especially if you have seen it too. So if you know a way to get this information out there, I would be delighted.
I’m attaching a little bit of my own story, first written down about 12 years ago (not much has changed int the interim, except that I have gone much deeper into some of the early traumas and resolved some, but far from all). The depression resulting from being made to sleep alone from birth, and never touched much as a baby, is not yet resolved, though I am less depressed than I was, and am able to cope with long periods of isolation better than I could.
http://home.mweb.co.za/to/torngren/patparent.html
If you can think of a way to get this information verified and got out to parents, we could avoid so much unnecessary suffering. The problem I see is that any individuals who were not traumatised, have probably not elected to go into such a radical therapies, and not including them, would skew the outcome… The only solution I could think of would be to say that the majority of people involved in the “deep feeling therapies” (where we revisit childhood and babyhood) have experienced early deprivation as traumatic.
[…] Well, now, I’ve read the study and I’m not terribly impressed. Actually, I’ve read enough sketchy “peer-reviewed” studies to make me a lot less impressed with the term peer review than I used to be. And this study definitely had it’s shaky points – primarily that they don’t have very good data on what either group of parents (one got an educational session on sleep training, the others did what they wanted – which could have included sleep training for all we know). I won’t go into the details because other bloggers have already done an excellent job of finding the holes in the study. If you want details here are two examples by two top Canadian parenting bloggers PhD in Parenting and Evolutionary Parent. […]