Perhaps you've seen the headline that is making rounds and has investment managers elated – actively managed endowment funds outperform their peers. The conclusion comes thanks to a study by the Commonfund Institute (available here) that examined data from the NACUBO-Commonfund Study of Endowments Dataset (covering 2006-2013).
While the conclusion is an intriguing one that runs counter to conventional wisdom about active vs. passive investment approaches, I don't think the conclusions are supported by the methodology. To elaborate on this, I will first explain what the study's approach was and then consider its flaws.
The Commonfund study took the percent of each endowment fund's equity pool that was actively managed (we'll call that the amount of "activeness") and linked it with that endowment fund's annual return. The ultimate question is whether activeness and return are connected. To address this statistically, the study grouped the endowments into 25 different "activeness buckets" (those 0-4% active; those 4-8% active; etc.) and took the average endowment return of each bucket. It then performed a regression analysis that statistically examined whether there is a connection between being in a more active bucket and having a higher return. The results show a positive (and statistically significant) coefficient on the activeness term, reflecting that higher activeness does, on average, increase returns. Further, they showed that the level of activeness can explain 14% of all variability in returns (i.e., the regression r-squared was 0.14).
The evidence seems convincing, especially due to the apparent statistical rigor. The problem is this – regression analysis assumes (among other things) that each observation deserves equal weight since each is equally accurate; however, the way the data was put together ensures that assumption does not hold. Recall, each observation in the regression represents the average return in one "activeness bucket". However, most of those averages were comprised of very few observations (see their figure II on p. 6), while one of the averages represents almost 50% of all the data. Much like the fact that averages from political polls are more accurate the more people are surveyed, the same should be true of this data, meaning that one of the buckets should receive much more weight than the others. Failing to take this into account can lead to false conclusions.
Consider the data in the following figure as an example:
Clearly, the data on the horizontal (X) axis has no connection with that on the vertical (Y) axis (by design). This is confirmed by the r-squared value of zero.
Now, what if we decided to split the data into three equally-spaced buckets (much like the Commonfund study) – those with X values between 0 and 1, those between 1 and 2, and those between 2 and 3 – and took the average Y value for each bucket. Using this technique yields the following figure:
Note that the process not only creates the impression of a positive association between X and Y, but it also generates a high r-squared value, suggesting that 90% of variation in Y can be explained by X.
Of course, this is an extreme case, but it highlights the problem of converting the data to buckets, especially ones of unequal size. It should also be stressed that this problem is beyond being a theoretical possibility. In the Commonfund study, the point for the largest activeness bucket (the one representing nearly half of all the data) lies notably below the best fit line, suggesting that properly including all data will seriously undermine (even eliminate?) any positive association.
To address this concern, the Commonfund study should perform a regression analysis on all the data points without grouping them into buckets. Though there are sure to be concerns with this approach too, it would certainly be more consistent with the underlying assumptions of the statistical techniques employed and would also make the best use of the data. And, while the authors are at it, they should risk-adjust returns before conducting the analysis – after all, it is easy to generate higher (average) returns by taking on risk and/or leverage, the hard part is generating more returns without taking on more risk.
In short, I find the evidence presented thus far that supports active endowment management promoting greater returns as entirely unconvincing. That said, I look forward to seeing more evidence that could sway this view.