The Deep Future Analytics CECL Study: Alternatives, Impacts, Accuracy, and Complexity, 16 Mar 2017

Joseph L. Breeden, Principal Investigator

This study was sponsored by Allied Solutions, NAFCU - National Association of Federal Credit Unions, and OnApproach. Any opinions expressed here are solely those of the author and may not represent the opinions of the sponsors.

Executive Summary

The new accounting rules for estimating loan loss reserves offer general guidelines and a list of possibilities, but no specific recommendations for how best to implement those rules. The present study uses a large mortgage dataset from Fannie Mae and Freddie Mac to test a range of models and options. The results quantify the pros and cons of these options.

Study Design

For the underlying models, the following were tested: time series correlations to macroeconomic data, roll rate models, vintage models, state transition models, and discrete time survival models. These models were assessed for accuracy, robustness to small data sizes, complexity, computation time, and procyclicality of lifetime loss estimates.

In all cases, scenarios were created with 24-month macroeconomic history followed by mean-reversion to long-run macroeconomic conditions. Undoubtedly, many practitioners will create two separate models, a near-term model with a macroeconomic scenario and a long-run through-the-cycle loss model. Using a single model with a mean-reverting macroeconomic scenario is preferable, because the active portfolio is used for the lifetime loss forecast rather than an average of past portfolios. It also avoids needed to validate two separate models.

The guidelines also mention the option of using a discounted cash flow approach. DCF is not a model so much as a system of equations for aggregation, since it requires estimates of default and attrition probabilities as estimated in the models tested here. Therefore, all model results were shown as direct loss aggregation, discounted loss aggregation, and DCF aggregation of cash flows simulated from the loss estimation models.

Results

The following results are intended to be used to assess trade-offs in CECL implementation details.

Foreseeable Future

Using mean-reverting scenarios here allowed the model to adapt to the current portfolio for the lifetime estimation rather than use an average over past portfolios, but at greater complexity. Conversely, it requires only one model rather than two. Even though most practitioners will use a through-the-cycle average default rate as the long-run model, we know from Basel II that these are actually models with their own complexities in estimation.

Accuracy

Projecting losses via time series models of default and pay-down rates produced an average 3-year cumulative error rate of 17-19%. In itself, that will raise concerns with validators, but the accuracy is unchanging relative to the amount of training data, which can be useful for very small or noisy data sets. Vintage models were consistently high performers in terms of accuracy with 1% to 3% error rates. Discrete time survival models and state transition models both perform well (6.5% to 7.5%), but not better than vintage models, showing that loan-level modeling does not guarantee more accuracy. Vintage, state transition, and survival models all had similar scaling properties versus size of training data. Roll rate models were consistently the worst performers at 15% to 20% error rates. Moving averages of historic loss rates are unsuited to lifetime loss forecasting at 60+% error rates. Overall, roll rate and historic average models should not be used for long-lived products.

Creating separate models by US state did not provide greater accuracy when compared to a single national model of the same portfolio. Geographic segmentation provides advantages in business application but not model accuracy.

The guidelines state that vintage modeling is not a requirement. If we assume that "vintage model" refers to any approach that adjusts credit risk and prepayment risk based upon the age of the loan, then the results show significant increases in accuracy for techniques incorporating this (vintage models, state-transition models, and discrete time survival models) as compared to those that do not include it (time series and roll rates).

Accuracy vs. Complexity

The loan-level models (state transition and survival) were by far the most complex in terms of numbers of coefficients and computational time. This complexity did not provide any increased accuracy relative to vintage models, but it does provide business value in account management, collections, pricing, and strategic planning.

The added complexity of roll rate models when compared to time series models provided little benefit other than the change to be more accurate for the first six months of the forecast. Vintage models were the overall winners in the accuracy versus complexity trade-off, so long as sufficient data exists for robust estimation.

Optional DCF

Starting from a lifetime loss forecast, using a time-value of money discounting of the projected monthly losses using the par rate on the mortgage results in a 20% to 30% decrease in the reserve amount. Estimating the principle and interest payments adjusted for the risk of default or prepayment from the loss model and then discounting with the par rate on the mortgage results in a 70% to 80% reduction in the loss reserve as compared to the original lifetime loss forecast.

Old vs. New Rules

The magnitude of the change from the old loan loss rules to CECL will depend strongly on the lifetime of the asset and the point in the economic cycle when the adoption occurs. For 30-year fixed mortgage, the average life of loan is about 5.5 years and the lifetime loss reserve will be 4 times a historic average approach with 24 month loss emergence period. If adoption had occurred just before the onset of the last recession, the adjustment would have been 10x. At the peak of the recession the change would have been 2x. Well into recovery they would have been at parity.

If the full discounted cash flow approach is compared to the previous approach, the overall increase is only 26% through-the-cycle, with wide variations through the economic cycle, both positive and negative.

Conclusion

By design, the new CECL rules provide a significant amount of flexibility in implementation. As seen from this study, even with a straightforward product like 30-year fixed rate conforming mortgages, the range of models listed in the CECL guidelines can produce a range of lifetime loss numbers that vary by a factor of 2. With the option of discounted cash flows, then the range of final answers would vary by a factor of 4.

Being able to choose options that will create such different answers will put the burden on lenders not only to choose the most appropriate models for their portfolios, but in doing so to also choose the level of loss via the models chosen, and to defend that choice to validators, auditors, and examiners.

Prev Blog Next