Over the past year there has been a lot of hoopla in the industry concerning possible changes to CSA scores. We thought it would be a good idea to answer the top questions to help put you in the know. Of course, as always, all of us here at CAB are happy to talk with you to give you further insight if you want to delve even deeper into the intricacies of IRT. So here are the answers to the top questions that you have asked:
Q1) I keep hearing about the new IRT CSA model. What is IRT?
IRT (Item Response Theory) is a complex statistical model whose application has been successfully used in the education and medical sectors. While it hasn’t been widely used in transportation, the FMCSA, at the recommendation of the National Academy of Sciences (NAS), is working to develop an IRT model to see if it will be more effective at targeting motor carriers for intervention in order to prevent and reduce crashes.
Q2) Why is IRT better than the current model?
Right now no one knows if it is better. It is important to note that the NAS found that the current SMS methodology has been effective at helping the FMCSA prioritize motor carriers for intervention. Many industry experts including predictive modelers, data scientists and actuaries have validated the predictive nature of the current SMS methodology and correlated it to loss history. The NAS, after evaluating concerns presented by interested stakeholders, recommended a pilot program to assess whether an IRT model would provide a more accurate basis for deciding when intervention is needed. It is possible that IRT can provide a more data driven approach to the scoring process.
Q3) What new data will the IRT model use to improve the scores?
The same critical data points that are currently used to drive the BASIC scores will continue to be the foundation of the IRT score. While an IRT model could theoretically incorporate additional data should it become available, for now the FMCSA is testing the new process using the same inspection, violation and crash data points, as well as the motor carrier self-reported VMT and fleet records they have always used.
Q4) Will there still be BASIC Scores?
Here’s where the IRT scoring process becomes interesting. Using the same underlying data and a new IRT process, the FMCSA will continue to calculate separate BASIC scores that focus on specific Behaviors. In addition to this, the new scoring process will also generate a Safety Culture score, based on the overall safety performance of the motor carrier. It will be used by the FMCSA to prioritize motor carriers for intervention.
Q5) When are the new scores coming out?
That’s an impossible question to answer with any degree of certainty. Simply put, IRT is a model with various possible implementations. The FMCSA has not yet determined whether it will even transition to IRT. It is still working through whether it can be effectively applied to the transportation sector, and if so, in what form. The FMCSA has only provided high level target dates which it acknowledges are subject to change. Right now it is in the midst of a small scale pilot program. The plan is to run a full scale of the model by April 2019. In June, 2019, assuming all of the preceding takes place on time, it will evaluate the results and the effectiveness of the model. Whether the tested model will demonstrate that it is an improvement, and therefore should be implemented permanently, has yet to be seen.
Q6) Will CAB be releasing an early version of the new score?
The FMCSA has not yet worked out all the intricate details of the new methodology. As it works through the small scale pilot stage, it will be calibrating the scoring process. Releasing a “guesstimated” score prematurely will only lead to confusion. While there are vendors on the market that are offering early access to IRT based scores, CAB will not be doing this until the FMCSA publishes its final guidance. At CAB we are, and always have been, committed to publishing scores which are in complete alignment with the FMCSA methodology. When the CSA scores were pulled from the public view, we continued to provide precise scores rather than offer ‘enhanced’ scores offered by other companies. While it is possible to use the inspection, violation and crash data together with IRT to create a score, without defined guidance, it’s no different than building a house out of lumber. Without a blueprint, two builders with a pile of lumber will likely build different houses. If and when the FMCSA decides on a new direction CAB will provide support to educate and help transition our customers to the new process.
Q7) I’m a geek. Give me the technical stuff!
The new scoring system has several interesting proposals:
a) In addition to the safety score there will also be a confidence score. This will help identify cases where the score is based on thin data and may not be as reliable.
b) Along with the BASIC scores there will be a new Safety Culture Score which is intended to measure the carrier’s overall safety practices.
c) There are currently 3 time weight multiples (0-6 months = 3 , 6-12 months = 2, 12-24 months = 1). It may increase to 24 distinct multiples. As the data included is from the past 24 months and the scores are generated once a month, this allows violations to age each time the motor carrier is scored.
d) Various additional factors are being assessed to see whether they can improve the analysis. For example Safety Culture Scores may be subject to modification depending upon the nature of the carrier’s operations, certain violations may be given greater weight if they occur during certain seasons.
So here you have it! The lowdown on the current status of CSA and the IRT model. Now you can rest easy as you head into the New Year. CAB has your back and will keep you up to date with any new changes.
Happy New Year!
The CAB Gang
