Forecasting Series: Scenario Forecasting
One of the major tenants of SensibleAI Forecast is the transparency that can be gleaned with insights into forecasted values. When we analyze a utilized model, for example, we can see individual features as well as the extent to which they impacted the end result of our target variable. But what if we want to understand what would happen in the absence, presence, or other modification of the features? How can we not only see it for a particular value (which could perhaps be done with mental calculation), but across the entire forecast period for all intersections?
This article will seek to accomplish two major objectives:
- Introduce the different types of features and events which can be scenario modeled.
- Inform best practices when implementing Scenario Modeling into engagements.
Scenario Forecasting in Features
Features
When applying features to Scenario Modeling, we can quite simply select it in the Data > Features step of Model Build. Of course, Scenario Modeling will only apply to those features which are committed.
We can think of feature sets in two categories as it relates to scenario modeling:
-
Binary Features: These features are identified with ones and zeroes, and indicate if the feature occurred (functionally similar to events). In a practical sense, there is not a lot of difference in the execution of these and how it applies to model results.
-
Integer/Decimal Features: These are the features that have a value which may be useful for machine learning on our target variable. When deciding to Scenario Model over integer or decimal values, it is important to have a strong understanding of what values would be good scenarios to create. For example, a scenario in which the federal funds rate was 20% is likely never going to happen and pointless to model on.
For any application of features in Scenario Modeling, it is important to have values for dates that go through your last forecast period (if you’re doing a Deployed Model Forecast). If not, you will have to input these values into each scenario.
Events
Events, as mentioned earlier, structurally are similar to features for the purpose of Scenario Modeling. One benefit of events is the ability to store a bunch of them in a single 2-column Excel upload, rather than worrying about which individual series, date, and value would be needed for a feature set.
Similar to features, if we want to implement an event in Scenario Modeling, it is best practice to have values for this event go past your last forecast period. The Scenario Modeling table will have a column for putting these values in regardless, but you can avoid worrying about inputting it for each scenario.
Best Practices in Scenario Modeling
When we decide to do Scenario Modeling, we open ourselves up to a limitless set of possibilities to forecast our trained model onto. With this great power, of course, comes some responsibility. Follow these general best practices to ensure our Scenario Modeling results are both accurate and fruitful for business purposes.
-
Create Comprehensive Scenarios: While it may be tempting to use every possible combination of events in Scenario Modeling, keep in mind how events are related. For example, if bank interest rate and the stock indices we are using are heavily correlated, it might not make sense to display a scenario in which one goes up and one goes down.
-
Mix Strong Indicators with Low Confidence Events: Include both high-certainty elements (based on strong trends) and areas of significant uncertainty to capture a broad spectrum of futures.
-
Identify High Impact Features: Scenario Modeling is most beneficial when we are considering possibilities that markedly change our results. Consider using these scenarios to create a “Cone of Plausibility” rather than predicting the future.
This is an interesting read about the cone of plausibility, for additional context: https://www.sciencedirect.com/science/article/pii/S0016328722000957
-
Collaborate With Stakeholders: Ultimately, Scenario Modeling is a way for the end users to 1) Have more confidence in the results, regardless of scenario and 2) Prepare for uncertainty in their business. It is paramount to keep them in the feedback loop of scenario creation and deployment.
-
Be Mindful of Naming: Since Scenario Modeling is based on forecast names, it is important to be diligent in naming conventions. A single, cohesive scenario should share the same name throughout. However, some scenarios may be best off changing their names if the purpose changes. Keep naming in mind for ease in deployment, FVA creation, and overlaying different projects.