Unlocking the Benefits of Forecasting – An Engineering Perspective

Patrick Hofmann, Manager, Data Science at Magnite

August 22, 2024 | 5 min read

Why is forecasting important? Where do we use it?

In today's ad environment, publishers require accurate forecasting tools to quickly determine if sufficient inventory exists to sell a targeted campaign. This fuels the ability to confidently transact on direct-sold campaigns by answering questions such as "How many impressions can I expect for Supply Partner X in New York on CTV Devices over the next 30 days?"

Magnite's SpringServe ad server provides forecasting tools for three primary metrics - impressions, requests, and revenue. Publishers can instantly view forecasts for their top supply and demand objects within SpringServe, as well as custom forecasts to determine what's likely to happen for filtered slices of existing inventory. Soon, publishers will also be able to forecast new potential campaigns with key targeting applied. We currently support 30-day forecasts but are working on optimizing our solutions to help publishers look as far as 180 days in the future.

How do we measure accuracy?

When building a forecasting system across disparate supply accounts, rigor and accuracy are crucial. At Magnite, we seek an accuracy measure that gives both a qualitatively desirable forecast as well as one that yields accurate volumes in aggregate. As such, there are a number of suitable metrics, which fall into three main categories.

Scale-dependent errors are measured on the same scale as the data, making it difficult to compare time series on different scales. The two most commonly used measures of this type are the Mean Absolute Error (MAE) - which equally weighs all data points - and Root Mean Squared Error (RMSE) - which aims to minimize extreme errors (at the expense of worse average performance).

Percentage errors have the advantage of being scale-independent and, thus, are frequently used to compare forecast performance across different data sets at different scales. However, measures based on percentage errors have the disadvantage of being infinite or undefined for any nil observation, as well as assuming a scale based on quantity. Furthermore, the most commonly used percentage error measure, Mean Absolute Percentage Error (MAPE), can be biased and tends to systematically select those forecasts whose predictions are too low.

Scaled errors (or relative error measures) are often useful when we wish to inter-compare forecasts of time series with very different data scales. This is true on our ad-serving platform across publishers, as some serve thousands of ad requests while others serve billions. For instance, Theil's U-Statistic is a relative accuracy measure that compares the forecasted results with the results of forecasting with minimal historical data. Mean Absolute Scaled Error (MASE) has become a more popular scaled error measure due to simpler computation that makes it better for comparing data at different scales since it's insensitive to outliers and easily explainable.

At Magnite, we balance these error metric trade-offs by using multiple measures rather than relying on a single statistic. This way, we have measures that can be widely communicated while being statistically robust.

How do we define a baseline for measuring progress?

In order to measure progress over time in the face of seasonality and inventory changes, we established a test suite, which is a fixed collection of inventory tags representative of our overall inventory for a fixed period of time. This allows us to benchmark how we're doing as we make model enhancements without worrying about whether the improvements are due to chance. To help resolve seasonal effects, we also select rolling forecast start dates across an entire year. In addition, we have dashboards to monitor our performance day-by-day to find and improve any divergences in performance.

What models are used to forecast time series?

There are numerous modeling approaches for forecasting time series that are important to consider given elements such as seasonality as a driver of variability in our data, where we see not only hourly and daily cycles but day-of-week, quarterly, and yearly cycles as well. Each method for forecasting time series data has its own trade-offs. Traditional statistical methods, such as auto-regressive and exponential smoothing models, are simpler to understand and explain but might not yield the most accurate results. More novel packages, such as Prophet or XGBoost, might do a great job handling seasonality but can yield inaccurate trends. While deep-learning-based algorithmic approaches such as TimesFM might yield an accurate forecast, they have no explainability or ways to tailor results to a specific domain.

At Magnite, we've struck a fine balance between an algorithm that is flexible and accurate while being computationally efficient and economical. Using a combination of traditional statistical models, domain-specific enhancements, and seasonality, our accuracy fuels client trust in our forecasts.

Forecasting is an inherently difficult problem without a one-size-fits-all solution. To provide accurate and timely forecasting to our customers, we continue to test, monitor, and fine-tune our forecasting algorithms for each specific application.

Tags: CTV, Forecasting, Measurement