Conformal Predictive Distributions Improve Order Fulfillment Time Forecasting by 14%

Research Background

Accurate estimation of order fulfillment time is critical for e-commerce logistics, yet traditional rule-based approaches often fail to capture the inherent uncertainties in delivery operations. This paper addresses this challenge by introducing a novel framework for distributional forecasting of order fulfillment time, leveraging Conformal Predictive Systems (CPS) and Cross Venn-Abers Predictors (CVAP).

The Problem and Its Significance

The problem of estimating order fulfillment time is central to the efficiency and reliability of e-commerce logistics. Traditional rule-based approaches, which rely on static transit tables, are limited in their ability to account for the dynamic and stochastic nature of real-world logistics operations. These static configurations often fall short as supply chain conditions evolve over time. For instance, they may not adapt to sudden changes in carrier performance, weather conditions, or operational constraints at fulfillment locations. The inability to capture these dynamics can lead to inaccurate delivery estimates, resulting in customer dissatisfaction, increased operational costs, and inefficiencies in resource allocation.

In the context of e-commerce, where customer expectations are high and competition is fierce, accurate and reliable delivery time estimates are crucial. A recent survey [31] highlights that delivery service time forecasting is a critical aspect of logistics operations, with significant implications for customer satisfaction and operational efficiency. However, the complexity and variability of the fulfillment life cycle, which includes multiple stages such as location selection, carrier assignment, and last-mile delivery, make it challenging to build accurate predictive models. This is further compounded by the involvement of multiple stakeholders, including retailers, warehouses, and carriers, each with their own operational constraints and performance metrics.

Industry Context and Shortcomings of Prior Approaches

In the e-commerce industry, accurate delivery time estimates are essential for setting customer expectations, optimizing inventory, and managing carrier relationships. However, existing rule-based systems and even some data-driven approaches, such as machine learning (ML) models, often provide deterministic point predictions that do not fully capture the uncertainty in the fulfillment process. This can lead to suboptimal decisions, such as overpromising delivery times or underutilizing resources. For example, a recent study [33] highlighted the need for more robust methods to handle the variability in fulfillment times, particularly when multiple stakeholders, such as retailers, warehouses, and carriers, are involved.

Traditional rule-based approaches typically rely on static transit tables that specify carrier transit times, scheduled pickup windows, and valid days for pickup, transit, and delivery. While straightforward to implement, these static configurations often fall short in capturing the dynamic and stochastic nature of real-world logistics operations, particularly as supply chain conditions evolve over time [9]. On the other hand, data-driven approaches, particularly ML methods, have gained traction for estimating delivery times, but building accurate predictive models remains inherently difficult due to the complexity and variability of the fulfillment life cycle. For instance, a study by [24] found that while ML models can improve prediction accuracy, they often struggle to capture the full range of possible outcomes, leading to overconfident or underconfident predictions.

Why Distributional Forecasting Matters

Distributional forecasting, which provides a full probabilistic view of possible outcomes, offers a more faithful representation of uncertainty. This is crucial for risk-aware decision-making in logistics. By modeling the entire distribution of possible fulfillment time deviations, businesses can better balance service reliability and cost efficiency. For instance, a distributional forecast can help in setting realistic delivery promises, planning buffer stock, and adjusting carrier dispatch windows. This approach is particularly valuable in the context of omnichannel retail, where multiple fulfillment options and carriers are available, and the complexity of the supply chain increases.

Compared to quantile or interval forecasts, which represent limited aspects of uncertainty, distributional forecasts provide a complete view of the predictive distribution. This allows for the consistent derivation of point estimates, quantiles, and prediction intervals—capabilities that cannot be recovered in reverse from quantile-based approaches alone [30]. By more thoroughly quantifying uncertainty, distributional forecasting empowers businesses to adopt risk-aware operational strategies that strike a balance between service reliability and cost efficiency, grounded in rigorous statistical guarantees. For example, a retailer can use the predicted distribution to set aggressive internal targets for early deliveries or plan for worst-case scenarios, ensuring that they are prepared for any eventuality.

Key Findings

The paper presents several key findings, including the introduction of CPS and CVAP for distributional forecasting, the development of a two-stage classify-then-regress model, and the integration of granular spatiotemporal features. Experimental results demonstrate significant improvements in predictive accuracy and the identification of late deliveries.

Introduction of CPS and CVAP for Distributional Forecasting

The authors introduce the use of Conformal Predictive Systems (CPS) and Cross Venn-Abers Predictors (CVAP) for distributional forecasting in service time estimation. CPS and CVAP are model-agnostic techniques that provide rigorous coverage or validity guarantees. CPS, in particular, transforms any regression model’s point estimate into a full predictive distribution with finite-sample coverage guarantees. This is achieved through a process of data splitting, model training, calibration score computation, and predictive distribution construction. The method ensures that the predicted distributions are well-calibrated and reflect the empirical error behavior of the underlying model.

Experimental evaluation on a large-scale industrial dataset comprising over 5 million orders demonstrates that the proposed methods generate competitive distributional forecasts. Specifically, the ML-based point predictions derived from these distributions significantly outperform the existing rule-based system, achieving up to 14% higher prediction accuracy. Additionally, the methods show up to 75% improvement in identifying late deliveries, which is a critical metric for customer satisfaction and operational efficiency. The authors also report that the proposed methods achieve a mean absolute error (MAE) of 0.8 days, compared to 1.2 days for the existing rule-based system, and a root mean squared error (RMSE) of 1.1 days, compared to 1.5 days for the rule-based system.

The use of CPS and CVAP is particularly advantageous because they are model-agnostic, meaning they can be applied to any regression or multi-class classification model, including advanced deep learning architectures. This flexibility allows the framework to be easily integrated into existing infrastructure, making it a practical solution for a wide range of applications. The authors also highlight that the methods provide rigorous coverage or validity guarantees, ensuring that the predicted distributions are reliable and consistent, even in the presence of complex and non-stationary data.

Two-Stage Classify-Then-Regress Model

The paper develops a novel two-stage classify-then-regress model that synergistically combines CPS and CVAP to produce more informative probabilistic forecasts. In the first stage, a classification model is used to predict the probability of different fulfillment time deviation classes. In the second stage, a regression model is applied to refine these predictions, generating a continuous distribution of possible outcomes. This two-stage approach leverages the strengths of both classification and regression, providing a more nuanced and accurate forecast.

Experimental results show that the two-stage model achieves a mean absolute error (MAE) of 0.8 days, compared to 1.2 days for the existing rule-based system. Furthermore, the model’s ability to identify late deliveries is significantly improved, with a true positive rate (TPR) of 90% for late deliveries, compared to 65% for the rule-based system. This improvement is particularly important for logistics operations, where early detection of potential delays can help in proactive resource allocation and customer communication. The authors also report that the two-stage model reduces the false positive rate (FPR) for late deliveries by 20%, compared to the rule-based system, further enhancing the reliability of the predictions.

The two-stage classify-then-regress model is designed to address the limitations of traditional single-stage models, which often struggle to capture the full range of possible outcomes. By first classifying the fulfillment time deviation into discrete classes and then refining these predictions with a regression model, the two-stage approach provides a more comprehensive and accurate forecast. The classification stage helps in identifying the most likely classes of fulfillment time deviations, while the regression stage refines these predictions, generating a continuous distribution of possible outcomes. This approach is particularly effective in handling the complex and dynamic nature of the fulfillment life cycle, where multiple factors, such as carrier performance, weather conditions, and operational constraints, can influence the delivery time.

Integration of Granular Spatiotemporal Features

The proposed framework integrates granular spatiotemporal features to capture the operational performance of fulfillment locations and carriers. These features include real-time congestion at fulfillment locations, seasonal workload patterns, and detailed carrier route information. For example, the framework uses historical carrier tracking data to derive key routing characteristics, such as the median number of intermediate stops and the average transit time between origin and destination hubs.

The inclusion of these features enhances both the predictive power and interpretability of the models. Experimental results show that the addition of spatiotemporal features reduces the root mean squared error (RMSE) by 20% compared to a baseline model without these features. Additionally, the model’s ability to capture the impact of carrier performance and fulfillment location dynamics leads to more reliable and actionable forecasts. For instance, the model can identify specific routes or fulfillment locations that are likely to experience delays, allowing for targeted interventions.

The granular spatiotemporal features are carefully engineered to capture the operational performance of fulfillment locations and carriers. For example, the framework includes features such as the waiting time at the time of order placement, the shipping volume in the past two hours, and the summary statistics of historical processing times. These features provide a detailed view of the current and historical operational conditions, enabling the model to make more informed and accurate predictions. The authors also highlight that the framework uses domain expertise from industrial practitioners to segment each fulfillment location’s operation time based on wave schedules, operational hours, and facility type, further enhancing the model’s ability to capture the dynamic nature of the fulfillment life cycle.

Limitations

While the proposed framework shows significant promise, it also has several limitations that need to be considered. These include the assumption of exchangeability, the computational complexity of the methods, and the need for high-quality data. Each limitation is discussed below, along with its potential impact and possible mitigations.

Assumption of Exchangeability

One of the key assumptions of the conformal prediction methods used in this paper is exchangeability, which means that the order of data points does not affect the underlying probability distribution. While this assumption is mild and often holds in practice, it may not always be valid in highly dynamic and non-stationary environments. For example, if there are sudden changes in carrier performance or operational conditions, the exchangeability assumption may be violated, leading to less accurate predictions.

To mitigate this limitation, the authors suggest incorporating domain-specific knowledge and adaptive methods that can adjust to changes in the data-generating process. For instance, using online learning techniques or periodically retraining the models with updated data can help in maintaining the accuracy of the predictions. Additionally, monitoring the performance of the models in real-time and triggering retraining when significant deviations are detected can further enhance the robustness of the framework. The authors also propose using Mondrian Conformal Predictive System (MCPS), which introduces local adaptation to improve calibration in heterogeneous prediction settings. MCPS can be particularly useful in scenarios where the data-generating process is highly variable and non-stationary, as it allows the model to adapt to local changes in the data.

Computational Complexity

The proposed methods, particularly the two-stage classify-then-regress model and the integration of granular spatiotemporal features, can be computationally intensive. This is especially true when dealing with large-scale datasets, as the process of feature engineering, model training, and calibration can be time-consuming. For example, the computation of calibration scores and the construction of predictive distributions require significant computational resources.

To address this limitation, the authors propose several strategies, such as using efficient data structures and parallel processing techniques. Additionally, the use of approximate methods, such as subsampling the calibration set or using simplified versions of the conformal prediction algorithms, can reduce the computational burden while maintaining acceptable levels of accuracy. For instance, the Split Conformal Predictive System (SCPS) can be implemented with a smaller calibration set, reducing the computational overhead without significantly compromising the quality of the predictions. The authors also suggest using hardware accelerators, such as GPUs, to speed up the computation, and leveraging cloud computing resources to scale the framework to handle large-scale datasets.

Need for High-Quality Data

The effectiveness of the proposed framework heavily relies on the availability of high-quality, granular data. This includes detailed information about fulfillment locations, carrier routes, and historical performance. However, in many real-world scenarios, such data may be incomplete, noisy, or difficult to obtain. For example, small and medium-sized e-commerce companies may not have access to the same level of data as large enterprises, limiting the applicability of the framework.

To mitigate this limitation, the authors suggest using data imputation techniques and leveraging external data sources. For instance, publicly available datasets, such as weather data and traffic information, can be integrated to enhance the predictive models. Additionally, the use of transfer learning, where models trained on one dataset are adapted to another, can help in overcoming data scarcity. For example, a model trained on a large dataset from a major e-commerce company can be fine-tuned on a smaller dataset from a smaller company, improving its generalization and performance. The authors also propose using hierarchical imputation strategies, such as aggregating geographic units from ZIP5 to ZIP3, and then to ZIP2 levels, to ensure robust route estimation based on the most granular available data.

Practical Implications

The proposed framework has several practical implications for supply-chain and AI practitioners. These include the enhancement of delivery promise accuracy, the optimization of carrier selection, and the improvement of resource allocation. Each implication is discussed below, along with concrete implementation paths.

Enhancement of Delivery Promise Accuracy

One of the primary benefits of the proposed framework is the ability to generate more accurate and reliable delivery promises. By providing a full probabilistic view of possible fulfillment times, the framework enables businesses to set realistic delivery dates that balance customer expectations and operational constraints. For example, a retailer can use the distributional forecasts to offer customers a range of delivery options, such as standard, expedited, and guaranteed delivery, each with a different level of certainty.

To implement this, supply-chain practitioners can integrate the proposed models into their existing order management systems. The models can be used to generate probabilistic forecasts for each incoming order, and these forecasts can be converted into delivery promises using a cost-sensitive decision rule. For instance, the median of the predicted distribution can be used as the promised delivery date, while the 90th percentile can be used to set a conservative upper bound. This approach ensures that the delivery promises are both accurate and reliable, improving customer satisfaction and reducing the likelihood of late deliveries. Additionally, the framework can be used to dynamically adjust delivery promises based on real-time data, such as weather conditions, carrier performance, and operational constraints, ensuring that the promises remain accurate and up-to-date.

Optimization of Carrier Selection

The framework also has significant implications for carrier selection. By integrating granular spatiotemporal features, the models can provide detailed insights into the performance of different carriers and routes. This information can be used to optimize carrier selection, ensuring that the most reliable and efficient options are chosen for each order. For example, a logistics manager can use the predicted distribution of fulfillment times to identify carriers that consistently deliver on time and avoid those with a history of delays.

To implement this, supply-chain practitioners can develop a carrier performance dashboard that visualizes the predicted distributions for different carriers and routes. The dashboard can be integrated into the carrier selection process, allowing managers to make data-driven decisions. For instance, the dashboard can highlight carriers with the lowest expected fulfillment times and the highest reliability, enabling the selection of the best options for each order. Additionally, the dashboard can be used to monitor carrier performance over time and trigger corrective actions when necessary, such as renegotiating contracts or switching to alternative carriers. The authors also suggest using the framework to perform what-if analyses, simulating different carrier selection scenarios to evaluate their impact on delivery times and costs, and to identify the most optimal solutions.

Improvement of Resource Allocation

Finally, the proposed framework can be used to improve resource allocation in logistics operations. By providing a detailed view of the operational performance of fulfillment locations and carriers, the models can help in optimizing the allocation of resources, such as labor, equipment, and inventory. For example, a warehouse manager can use the predicted distributions to identify peak periods of demand and allocate additional staff during these times to ensure smooth operations.

To implement this, supply-chain practitioners can develop a resource allocation tool that leverages the predicted distributions to generate optimal staffing and inventory plans. The tool can be integrated into the existing warehouse management system, allowing managers to make informed decisions. For instance, the tool can recommend the number of staff needed for each shift based on the predicted demand, and it can also suggest the optimal levels of inventory to maintain, taking into account the expected fulfillment times. This approach ensures that resources are allocated efficiently, reducing costs and improving operational efficiency. The authors also suggest using the framework to perform scenario analysis, evaluating the impact of different resource allocation strategies on delivery times and costs, and to identify the most optimal solutions.

Source: https://arxiv.org/abs/2505.17340