News

Should you wait for a cheaper flight? A gradient-boosting study of 94,000 fares

We trained gradient-boosted decision trees on three weeks of Wingman's fare checks. The model ranks coming price drops well (AUC 0.77–0.83), but no waiting policy beat buying at once.

Should you wait for a cheaper flight? A gradient-boosting study of 94,000 fares

Abstract

Should a traveller buy a fare now or wait for a lower one? We studied this question on Wingman's price log: 93,769 fare checks on 1,020 routes from 22 October to 12 November 2026. We set up two prediction tasks. The first task predicts whether the fare drops by 5 % or more within 3 days. The second task predicts how much a buyer saves by waiting. For both tasks, we trained gradient-boosted decision trees (LightGBM) and tested them on later days that the models never saw.

The drop model ranked drops well. Its ROC AUC was 0.77 to 0.83, against 0.67 to 0.71 for a baseline that uses only the days left before departure. But no policy built on the models saved money against buying at once. For flights more than a week away, the policies came out about even. So the Wingman app keeps its rule-based advice for now.

1. Data

Each time Wingman checks a watched trip, it writes the cheapest result of each search to a price log. A row holds:

The log has no personal data.

We call the checks of one route, flight date, stop limit, travellers, cabin and currency a series. The log had 6,573 series. The median series has 6 checks, 10 % of the series have 20 or more, and the densest has 840. How often Wingman checks a series depends on the plan of the account that watches it and on the days left before departure. 93.5 % of the prices are in euros.

Densely checked series would fill the data set with near copies. So we kept at most one sample per series every 3 hours. This gave 54,896 samples.

2. The prediction tasks

Take a sample at time t with price p, and a horizon H of 3 days (7 days in a second test). Its window starts after t. The window ends at t + H or at the end of the flight date, whichever comes first.

Drop label. The label y is 1 when two checks in a row in the window both show a price of 0.95 p or less. Else y is 0. We ask for two checks so that a one-off blip in the results doesn't count as a drop.

Saving. We also measure what waiting gives a buyer: g = (p − p_wait) / p. If the window has a drop, p_wait is the higher price of the cheapest such pair of checks. A buyer who checks at either of the two times pays at most that price. Else, p_wait is the last price in the window. A positive g means that waiting saved money.

Censoring. We keep a sample only when its series has a check within 12 hours of the end of the window. Otherwise, the log doesn't show what happened. This left 35,253 samples with a 3-day label and 21,033 with a 7-day label.

3. Features

The models see 34 features. Each feature uses only checks at or before t, so no information from the future leaks in.

Table 1. The features. "÷" means "divided by".

GroupCountFeatures
Time6days left before departure, and its logarithm; weekday and month of the flight; hour and weekday of the check
Trip10distance between the airports (log); price in euros (log); travellers; cabin; checked bags; stop limit of the search; stops of the cheapest flight; direct flight listed; direct price ÷ cheapest price; flights listed (log)
Series: checks3earlier checks; age of the series; checks per day over the last 3 days
Series: price level5price ÷ the minimum, maximum, median and first price of the series; price at the minimum
Series: movement7price ÷ the price at the previous check, 24 hours ago and 72 hours ago; hours since the last change; over the last 7 days: share of checks with a change, price range ÷ median, number of 5 % steps down
Nearby dates3other flight dates within 7 days on the same route, checked in the last 3 days: their number; price ÷ their median; price ÷ their minimum

4. The model

4.1 Gradient-boosted decision trees

A decision tree sorts samples with yes-or-no questions about one feature at a time, such as "are fewer than 9 days left?". Each leaf at the end holds a number. One small tree is a weak predictor. Gradient boosting adds many of them:

F(x) = f₁(x) + f₂(x) + … + f_M(x)

Each new tree f_m fits the negative gradient of the loss of the trees before it. So each tree corrects what the earlier trees still get wrong. A learning rate η scales each tree down, so that no single tree decides much (Friedman, 2001). For the drop label, the loss is the log loss, and the predicted probability is σ(F(x)) = 1 / (1 + e^(−F(x))).

We used LightGBM (Ke et al., 2017). It sorts the values of each feature into at most 255 bins before it looks for splits, which makes training fast. It grows each tree leaf by leaf: at each step, it splits the leaf that reduces the loss most. At each split, it also learns where samples with a missing value go. This matters here: a new series has no price from 72 hours ago, and the model can use that fact directly.

Trees suit this data. The features have different scales, some values are missing, and the effects interact. For example, a 5 % rise means one thing 3 days before departure and another thing 3 months before. A tree model handles all three without extra work. A neural network usually needs much more data than three weeks of prices.

4.2 Settings

We chose the settings before the first test and didn't tune them on the test data:

Densely checked series give many samples that say nearly the same thing. So each sample gets the weight 1/√n, where n is the number of samples from its series.

4.3 Models of the saving

The drop model answers "will it drop?", but a buyer cares about money. So we trained two more LightGBM models on the saving g, with the same features and settings:

A third policy uses the drop model directly: wait when the probability of a drop is 0.5 or more.

4.4 Baselines

5. Evaluation

Time split. A random split puts checks from the same days in the training set and in the test set. The test then looks better than real use. So we split by time, with a gap. For a cut-off date c, the training set has only the samples whose whole window ends before c (t + H ≤ c). The test set has the samples from c on. So no training label uses a price from after c.

For H = 3 days, we used three cut-offs: 1 November (with the test until 4 November), 3 November and 5 November. The test sets of the last two overlap. For H = 7 days, the log allowed one cut-off: 1 November.

Metrics. ROC AUC is the probability that a random sample with a drop gets a higher score than a random sample without one. 0.5 is chance, and 1 is perfect. The Brier score is the mean squared error of the predicted probability. The Brier skill score compares it with the Brier score of the base rate: 1 − BS / BS_base (Brier, 1950). A positive value means better probabilities than the base rate.

Intervals. Samples from one route aren't independent. So we resampled whole routes (a cluster bootstrap, 200 to 300 resamples) and give 90 % intervals.

Decision value. For each policy, we took the mean saving g per decision over the test set, with g = 0 when the policy buys at once. So buying at once scores 0. A policy with perfect foresight waits only when g is above 0, and gives the upper limit.

6. Results

6.1 Ranking drops

Table 2. ROC AUC for a drop within 3 days, with 90 % intervals.

Cut-offDays to departureLogistic regressionLightGBM
1 Nov (to 4 Nov)0.71 (0.65–0.76)0.77 (0.72–0.83)0.83 (0.80–0.86)
3 Nov0.70 (0.68–0.72)0.71 (0.68–0.73)0.77 (0.75–0.79)
5 Nov0.67 (0.65–0.70)0.73 (0.70–0.76)0.77 (0.75–0.80)

The three test sets had 7,489, 18,558 and 13,586 samples, from 308, 596 and 588 routes. The share of drops was 7.6 to 9.0 % in the training sets and 11.6 to 16.3 % in the test sets.

LightGBM ranked drops best at every cut-off. At 7 days, with only 2,713 training samples, its AUC fell to 0.71 (0.67–0.74), and the AUC of logistic regression fell to 0.54.

In plain terms: show the model one sample with a coming drop and one without. It picks the right one about 8 times in 10.

6.2 Calibration

A model can rank well and still give wrong probabilities. The Brier skill score of LightGBM was only 1.4 to 4.7 %. The days-to-departure baseline did better, with 6.3 to 16.0 %. Table 3 shows why.

Table 3. The drop probability that LightGBM predicted, against the real share of drops, by fifths of the test set (cut-off 3 November).

Fifth of the test setMean predictionReal share of drops
1 (lowest)0.5 %2.0 %
22.1 %4.8 %
37.6 %12.5 %
424.2 %21.4 %
5 (highest)62.9 %35.7 %

The predictions are too low at the bottom and too high at the top. The share of drops almost doubled from the training days to the test days (8.2 % to 15.3 %), so the base level moved. At the top, the model is too sure of itself. Calibration on recent data, for example with isotonic regression, could correct part of this. But calibration needs its own slice of recent data, and three weeks don't give enough.

6.3 What the model uses

The split gain of the drop model shows which features it uses most:

Two things stand out. First, the recent trend ranks low: the change since the previous check, or since 24 or 72 hours ago. A fall or a rise in the last days says little about the next days. Second, the number of earlier checks ranks high. A series that Wingman checks often has more chances to show two low checks in a row. So part of what the model learns is how often Wingman checks a flight, not only how its price behaves.

6.4 Decision value

Table 4. Mean saving per decision against buying at once: 3-day horizon, all test samples, 90 % intervals. A negative value means that the policy paid more.

Policy1 Nov (to 4 Nov)3 Nov5 Nov
Buy at once000
Always wait−3.5 % (−5.3 to −2.1)−5.7 % (−7.2 to −4.0)−5.3 % (−7.1 to −3.8)
Drop model, wait if P ≥ 0.5−1.5 % (−2.9 to −0.4)−2.7 % (−3.9 to −1.7)−0.6 % (−1.2 to −0.1)
Saving regression−1.9 % (−3.4 to −0.5)−0.6 % (−1.2 to −0.1)−0.7 % (−1.3 to −0.1)
Cost-sensitive classifier−0.7 % (−2.1 to +0.4)−2.1 % (−3.2 to −1.0)−0.3 % (−0.8 to +0.2)
Perfect foresight+2.8 %+2.9 %+3.0 %

No policy beat buying at once. The drop model was right in 41 to 47 % of its "wait" decisions, but a wrong wait cost more than a right wait saved. Waiting 7 days was worse: always waiting cost 14.6 % on average, and the drop-model policy cost 5.2 %.

Most of the loss comes from the last week before departure. Figure 1, at the top of this post, shows how often a price moved by 5 % or more within 3 days. In the last 7 days, the price was 5 % higher three days later in 52 % of the samples, and it dropped in only 29 %. From 8 to 60 days out, drops and rises came about equally often, each in 24 to 43 % of the samples. More than 60 days out, only 4 to 5 % of the prices dropped within 3 days.

So we ran the test again without the last week: only flights more than 7 days away. Always waiting then cost 0.6 to 1.5 % at 3 days and 1.9 to 2.3 % at 7 days. Over the 5 test periods and the 3 policies, the model policies landed between −1.3 % and +0.5 % per decision.

One of these 15 intervals lay above zero: the cost-sensitive classifier at 3 days and cut-off 1 November, with +0.5 % (+0.2 to +0.9). With 15 comparisons, one such interval is likely by chance. The same policy also lost money at the other two 3-day cut-offs. We read the result as "about even", and that isn't enough to tell anyone to wait.

7. Limitations

8. Related work

Etzioni et al. (2003) asked the same question with fares that they collected for two routes in the United States. They found that a learned policy could save money against buying at once. Groves and Gini (2013) studied more routes and chose the features automatically. Both studies followed few routes over a longer time. Our data is the other way round: many routes over a short time. We think that the short time is the main reason why our models don't pay yet.

9. Conclusion and next steps

The drop model ranks well, but it doesn't yet give a buyer a way to save money. So the Wingman app keeps its buy-or-wait rules for now. The rules use the usual price for the route, the history of the watch and the days left before departure.

Next steps:

For travellers, the data supports three simple rules:

References