XGBoost: its loss, its split, and every hyperparameter

Audience

You know what a decision tree predicts, you can read Python, and you have seen a derivative as the slope of a function. The lesson does not assume that you know gradient boosting, Taylor expansions, or how any tree library works. It explains each idea before it uses it.

Version

Every number comes from xgboost 3.2.0 on Python 3.13, with CPU only. A number on this page either comes from that library or is marked as read from its source or docs. Anything inferred is marked inference.

Scope

This page is complete on its own. Its check script recomputes every number on it, and it lives in assets/xgboost-lesson/verify_xgboost_lesson.py.

After this lesson, you can

• derive the leaf weight and the gain from the objective;
• compute g and h for a loss, and say why that is all that the tree builder needs;
• grow a tree by hand and predict what get_dump shows;
• say which term of which formula any parameter changes;
• predict how gamma, lambda, alpha and min_child_weight differ.

How to read this page. Sections 1 to 8 are one argument, so read them in order. Sections 9 and 10 cover the machinery around the formulas, and section 11 puts every step of training and prediction in order. Section 12 is a reference you can search. Figures in sections 1 to 11 and 13 change when you use their controls. The practice boxes check each answer, and a box is mastered after two right answers in a row. Closed-book checkpoint quizzes follow sections 4, 6, 8, 10 and 11.

1The running rows

Every worked example uses these eight rows. Each row is a student. There are two features, hours studied and hours slept. There are two targets: an exam score for regression, and a pass or fail for classification.

The scores have mean 57.5. A model with no information would predict 57.5 for everyone. Its root mean squared error (RMSE) on these rows is 17.5. The goal is to do better with small trees.

Four words are used all through the lesson. A leaf is an end node of a tree, and it holds one number. That number is the leaf's weight, written w. The margin is the model's raw output, before any conversion such as turning a score into a probability. The loss l measures how wrong one prediction is for one row.

2A boosted model is a sum of trees

A boosted model does not grow one big tree. It starts from one number, $F_0$, and adds trees one at a time. Each tree $f_t$ maps a row to a leaf weight. The prediction after $t$ trees is the sum so far:

The number $\eta$ is the learning rate, also called shrinkage. In XGBoost it is eta. It multiplies each tree's output before the tree joins the sum, so the model takes only part of each correction. The next tree then sees what is left and corrects part of that.

Under squared error, "what is left" is the residual, $y-F_{t-1}\left(x\right)$. The figure shows that residual as a line from each prediction to the true score. Move the slider and watch the lines shrink. Set eta to 1, and the lines shrink quickly, but each tree then fits individual rows more closely, which is how a boosted model overfits.

This shows the first property to remember: the trees are not trained together. When tree $t$ is built, all earlier trees are frozen. So training reduces to one repeated question: given the current predictions, which one tree helps most? The next three sections answer it.

Add one tree to a prediction

A tree's leaf weight never joins the sum in full. It is multiplied by eta first. Here is row 1 in the first tree of section 7, with eta 0.3:

1. Read the weight. Row 1 goes to a leaf whose weight is w = −12.
2. Shrink it by eta. The step is 0.3 × (−12) = −3.6.
3. Add the step. The prediction moves from 57.5 to 53.9.

3The objective: loss plus a price on complexity

To choose a tree, we need a number that says how good a candidate is. XGBoost uses an objective: the total loss over all rows plus a penalty on each tree's complexity.

The penalty is where XGBoost differs most from plain gradient boosting. For a tree with $T$ leaves and weights $w_1,\dots ,w_T$, it is

• γT charges a fixed price per leaf. A split adds one leaf, so it must pay at least $\gamma$. This is gamma.
• ½λΣw² charges for large weights, and it charges most for the largest. This is lambda.
• αΣ|w| also charges for size, but its charge per unit stays the same as a weight approaches zero. So a weight can become exactly 0. This is alpha.

The penalty shapes the leaf weights themselves, not only the choice of split. Section 5 shows how.

What one split buys

Take the squared loss $l=\frac{1}{2}(y-\widehat{y}{)}^2$ on our rows, with $\eta =1$ and $\lambda =1$. Before any tree, every row predicts 57.5, and the total loss is 1225. Now split on study < 4.5. Each leaf's weight is the sum of its residuals divided by its row count plus λ, a rule that section 5 derives. The left leaf's residuals sum to −60, so its weight is −60/(4 + 1) = −12, and the right leaf's sum to 60, so its weight is 12. The loss falls to 361. The penalty adds 144, so the objective is 505. The split lowered the objective by 720.

Price a tree

The penalty has one part for each term of $\Omega$. Here is the split on study < 4.5 from above, with gamma 0, lambda 1 and alpha 0:

1. Price the leaves. γT = 0 × 2 = 0.
2. Price the sizes. ½λΣw² = ½ × 1 × ((−12)² + 12²) = 144, and the alpha term is 0.
3. Add the loss. The objective is 361 + 144 = 505.

4Taylor expansion: reduce any loss to two numbers

When tree $t$ is built, earlier trees are frozen. So write the objective as a function of this tree only. Terms for old trees are constants, so we drop them.

For squared error, this is easy to minimize. For a general loss it is not, because $f_t$ sits inside $l$. So XGBoost replaces $l$ with a parabola that matches it at the current prediction.

Slope, curvature, and the parabola

The derivation below uses two derivatives of the loss, taken at the row's current prediction. The first derivative is the slope: how fast the loss changes as the prediction moves. The second derivative is the curvature: how fast that slope itself changes. Near the current prediction, a parabola with the same value, slope and curvature follows the loss closely. That parabola is the Taylor expansion of second order.

Here is one row under logistic loss, with label 1 and current margin 0. Its loss is 0.6931, its slope is −0.5, and its curvature is 0.25. For a step of w = 1, the parabola gives 0.6931 − 0.5 + ½ × 0.25 = 0.3181, and the true loss is 0.3133. For a step of w = 3, the parabola gives 0.6931 − 1.5 + ½ × 0.25 × 9 = 0.3181 again, but the true loss is 0.0486. So the parabola is close for small steps and far off for large ones. Fig. 4.1 below shows the same comparison at any margin.

The derivation, one step at a time

Look at what happened to the loss. It is gone. All that remains of it is two numbers per row, $g_i$ and $h_i$. In this library, g is called the gradient and h is called the hessian. g says which way the loss falls and how steeply. h says how quickly that slope changes, which tells us how far to trust it.

Why one tree builder can serve any loss

The code that builds the trees never looks at the loss. It receives two arrays, g and h, and it works only with their sums. To support a new loss, you write a function that returns those two arrays. Nothing else changes. That is what the obj argument of xgboost.train does.

The loss has to be twice differentiable for this to work as derived, with $h$ positive so the parabola opens upward. Losses that fail this need special handling, which section 6 describes.

Compute g and h for one row

Substitute the row's current prediction into the two derivatives. Under squared error, $g$ is the prediction minus $y$, and $h=1$. Under logistic loss with probability $p$, $g=p-y$ and $h=p(1-p)$. Section 6 derives these and the formulas for the other objectives.

1. Squared error, row 1. The prediction is 57.5 and y = 35, so g = 22.5 and h = 1.
2. Logistic loss, row 1. The margin is 0, so p = 0.5. With y = 0, g = 0.5 and h = 0.5 × 0.5 = 0.25.

5Leaf weight, leaf score and gain

Now use the structure of a tree. A tree gives every row in a leaf the same value, the weight $w_j$. So we can group the sum by leaf. Let $I_j$ be the rows in leaf $j$, and define the two sums that run the whole algorithm:

The best weight for a leaf

Read this as a Newton step: one step of Newton's method jumps straight to the lowest point of a parabola, and here the parabola is the leaf's bracket above. Under squared error, $h=1$ and $g$ is minus the residual, so $w^{*}$ is the mean residual in the leaf, pulled toward zero by $\lambda$. With $\lambda =0$, it would be exactly that mean.

The tree then stores $\eta w^{*}$, as section 2 showed. Here is the left child of the first split in section 7:

1. Sum the rows. The child has G = 60 and H = 4.
2. Take the Newton step. w = −60/(4 + 1) = −12.
3. Store eta times it. With eta 0.3, the leaf stores −3.6.

The score of a leaf

Put $w_j^{*}$ back into the bracket to see how low the objective goes for that leaf:

So for a whole tree with a fixed structure, the objective is

This is how XGBoost scores a tree shape without knowing anything else about the loss. A leaf is good when its rows pull in the same direction, because then $|G_j|$ is large compared with $H_j$.

The gain of a split

Now compare a leaf with the two leaves that would replace it. The rows in $I$ divide into $I_L$ and $I_R$, so $G=G_L+G_R$ and $H=H_L+H_R$. The objective changes by the score after, minus the score before, minus one more $\gamma$ for the extra leaf:

This is the split function. It is what the tree builder evaluates at every candidate threshold. Everything about XGBoost's trees comes from this one line and from the leaf weight above.

What the library does differently

The paper's formula has a factor of one half and subtracts $\gamma$. The library does neither. Its number is the one that get_dump calls gain and that feature_importances_ averages:

It then compares that number directly with gamma. So in the library, gamma is a threshold on twice the objective drop. Setting gamma=500 acts like $\gamma =250$ in the objective as written above. The docs' tutorial writes the formula with the one half, and the code leaves it out.

1. Score the children. For study < 4.5, each child scores 60²/5 = 720.
2. Subtract the parent. The root has G = 0, so its score is 0, and the dumped gain is 1440.
3. Compare with gamma. The split stays while its gain reaches gamma, so gamma=1000 keeps it and gamma=1441 removes it.

Lambda in the running rows

With $G_L=60$ and $H_L=4$ for the left child of that first split, $\eta =1$ in the table below. Lambda shrinks both the weight and the gain:

How reg_alpha soft-thresholds G

Now keep $\alpha$. The bracket for a leaf becomes $Gw+\frac{1}{2}(H+\lambda )w^2+\alpha |w|$. The absolute value has a kink at 0, so the two sides need separate cases.

These three cases are one operation, called soft thresholding. It moves $G$ toward 0 by $\alpha$ and stops at 0:

This is why alpha can zero a leaf, and why it can remove a split. On the running rows with lambda 1 and $G_L=60$, any alpha of 60 or more removes the gain completely, so the node stays a leaf: no split at alpha 60.

1. Soft-threshold G. With alpha 10, G = 60 moves 10 toward 0, to T = 50.
2. Take the Newton step. w = −50/(4 + 1) = −10.000, as in the table.

How max_delta_step clips the weight

The parameter max_delta_step $m$ sets a hard limit on the size of a weight, applied after the formula and before eta. When $m\&gt;0$:

The node's score then cannot use the closed form, because the weight is no longer the minimizer. So the library evaluates the objective at the clipped weight:

With no clipping and no alpha, this equals $G^2/(H+\lambda )$. With clipping it is smaller, so clipping lowers both the weight and the gain.

When alpha and max_delta_step are both set, the source uses this same score with the raw $G$, and no alpha term appears in it. On our rows with alpha 10 and max_delta_step 5 that gives gain 950.000, which equals the dump.

1. Take the Newton step. For the left child, w = −60/5 = −12.
2. Clip it. With max_delta_step=5, the weight becomes −5.000.
3. Score the clipped weight. Each child scores −(2 × 60 × (−5) + 5 × 25) = 475, so the gain is 950.000.
4. Store eta times it. The clip comes first, so with eta 0.3 the leaf stores 0.3 × (−5) = −1.5.

6g and h for each objective

Choosing an objective means choosing these two formulas, plus the link that turns the margin into an output. In the table, $m$ is the margin, $p$ is the output after the link, and each row's values are multiplied by its sample weight.

In the ranking row, $\sigma$ is the sigmoid of the score difference, and $\Delta$ is the change in the metric if $i$ and $j$ swapped: the change in NDCG, the change in MAP, or 1 for pairwise. It may be divided by $|s_i-s_j|+0.01$ (lambdarank_score_normalization). Each query's sums may be scaled afterwards (lambdarank_normalization).

Read g and h from the table

Find the objective's row in the table, then substitute the margin and the label.

1. Find the formulas. Under reg:gamma, $g=1-y{e}^{-m}$ and $h=y{e}^{-m}$.
2. Substitute. At m = 0 with y = 4, g = 1 − 4 = −3 and h = 4.

Why L1-type losses need special handling

The absolute loss $|m-y|$ is a V shape. Its second derivative is 0 everywhere except the kink, where it is undefined. If you used that honest $h=0$, the weight would be $-G/\lambda$, and it would grow with the row count rather than settle at a sensible size. Pinball loss has the same shape with two different slopes.

XGBoost handles this in two steps. It grows the tree structure with a placeholder $h=1$, so the gain formula still runs. Then it throws away the leaf values and sets each leaf directly to the value that minimizes the true loss. For absolute loss, the best constant is the median of the residuals in that leaf. For pinball, it is the $\alpha$ quantile.

Here is that on our rows, with reg:absoluteerror. The intercept is the median of the labels, 52.5. The tree splits at study < 5 with gain 6.400, computed from $g=\pm 1$ and $h=1$.

• Left leaf, 4 rows, residuals −17.5, −12.5, −7.5, −2.5. Their median is −10.000, and times eta 0.3 that gives leaf −3.000.
• Right leaf, 4 rows, residuals 2.5, 17.5, 27.5, 32.5. Their median is 22.500, and times eta 0.3 that gives leaf 6.750.

The placeholder formula would have given a leaf of −0.240 on the left. That is what tree_method="exact" actually stores: the −0.240 and 0.240 come from an exact run, because the refit happens only under hist and approx. No warning is shown. So use hist with these objectives.

For pinball with quantile_alpha=0.75, the intercept is 77.5, and the split is study < 7 with gain 1.071. The left leaf has 6 residuals (−42.5, −37.5, −32.5, −27.5, −22.5, −7.5), whose 0.75 quantile is −18.750, so the leaf stores −5.625. The right leaf has 2 residuals (2.5, 7.5), so it stores 2.250 from a quantile of 7.500. Under exact, these would be the placeholders −0.064 and 0.150. The quantile rule is the library's own, with position $\alpha (n+1)$ in the sorted values, from common::Quantile in src/common/stats.h.

Since the placeholder $h$ is 1 and $H$ counts rows, min_child_weight counts rows for these objectives. Also, lambda and alpha affect only the structure, since the refit replaces the leaf values (inference from the refit code). And reg:pseudohubererror exists because it avoids all of this: it is twice differentiable, with a real $h$.

1. Collect the residuals. The left leaf of the reg:absoluteerror tree holds −17.5, −12.5, −7.5, −2.5.
2. Take the quantile. Their median is −10.000.
3. Shrink by eta. The leaf stores 0.3 × (−10) = −3.000.

7Two trees by hand

Now do the whole algorithm once. The settings are objective="reg:squarederror", tree_method="exact", max_depth=2, eta=0.3, lambda=1, gamma=0, min_child_weight=1.

Round 1: g and h

XGBoost first estimates the intercept. Under squared error, this is the mean, 57.5. So every row starts at 57.5. For squared error, $g=\widehat{y}-y$ and $h=1$.

Round 1: scan every split at the root

At the root, $G=0$ and $H=8$. So the parent's score is $0^2/(8+1)=0$. Each candidate threshold sits halfway between two neighboring distinct values. Sleep has 5 distinct values, so it has only 4 candidates. Here is one by hand, study < 4.5. Rows 1 to 4 go left:

Every candidate follows the same arithmetic:

The best is study < 4.5. Notice that the second best, study < 6.5, is not a neighbor of it. Gain is not a smooth curve along a feature, so the algorithm has to check them all.

Round 1: the left child stays a leaf

The left child holds rows 1 to 4, with $G=60$ and $H=4$. Its own score is 720, from ${60}^2$ over $4+1$. Scan it the same way:

Every gain is negative, so depth 2 is allowed but no split happens. All four residuals in this node have the same sign. Splitting them cannot separate rows whose residuals have opposite signs. It only makes two leaves with smaller $H$, and $\lambda$ then pulls harder on each. This is the objective at work. Gamma is 0 here, so the refusal comes from $\lambda$ alone. Set $\lambda$ to 0 in Fig. 7.1, and the same node splits, with gain 100.000.

Round 1: the right child splits

The right child holds rows 5 to 8, with $G=\mathrm{-60}$ and $H=4$.

The best is sleep < 5.5. Row 6 goes left, and rows 5, 7, 8 go right. Row 6 is the student who scored less than the others on this side, and its residual has the opposite sign, so it is exactly the row that this split should isolate.

Round 1: leaf weights and the update

Each leaf's weight is $w=-G/(H+\lambda )$, and the tree stores $\eta w$.

Add each row's leaf to its prediction. The RMSE falls from 17.5 to 13.9108.

Round 2: the residuals changed, so everything is recomputed

Now $g$ has changed, though $h$ is still 1. The second tree fits what the first tree left.

Scan the root again:

It is the same split, but now with gain 844.785 rather than 1440. The first tree took 30% of the correction, so the same split still has the most to give. The left child again stays a leaf, and the right child splits again:

The RMSE falls to 11.2634. It takes many more rounds to get far, because of eta. Try Fig. 2.1 for that.

Comparing with the real model

Here is what booster.get_dump(with_stats=True) prints for this exact model. Each number in the tables above matches it. Note that the leaf holds $\eta w$, and cover is the node's $H$.

The default method hist grows the same trees with the same gains. Only how it writes the threshold differs: f0<5 instead of f0<4.5, because hist compares with the next value in its bins rather than a midpoint. The dump's missing also differs, 2 for hist and 1 for exact, since neither saw a missing value here and each has its own scan order. The dumped gain, 1440, is the same.

Scan a small node yourself

Each practice node below has its rows sorted by one feature, with h = 1 per row, as under squared error. Score every threshold with the gain formula, and pick the largest, as the root scan of round 1 did:

1. Score the parent. At the root, G = 0 and H = 8, so the parent's score is 0.
2. Score every threshold. study < 4.5 gives 60²/5 + (−60)²/5 − 0 = 1440, and the table lists the other thresholds.
3. Choose the largest. No other candidate beats 1440, so the root splits on study < 4.5.

8The same rows under logistic loss

Squared error does not show half of the mechanism, because $h$ is always 1. Logistic loss shows it. Use the pass column as the label, and keep the same settings. The objective is binary:logistic. The margin $m$ is in log-odds, $p=1/(1+e^{-m})$, $g=p-y$, and $h=p(1-p)$.

This section is a faded example. It shows the first step in full. After that, it asks you to predict each step before showing it, and in the last part you do the scan yourself.

XGBoost estimates base_score as the mean of the labels, 0.5. It converts that to a margin of 0 with the logit, ln(p/(1 − p)), which undoes the formula for p. So every row starts at $p=0.5$, with $g=0.5$ or $\mathrm{-0.5}$, and $h=0.25$.

The default min_child_weight now counts differently

min_child_weight requires $H_L$ and $H_R$ to reach 1. With $h=0.25$ per row, a child needs 4 rows. On eight rows, that leaves only the even four-and-four splits. Under squared error, the same setting needed one row.

Only study < 4.5 and sleep < 6.5 pass. The chosen split gives leaves of −0.15 and 0.15.

Then round 2 grows nothing. Now $p$ has moved away from 0.5, so $h$ has fallen below 0.25 for every row. Four rows no longer sum to 1, so no split is valid, even the even split:

The default of 1 stopped this model after one split. With few rows and an output near 0 or 1, this is a common cause of a model that learns nothing. It comes from $h$, not from the row count.

With min_child_weight = 0

Remove the block, and every candidate is back. The rows start from the same g and h:

Now the second round. Here is the point of this section: $h$ now differs from row to row. A row that the model is more sure about has a smaller $h$, so it counts for less in $H$. That is how this one formula becomes a Newton step on a curved loss.

Note that the dump's cover is $H$, not a row count. A leaf reporting a cover of 0.25 holds one row here. This is why cover looks so odd on classification models.

Rows needed under logistic loss

min_child_weight counts h, so the number of rows that a child needs depends on p.

1. Find h per row. At p = 0.5, h = 0.25.
2. Divide and round up. 1/0.25 = 4, so each child needs 4 rows.

9Finding splits: which candidates get scanned

The gain formula says how to score a candidate. It does not say which candidates to try. That is what tree_method decides. The gain formula is the same under all of them.

Exact greedy

For one feature, sort the rows by value. Walk through them, moving one row at a time from one side to the other. Each move updates $G_L$ and $H_L$ with one addition, and $G_R=G-G_L$ follows. So a whole feature costs one sort plus one pass, and no row is summed twice. That is the scan you did by hand. With millions of rows and many features, though, one pass per node per feature is still too slow.

Histograms

The idea is to pick a limited set of cut points per feature and replace each value with its bin number. Then for a node, instead of walking rows in order, add each row's $g$ and $h$ to its bin. Now a scan walks the bins, not the rows. Each candidate still uses the same gain formula on the running sums.

On our rows with hist, each value has its own bin, so max_bin is not limiting and the gains equal exact's. With max_bin=2, each feature has one threshold left, near its median. On these rows the best split happens to be that median cut, so the root still finds f0 < 5 with gain 1440.000. On the larger dataset in the max_bin table, two bins are far worse, because the best split is usually somewhere else.

Histogram subtraction

A node's rows are exactly its two children's rows. So the parent's histogram is the sum of the children's histograms, bin by bin. That means you only need to build the histogram of one child, normally the smaller one. The other follows by subtraction, at a cost proportional to the bins, not the rows. Here is that on the running rows, for the sleep feature, after the root split on study:

Weighted quantile sketch: why h is the weight

Which cut points are best? Equal numbers of rows per bin sounds natural. But look again at the quadratic objective for a row. Complete the square:

This is a weighted least-squares problem. The target for each row is $-g_i/h_i$, and its weight is $h_i$. So a row with larger $h$ matters more to the objective, and it deserves finer resolution. The paper therefore picks cut points so that each bin holds about the same total $h$, not the same row count. A sketch is a compact structure that finds these weighted quantiles without sorting everything in memory.

This is why approx sketches again before each tree: $h$ changes every round. And it is why hist, sketching once with no $h$, is faster but doesn't adapt. The docs say that on losses with varying $h$, approx sometimes gives better accuracy, but hist with a higher max_bin can usually match it. On squared error, $h$ is constant, so the two agree: on the 2000 training rows used in section 12, they gave the same test RMSE to three decimals, in the tree_method table.

Missing values: learn a default direction

XGBoost needs no imputation. For each split, it learns which side missing values should go to. The scan runs over present values only, with the missing rows' $g$ and $h$ added to one side, then again with them added to the other side. The better of the two becomes that split's default direction.

Hide sleep for rows 2 and 7, and take sleep alone at the root of the first tree. The missing rows have $G=\mathrm{-5}$ and $H=2$. The remaining values are 4, 5, 7, 8, so the candidates are at 4.5, 6 and 7.5. Each one is computed both ways:

The best is sleep < 6, with missing rows going right, for gain 107.143.

When there are no missing values, a direction is still stored but it was not learned from anything. So on new data with a missing value, a model never trained on missing values sends it down an arbitrary side. exact and hist even disagree on it: section 7 showed each choosing a different default. inference from those dumps and the scan order in source.

1. Send the missing rows left. At sleep < 6, the gain is 40.000.
2. Send them right. The gain is 107.143.
3. Keep the larger. The default direction is right, with gain 107.143.

Categorical splits

With enable_categorical=True and a column marked c, XGBoost can split a category set directly. There are two forms, and max_cat_to_onehot chooses between them: fewer categories than that value means one-hot, and otherwise partition.

Take twelve rows with six categories, two rows each, whose labels have mean 27.000.

• One-hot tests one category against all the others. The best of the six gives gain 1060.606 by hand and 1060.606 in the dump, sending category 3 one way.
• Partition sorts categories by their leaf weight, then scans the sorted order like a numeric feature. The order here is 0, 2, 4, 1, 5, 3. For a squared-type objective, sorting this way finds the best two-group split without checking all $2^{k-1}$ of them, which is Fisher's result, cited in the docs. It gives gain 2314.286 by hand and 2314.286 in the dump, sending categories 0, 2, 4 one way.

Partition gets a much larger gain from one split. max_cat_threshold limits how many categories that sorted scan may place on one side. exact rejects categorical data.

10Growth, pruning and the rest of the algorithm

Tree growth: depthwise or lossguide

With the same leaf budget, a tree can be grown in two orders. depthwise finishes each level before the next. lossguide always expands the leaf with the largest gain. The figure shows both on real trees, with 8 leaves each.

Lossguide always finds at least as much gain for the same leaf count here, because it spends each leaf on the best available split. It can also go deep on one side, which is why it can overfit sooner. The grow_policy table in section 12 shows this. The check script requires this gain ordering on this example only; it isn't a theorem.

Pruning with gamma: two different behaviors

The source and the behavior differ by method. hist and approx test gamma when they consider a node. A split below gamma isn't made, so its children are never tried. exact first grows the whole tree as long as gain is positive. Then its prune updater walks up from the leaves and removes a split only if its gain is below gamma and both children are leaves.

That matters for a split that looks weak but sets up strong ones. Twenty rows in an XOR pattern show this: neither feature helps alone, but both together separate the rows. With gamma 0, the root's gain is only 0.000125, while its children's gains are 2.450 and 2.500. Now set gamma to 1:

So the same gamma gives a different tree, and a comparison between tree methods has to allow for that.

1. Read the gains. The XOR root has gain 0.000125, and its children have 2.450 and 2.500.
2. Apply the rule of exact. Both child splits reach gamma 1, so they stay. The root is below 1, but its children are not leaves, so it stays too: 4 leaves.
3. Apply the rule of hist. The root is below 1, so it is never split: 1 leaf.

base_score: where the sum starts

$F_0$ is estimated from the training labels. The rule depends on the objective:

When scale_pos_weight is not 1, logistic objectives use a Newton step from margin 0 rather than the mean, giving 0.7311 on our rows at a weight of 3. The value for Pseudo-Huber shows the risk in the Newton rule. At margin 0 and labels near 57, each $h$ is tiny, so $-G/H$ is huge. Setting base_score by hand avoids that. In any case its effect fades with rounds, as the base_score table in section 12 shows.

Sampling

subsample draws rows each round, and colsample_bytree, colsample_bylevel and colsample_bynode draw features. Neither changes the formulas. Row sampling removes rows from the sums. Column sampling removes candidates from the scan. Two trees that see different data make different mistakes, and averaging different mistakes reduces variance, which is the same reason that random forests work. num_parallel_tree builds this in directly: several trees per round, each on its own sample, with eta divided among them.

DART: boosting with dropout

Under booster="dart", each round drops a random subset of existing trees. Then $g$ and $h$ come from the model without them, so the new tree is fitted to a partial model. The idea is to stop later trees from only touching up a few rows. Afterwards, the new tree and the dropped trees are rescaled, so that the total stays near where it was.

The rule depends on normalize_type. If $k$ trees were dropped, tree sets the new tree's weight to $1/(k+\eta )$ and multiplies dropped trees by $k/(k+\eta )$. forest uses $1/(1+\eta )$ for both.

gblinear: boosting without trees

Under booster="gblinear", each $f_t$ is a linear function. Each round does coordinate descent: for one feature at a time, it takes a Newton step on that coefficient, using $g$ and $h$ weighted by the feature's value. The update from the source is

where the sign of $\alpha \prime$ pushes the coefficient toward zero, and the update stops at $-w_k$ if alpha would carry it across zero. The bias takes its own Newton step first, with no penalty: 0.0000 after one full step on our rows. Sums of linear functions are linear, so more rounds refine the same coefficients, unlike trees. The final model is regularized linear regression, solved iteratively. It can't capture any interaction or curve.

Multi-class and multi-output

Under softmax with $K$ classes, each row has $K$ margins, so $K$ pairs of $g$ and $h$. By default, each round grows $K$ trees, one per class, each on its own $g$ and $h$: 3 classes over 5 rounds gave 15 trees. With multi_strategy="multi_output_tree", one tree covers all outputs, and each leaf holds a vector of size 3: 5 rounds gave 5 trees. Weights come from each output's formula, and gains add over outputs. The docs mark this as experimental, and it needs hist.

Early stopping

Rounds keep lowering training loss, but held-out loss eventually rises. Early stopping watches an evaluation set after each round, and stops when early_stopping_rounds rounds pass with no improvement. With several sets or metrics, the last of each is used.

What it returns needs care. In the run below, the best round was 28 (zero-based), and the booster still holds 39 rounds. The wrapper's predict stops at best_iteration, but Booster.predict uses every tree unless you pass iteration_range. That is an RMSE of 1.6887 versus 1.6916 on the same booster. EarlyStopping(save_best=True) is the way to cut the model.

Types of feature importance

These all come from the same dump, read differently. On our two trees:

The sklearn property feature_importances_ is average gain, normalized to sum to 1: 0.8449 and 0.1551 here. The three views can rank features differently. A feature used once with a large gain scores high on gain and low on weight. None of these says what a feature contributes to one prediction. For that, use pred_contribs.

How XGBoost differs from classic gradient boosting

11One run, from the data to a prediction

Sections 2 to 10 explained each mechanism separately. This section puts them in the order that the library runs them, and it names each parameter at the step where it acts. A parameter changes the model only at its step. So if you know the step, you know what the parameter can change and what it cannot.

The order below is for the default settings: booster="gbtree", tree_method="hist" on the CPU, and one output per row, as in regression or binary classification. The table at the end of this section lists what changes under other settings.

Before the first round

1. Bin the features. Under hist, each feature gets at most max_bin cut points (256 by default), and each value is replaced by the number of its bin. The cut points come from the feature values, weighted by sample_weight only, so they stay fixed for the whole run (section 9). A value equal to missing (NaN by default) is stored as absent, and so is any entry that a sparse matrix does not store. With enable_categorical=True, a column of type c gets one bin per category. The wrapper's fit bins when it builds its QuantileDMatrix, and xgb.train with a plain DMatrix bins when it grows the first tree. Both give the same cut points, because the cut points do not use g or h.
2. Set the starting value. If you don't set base_score, the library estimates it from the training labels at the start of the first round. The rule depends on the objective, and a scale_pos_weight other than 1 changes the rule for the logistic objectives (section 10). The value is stored as a margin, so under the logistic objectives 0.5 becomes the logit 0. If the training matrix has a base_margin, each row starts from its own base_margin instead.

Each round

The wrapper runs n_estimators rounds (100 when unset), and xgb.train runs num_boost_round rounds (10 by default). Each round runs steps 3 to 14.

3. Compute each row's margin. A row's margin is its starting value plus the stored leaf value of every earlier tree. The library keeps the training margins in a cache, so each round adds only the newest trees.
4. Compute g and h. The objective gives g and h at the row's margin (section 6). Both are multiplied by the row's sample_weight. Under reg:squarederror, reg:gamma and the three logistic objectives, a row whose label is exactly 1 also has its weight multiplied by scale_pos_weight. Under the logistic objectives, h is at least 10⁻¹⁶.

A round then grows num_parallel_tree trees (1 by default), and each uses eta divided by num_parallel_tree as its learning rate. A multi-class model does this once for each class. Steps 5 to 12 run once for each tree.

5. Sample the rows. With subsample below 1, each row is kept with probability subsample, independently of the other rows, so the number of kept rows varies. In this release, the first two rows of the training data are an exception: every tree keeps them, as Fig. 11.3 shows. A row that is not kept gets g = 0 and h = 0 for this tree. So it adds nothing to any sum, but it still goes through the tree, and step 13 still adds the tree's leaf to its margin. sampling_method="gradient_based" keeps a row with a probability that grows with $\sqrt{g^2+0.1h^2}$, and it divides a kept row's g and h by that probability. seed starts the random generator once, and seed_per_iteration=True restarts it every round from the seed and the round number.
6. Sample the tree's features. colsample_bytree keeps $max(1,\lfloor \text{rate}\times \text{count}\rfloor )$ features for the whole tree, chosen at random. With feature_weights, a feature's chance of being chosen follows its weight.
7. Start the tree at the root. The root's G and H are summed over all rows, and the rows that step 5 did not keep add 0. The root starts as a leaf that stores $\eta w$, with $w=-{\text{T}}_{\alpha }\left(G\right)/(H+\lambda )$ clipped by max_delta_step, and w is 0 when H is below min_child_weight. If the root never splits, the tree is that one leaf.

Nodes are taken in grow_policy order. depthwise takes every node of one depth before the next depth, and lossguide takes the node with the largest gain first (section 10). Steps 8 and 9 run when a node is created, and steps 10 and 11 run when its turn comes.

8. Choose the node's candidate features. The node starts from the tree's features of step 6. colsample_bylevel draws a subset of them once for each depth of this tree, and colsample_bynode draws a subset of the depth's features for each node. Each draw uses the same count rule as step 6, so the counts shrink in turn: with 10 features and 0.5 for both of the first two, each depth gets 2. With interaction_constraints, every feature is allowed at the root. Below the root, a feature is allowed only if a split above this node used it, or it is in a group that holds every feature used above this node.
9. Score every candidate split. For each candidate feature, the node's histogram is scanned bin by bin (section 9). The scan first counts the rows with a missing value on the right. Only if the node has missing values for this feature is the scan repeated with them on the left. A candidate is skipped when either side has an H below min_child_weight. Its gain is the two children's scores minus the node's score, and lambda, alpha and max_delta_step enter those scores (section 5). With monotone_constraints, each child weight is first clamped to a range that the splits above set. If the two weights are then in the wrong order, the candidate gets a gain of −∞. A categorical feature is scanned one-hot or as a sorted partition, by max_cat_to_onehot and max_cat_threshold (section 9). The node keeps its best candidate.
10. Decide whether to split. The node splits only if every rule in this list holds. Otherwise it stays a leaf.
    • The best gain is above 10⁻⁶.
    • Both children have an H above 0.
    • The best gain is not less than gamma, so a gain equal to gamma splits.
    • The node's depth is less than max_depth (6 by default, and 0 means no limit).
    • The tree has fewer leaves than max_leaves (0 by default, which means no limit).
    A child that would sit at depth max_depth is never scored in step 9, because it could not split.
11. Split, and set the two new leaves. Each child becomes a leaf that stores $\eta w$, with w from its own G and H as in step 7, and clamped to its monotone range. Here η is eta divided by num_parallel_tree. The child keeps this value unless it splits later. Then the children's histograms are made: the child with the smaller H is built from its rows, and the other child's is the parent's minus it (section 9). max_cached_hist_node limits how many histograms are kept for this. Steps 8 to 11 then repeat for the new children.
12. Refit the leaves, for L1-type objectives only. Under reg:absoluteerror and reg:quantileerror, after the tree stops growing, each leaf's value is replaced by η times the median, or the quantile_alpha quantile, of the residuals of the leaf's kept rows (section 6). Under every other objective, the leaves keep their values from steps 7 and 11.

After the round's trees are grown:

13. Add the trees to the margins. The new trees join the model, and each training row's cached margin gains the leaf value of each new tree. This includes the rows that step 5 did not keep.
14. Evaluate, and stop early if asked. After the round, the library computes each eval_metric on each evaluation set. With early_stopping_rounds, training stops when the last metric on the last set has not improved for that many rounds, and best_iteration records the best round (section 10). Otherwise the next round starts at step 3.

Prediction for a new row

15. Start from the starting value. The row starts at the stored base_score, as a margin, or at its own base_margin if you pass one.
16. Route the row through each tree. At each split, a missing value goes to the split's default side. A numeric value goes left if it is less than the threshold, and right if it is not, so a value equal to the threshold goes right. A category goes right if it is in the split's category set. At the leaf, the row's margin gains the leaf's stored value. The wrapper's predict uses the trees of rounds 0 to best_iteration when early stopping set it, and every tree otherwise. Booster.predict uses every tree unless you pass iteration_range.
17. Convert the margin. The objective's link turns the margin into the output. reg:squarederror returns the margin as it is, and binary:logistic returns $p=1/(1+e^{-m})$. predict_proba returns 1 − p and p, and XGBClassifier.predict returns class 1 only when p > 0.5. output_margin=True returns the margin without the link.

Two of these results lead to mistakes in practice. First, a model trained with base_margin still estimates base_score, and predict adds it to any row that comes without a base_margin. Trained with base_margin 0 on the running rows, one row predicts 15.3333 with its base_margin and 72.8333 without it, which is 57.5 more. Second, under hist, missing values at prediction time go right at every split that had none in training, because the scan with them on the left never ran. So a feature that first goes missing at prediction time is treated as a large value, without any warning.

Section 7's run, step by step

Here is the run of section 7 again, with tree_method="hist", which grows the same trees. The settings are max_depth=2, eta 0.3, lambda 1, and the defaults for everything else. Each row of the table is one step, with the settings that act there and what they did in each round.

Predict a new row

Take a new student who studied 6 hours and whose sleep is missing. Steps 15 to 17 give the prediction, and the missing value meets the second split of each tree.

1. Start from the starting value. The margin starts at 57.5.
2. Route through tree 1. Study 6 is not less than 5, so the row goes right. Sleep is missing, so the row goes to the default side, which is right. The leaf stores 4.6875.
3. Route through tree 2. The splits are the same, so the row reaches the leaf that stores 3.6328.
4. Add, then convert. 57.5 + 4.6875 + 3.6328 = 65.8203. Squared error has no link, so this is the prediction, and predict returns it.

Now take section 8's logistic model, with min_child_weight 0. The same row starts at margin 0. Sleep is missing at the root, so the row goes right, and study 6 is not less than 2, so it goes right again. The two trees add 0.3 and 0.2582, for a margin of 0.5582 and p = 0.6360. That is above 0.5, so XGBClassifier.predict returns class 1. A row with p exactly 0.5 gets class 0.

Place a parameter at its step

If you know where a parameter acts, you can answer most questions about it. A parameter in step 1 changes the data before any tree exists, so max_bin cannot depend on how training goes. A parameter in steps 5 and 6 changes which rows and features a tree may use, but not the formulas. A parameter in step 9 changes the scores of candidates, and one in step 10 only accepts or refuses the best candidate.

1. Find the step. colsample_bylevel acts in step 8, after colsample_bytree in step 6.
2. Apply the rule in order. With 10 features, step 6 keeps max(1, ⌊0.5 × 10⌋) = 5. Step 8 draws from those 5, so it keeps max(1, ⌊0.5 × 5⌋) = 2.
3. State the effect. Each node at that depth scores 2 features, and the formulas are unchanged.

What changes under other settings

Fig. 11.1 marks these steps when you choose a setting there.

12Every hyperparameter

This section has 152 entries, and it covers all 143 names found by enumerating the installed library and the documented list: XGBModel.__init__, get_params(), the fit, predict, train, cv and DMatrix signatures, the callback classes, get_config(), and save_config() under every booster, tree method and objective. The check script fails if any of them lacks an entry here. Wherever save_config() reports a default, the script also compares it with the card.

"Acts on" tells you which formula term or algorithm step the parameter changes. An alias is a second name for the same setting. The Python wrapper usually offers both names.