Huge increase in learning time introduced in version 0.81

[ldesreumaux]

I recently updated my version of xgboost from 0.80 to 1.0.2 and I noticed a huge increase in learning time.

I investigated the problem and found that this increase is caused by the Span class introduced in 2c50278 and first released in version 0.81. Compared to the structure that was used before, this class does some checks (SPAN_CHECK instructions in span.h). If you remove these checks, you will notice a huge decrease in learning time (> x2 !). In fact, the learning time gets back to what it was before version 0.81.

These checks should only be present in Debug mode.

[SmirnovEgorRu]

@ldesreumaux, thank you for reporting the issue.
Could you, please, provide more details (e.g. parameters and dimensions) to reproduce the issue on my side?

[ldesreumaux]

Here is a python script to reproduce the issue. With this script, there is a x2 improvement (200s/100s) on my laptop when I remove the checks in version 1.0.2.

import numpy as np
import xgboost as xgb

import time

def generate_data():
    X = np.random.normal(5, 2, [n_obs, n_features],)
    y = np.random.gamma(2, 4, n_obs)
    return X, y

seed = 42
num_boost_round = 1000
early_stopping_rounds = 10
learning_rate = 0.05
tree_method = "hist"

n_obs = 5*10**6
n_features = 50

params = {
    "learning_rate": learning_rate,
    "tree_method": tree_method,
    "subsample": 0.9,
    "seed": seed,
    "silent": 1,
    "verbosity": 0
}

np.random.seed(seed)
X_train, y_train = generate_data()
X_test, y_test = generate_data()
dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)

start_time = time.time()

model = xgb.train(
    params=params,
    dtrain=dtrain,
    num_boost_round=num_boost_round,
    early_stopping_rounds=early_stopping_rounds,
    evals=[(dtest, 'test')],
    verbose_eval=False
)

print("Time:", time.time() - start_time)
print("Trees:", model.best_iteration, "iterations")

[trivialfis]

A bit surprised your bottle neck is in objective. Is the slow down still in 1.0.2?

[trivialfis]

Which compiler are you using?

[ldesreumaux]

I am using Visual Studio 2015 on my Windows laptop.

I also had the same difference between version 0.80 and 1.0.2 with xgboost installed via pip in a Linux environment.

[trivialfis]

Thanks for sharing the information.

[trivialfis]

@ldesreumaux Could you give latest master branch a try? See if it makes thing a little better?

[ldesreumaux]

@trivialfis Thanks for the fix. With the latest master branch, the training times are in the same order of magnitude as before version 0.81, and there does not seem to be a very significant difference when I completely remove the span checks.