[CI] Refactor Jenkins CI pipeline + migrate tests to Jenkins

[hcho3]

See #4234 for details.

Overview of Jenkins CI pipeline:

[hcho3]

Will re-open after filling in all blanks

[hcho3]

@CodingCat I'm seeing this error with the latest master:

19/04/17 12:19:38 ERROR XGBoostSpark: when specifying eval sets as dataframes, you have to ensure that the number of elements in each dataframe is larger than the number of workers

https://xgboost-ci.net/blue/organizations/jenkins/hcho3-xgboost-ci-test/detail/refactor_jenkins/71/pipeline/33#step-68-log-2079

How serious is this error? Also, do we want to build a XGBoost JAR with one version of Spark and test it with another version of Spark?

[hcho3]

I took out all changes related to pylint and moved them to #4381, to keep this PR focused.

[CodingCat]

Oh, I didn’t notice the question here, I think it’s because we have too many cores in jenkins?

How many cores we have there?

[hcho3]

@CodingCat 8 cores

[CodingCat]

it’s strange...is the issue still there?

[hcho3]

@CodingCat For now, I am skipping JVM tests from the Build stage. Trying to export a test JAR so that I can run JVM tests in the Test stage. See https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/PR-4380/18/pipeline/173

[hcho3]

@CodingCat What's the meaning of this error?

- repartitionForTrainingGroup with group data which has empty partition
Tracker started, with env={DMLC_NUM_SERVER=0, DMLC_TRACKER_URI=172.17.0.2, DMLC_TRACKER_PORT=9091, DMLC_NUM_WORKER=16}
19/04/24 02:52:58 ERROR Executor: Exception in task 10.0 in stage 2.0 (TID 17)
java.util.NoSuchElementException: key not found: train
        at scala.collection.MapLike$class.default(MapLike.scala:228)
        at scala.collection.AbstractMap.default(Map.scala:59)
        at scala.collection.MapLike$class.apply(MapLike.scala:141)
        at scala.collection.AbstractMap.apply(Map.scala:59)
        at ml.dmlc.xgboost4j.scala.spark.XGBoost$.ml$dmlc$xgboost4j$scala$spark$XGBoost$$buildDistributedBooster(XGBoost.scala:148)
        ...
9/04/24 02:52:58 ERROR Executor: Exception in task 9.0 in stage 2.0 (TID 16)
java.util.NoSuchElementException: key not found: train

[hcho3]

Will re-open when things are ready. There's a long way to go, and I don't want to spam notification

[hcho3]

@CodingCat FYI, reducing the number of cores did the trick. Thanks for the suggestion

[CodingCat]

@hcho3, yeah, it should work, but 8 cores should not make the failure happen, I met it in a 40-cores machine...

Partitioner of spark may have some secretes that I don’t know...I will check