fix subset bug

[guolinke]

to fix #2744

[guolinke]

@jameslamb it seems the r tests randomly failed.

[StrikerRUS]

According to the logs, it cannot build dependencies:

* installing *source* package ‘data.table’ ...
** package ‘data.table’ successfully unpacked and MD5 sums checked
** using staged installation
zlib 1.2.11 is available ok
** libs
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include -fopenmp -fPIC  -Wall -g -O2  -c assign.c -o assign.o
clang: error: unsupported option '-fopenmp'
make: *** [assign.o] Error 1
ERROR: compilation failed for package ‘data.table’

...
/Applications/Xcode-11.3.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: ../../AMD.a(amd_l_dump.o) has no symbols
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DNTIMER  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c SuiteSparse_config.c -o SuiteSparse_config.o
ar -rucs ../SuiteSparse_config.a SuiteSparse_config.o
clang -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o Matrix.so CHMfactor.o Csparse.o TMatrix_as.o Tsparse.o init.o Mutils.o chm_common.o cs.o cs_utils.o dense.o dgCMatrix.o dgTMatrix.o dgeMatrix.o dpoMatrix.o dppMatrix.o dsCMatrix.o dsyMatrix.o dspMatrix.o dtCMatrix.o dtTMatrix.o dtrMatrix.o dtpMatrix.o factorizations.o ldense.o lgCMatrix.o sparseQR.o abIndex.o CHOLMOD.a COLAMD.a AMD.a SuiteSparse_config.a -L/Library/Frameworks/R.framework/Resources/lib -lRlapack -L/Library/Frameworks/R.framework/Resources/lib -lRblas -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin15/6.1.0 -L/usr/local/gfortran/lib -lgfortran -lquadmath -lm -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
ld: warning: directory not found for option '-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin15/6.1.0'
ld: warning: directory not found for option '-L/usr/local/gfortran/lib'
ld: library not found for -lgfortran
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Matrix.so] Error 1
ERROR: compilation failed for package ‘Matrix’

I wonder why is it using clang while we set gcc

LightGBM/.travis.yml

Lines 45 to 48 in 87b6396

	- if [[ $TRAVIS_OS_NAME == "osx" ]]; then
	export OS_NAME="macos";
	export COMPILER="gcc";
	export R_MAC_VERSION=3.6.1;

[guolinke]

@StrikerRUS it seems some R packages are built by clang.

[jameslamb]

This looks like Rdatatable/data.table#2406 and the installation notes for data.table has some information on requiring OpenMP.

I noticed that only the Mac builds are failing on Azure and Travis.

I think one solution to make the test less flaky is to use the available precompiled binaries for data.table when on Mac, which is what Matt Dowle suggests in that data.table issue.

For your consideration, I just opened #2752 to do just that. If it build successfully, I think we should merge it and then rebase this PR.

[StrikerRUS]

@guolinke Should a new separate issue be created for this to not lose it?

[guolinke]

PR Already created.

[StrikerRUS]

Ah, you mean #2754. Due to we have a lot of PRs modifying the same files, I overlooked it. Sorry about that!