Improve the CI container setup #365

tcojean · 2019-10-14T18:00:27Z

This PR updates the CI setup in accordance with what was previously in #307. This
allows to simplify #307 as well as multiple early benefits: testing ginkgo against
CUDA 10.1, gcc 9, clang 7 and 8. In addition, this allows the usage of the AMDCI
system as part of testing (CPU jobs, later AMD HIP ones also) which can help
reduce contention on the fineci system.

This closes #323.

Overview

Test jobs are fusioned with build jobs, this removes the need for artifacts
which Ginkgo overuses and can speedup the build process.
No CUDA containers are upgraded to gcc 9.1 and llvm 8
New CUDA 10.1 container and related jobs are introduced.
Use tags to allow using the AMDCI system and track between CUDA and CPU
requirements.

+ Test jobs are fusioned with build jobs, this removes the need for artifacts which Ginkgo overuses and can speedup the build process. + No CUDA containers are upgraded to gcc 9.1 and llvm 8 + New CUDA 10.1 container and related jobs are introduced. + Use tags to allow using the AMDCI system and track between CUDA and CPU requirements.

codecov · 2019-10-14T20:01:17Z

Codecov Report

Merging #365 into develop will decrease coverage by 69.58%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##           develop     #365       +/-   ##
============================================
- Coverage    89.98%   20.39%   -69.59%     
============================================
  Files          136      134        -2     
  Lines         6078     5104      -974     
============================================
- Hits          5469     1041     -4428     
- Misses         609     4063     +3454

Impacted Files	Coverage Δ
omp/solver/cgs_kernels.cpp	`0% <0%> (-100%)`	⬇️
reference/matrix/ell_kernels.cpp	`0% <0%> (-100%)`	⬇️
reference/matrix/dense_kernels.cpp	`0% <0%> (-100%)`	⬇️
reference/solver/cgs_kernels.cpp	`0% <0%> (-100%)`	⬇️
core/factorization/par_ilu.cpp	`0% <0%> (-100%)`	⬇️
core/matrix/identity.cpp	`0% <0%> (-100%)`	⬇️
core/matrix/coo.cpp	`0% <0%> (-100%)`	⬇️
omp/factorization/par_ilu_kernels.cpp	`0% <0%> (-100%)`	⬇️
reference/matrix/sellp_kernels.cpp	`0% <0%> (-100%)`	⬇️
core/base/perturbation.cpp	`0% <0%> (-100%)`	⬇️
... and 105 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update da35c73...ee6f233. Read the comment docs.

yhmtsai

LGTM
minor comments for keeping the same llvm index format.
By the way, codecov has pretty low rate in the relative coverage.
This PR should not hurt the coverage, right?

dev_tools/containers/README.md

dev_tools/containers/build_all_containers.sh

tcojean · 2019-10-17T11:45:45Z

I removed codecov because the coverage results seem to make no sense, so I am assuming there is a temporary bug with that tool. For proper coverage results, we can always check the dashboard:
https://my.cdash.org/index.php?project=Ginkgo+Project&date=

The Ginkgo team is proud to announce the new minor release of Ginkgo version 1.1.0. This release brings several performance improvements, adds Windows support, adds support for factorizations inside Ginkgo and a new ILU preconditioner based on ParILU algorithm, among other things. For detailed information, check the respective issue. Supported systems and requirements: + For all platforms, cmake 3.9+ + Linux and MacOS + gcc: 5.3+, 6.3+, 7.3+, 8.1+ + clang: 3.9+ + Intel compiler: 2017+ + Apple LLVM: 8.0+ + CUDA module: CUDA 9.0+ + Windows + MinGW and CygWin: gcc 5.3+, 6.3+, 7.3+, 8.1+ + Microsoft Visual Studio: VS 2017 15.7+ + CUDA module: CUDA 9.0+, Microsoft Visual Studio + OpenMP module: MinGW or CygWin. The current known issues can be found in the [known issues page](https://github.com/ginkgo-project/ginkgo/wiki/Known-Issues). Additions: + Upper and lower triangular solvers ([#327](#327), [#336](#336), [#341](#341), [#342](#342)) + New factorization support in Ginkgo, and addition of the ParILU algorithm ([#305](#305), [#315](#315), [#319](#319), [#324](#324)) + New ILU preconditioner ([#348](#348), [#353](#353)) + Windows MinGW and Cygwin support ([#347](#347)) + Windows Visual studio support ([#351](#351)) + New example showing how to use ParILU as a preconditioner ([#358](#358)) + New example on using loggers for debugging ([#360](#360)) + Add two new 9pt and 27pt stencil examples ([#300](#300), [#306](#306)) + Allow benchmarking CuSPARSE spmv formats through Ginkgo's benchmarks ([#303](#303)) + New benchmark for sparse matrix format conversions ([#312](#312)) + Add conversions between CSR and Hybrid formats ([#302](#302), [#310](#310)) + Support for sorting rows in the CSR format by column idices ([#322](#322)) + Addition of a CUDA COO SpMM kernel for improved performance ([#345](#345)) + Addition of a LinOp to handle perturbations of the form (identity + scalar * basis * projector) ([#334](#334)) + New sparsity matrix representation format with Reference and OpenMP kernels ([#349](#349), [#350](#350)) Fixes: + Accelerate GMRES solver for CUDA executor ([#363](#363)) + Fix BiCGSTAB solver convergence ([#359](#359)) + Fix CGS logging by reporting the residual for every sub iteration ([#328](#328)) + Fix CSR,Dense->Sellp conversion's memory access violation ([#295](#295)) + Accelerate CSR->Ell,Hybrid conversions on CUDA ([#313](#313), [#318](#318)) + Fixed slowdown of COO SpMV on OpenMP ([#340](#340)) + Fix gcc 6.4.0 internal compiler error ([#316](#316)) + Fix compilation issue on Apple clang++ 10 ([#322](#322)) + Make Ginkgo able to compile on Intel 2017 and above ([#337](#337)) + Make the benchmarks spmv/solver use the same matrix formats ([#366](#366)) + Fix self-written isfinite function ([#348](#348)) + Fix Jacobi issues shown by cuda-memcheck Tools and ecosystem: + Multiple improvements to the CI system and tools ([#296](#296), [#311](#311), [#365](#365)) + Multiple improvements to the Ginkgo containers ([#328](#328), [#361](#361)) + Add sonarqube analysis to Ginkgo ([#304](#304), [#308](#308), [#309](#309)) + Add clang-tidy and iwyu support to Ginkgo ([#298](#298)) + Improve Ginkgo's support of xSDK M12 policy by adding the `TPL_` arguments to CMake ([#300](#300)) + Add support for the xSDK R7 policy ([#325](#325)) + Fix examples in html documentation ([#367](#367))

The Ginkgo team is proud to announce the new minor release of Ginkgo version 1.1.0. This release brings several performance improvements, adds Windows support, adds support for factorizations inside Ginkgo and a new ILU preconditioner based on ParILU algorithm, among other things. For detailed information, check the respective issue. Supported systems and requirements: + For all platforms, cmake 3.9+ + Linux and MacOS + gcc: 5.3+, 6.3+, 7.3+, 8.1+ + clang: 3.9+ + Intel compiler: 2017+ + Apple LLVM: 8.0+ + CUDA module: CUDA 9.0+ + Windows + MinGW and Cygwin: gcc 5.3+, 6.3+, 7.3+, 8.1+ + Microsoft Visual Studio: VS 2017 15.7+ + CUDA module: CUDA 9.0+, Microsoft Visual Studio + OpenMP module: MinGW or Cygwin. The current known issues can be found in the [known issues page](https://github.com/ginkgo-project/ginkgo/wiki/Known-Issues). ### Additions + Upper and lower triangular solvers ([#327](#327), [#336](#336), [#341](#341), [#342](#342)) + New factorization support in Ginkgo, and addition of the ParILU algorithm ([#305](#305), [#315](#315), [#319](#319), [#324](#324)) + New ILU preconditioner ([#348](#348), [#353](#353)) + Windows MinGW and Cygwin support ([#347](#347)) + Windows Visual Studio support ([#351](#351)) + New example showing how to use ParILU as a preconditioner ([#358](#358)) + New example on using loggers for debugging ([#360](#360)) + Add two new 9pt and 27pt stencil examples ([#300](#300), [#306](#306)) + Allow benchmarking CuSPARSE spmv formats through Ginkgo's benchmarks ([#303](#303)) + New benchmark for sparse matrix format conversions ([#312](#312)) + Add conversions between CSR and Hybrid formats ([#302](#302), [#310](#310)) + Support for sorting rows in the CSR format by column idices ([#322](#322)) + Addition of a CUDA COO SpMM kernel for improved performance ([#345](#345)) + Addition of a LinOp to handle perturbations of the form (identity + scalar * basis * projector) ([#334](#334)) + New sparsity matrix representation format with Reference and OpenMP kernels ([#349](#349), [#350](#350)) ### Fixes + Accelerate GMRES solver for CUDA executor ([#363](#363)) + Fix BiCGSTAB solver convergence ([#359](#359)) + Fix CGS logging by reporting the residual for every sub iteration ([#328](#328)) + Fix CSR,Dense->Sellp conversion's memory access violation ([#295](#295)) + Accelerate CSR->Ell,Hybrid conversions on CUDA ([#313](#313), [#318](#318)) + Fixed slowdown of COO SpMV on OpenMP ([#340](#340)) + Fix gcc 6.4.0 internal compiler error ([#316](#316)) + Fix compilation issue on Apple clang++ 10 ([#322](#322)) + Make Ginkgo able to compile on Intel 2017 and above ([#337](#337)) + Make the benchmarks spmv/solver use the same matrix formats ([#366](#366)) + Fix self-written isfinite function ([#348](#348)) + Fix Jacobi issues shown by cuda-memcheck ### Tools and ecosystem improvements + Multiple improvements to the CI system and tools ([#296](#296), [#311](#311), [#365](#365)) + Multiple improvements to the Ginkgo containers ([#328](#328), [#361](#361)) + Add sonarqube analysis to Ginkgo ([#304](#304), [#308](#308), [#309](#309)) + Add clang-tidy and iwyu support to Ginkgo ([#298](#298)) + Improve Ginkgo's support of xSDK M12 policy by adding the `TPL_` arguments to CMake ([#300](#300)) + Add support for the xSDK R7 policy ([#325](#325)) + Fix examples in html documentation ([#367](#367)) Related PR: #370

tcojean requested review from thoasm, hartwiganzt, pratikvn and yhmtsai October 14, 2019 18:02

tcojean self-assigned this Oct 14, 2019

tcojean added this to the v1.1.0 milestone Oct 14, 2019

tcojean added reg:ci-cd This is related to the continuous integration system. is:enhancement An improvement of an existing feature. 1:ST:ready-for-review This PR is ready for review labels Oct 14, 2019

tcojean force-pushed the ci_improvements branch from 3e7ff65 to a4caa7a Compare October 14, 2019 18:26

tcojean added 2 commits October 14, 2019 20:56

Looks like Intel 19 is only compatible with GNU 8.

00a2db3

tcojean force-pushed the ci_improvements branch from 4bac100 to 00a2db3 Compare October 14, 2019 18:57

hartwiganzt approved these changes Oct 14, 2019

View reviewed changes

yhmtsai approved these changes Oct 14, 2019

View reviewed changes

dev_tools/containers/README.md Show resolved Hide resolved

dev_tools/containers/build_all_containers.sh Show resolved Hide resolved

yhmtsai approved these changes Oct 15, 2019

View reviewed changes

tcojean force-pushed the ci_improvements branch 4 times, most recently from b8d8214 to f0472c4 Compare October 17, 2019 09:26

Disable codecov as it as completely wrong report results.

ee6f233

tcojean force-pushed the ci_improvements branch from f0472c4 to ee6f233 Compare October 17, 2019 11:47

tcojean merged commit 9d16be7 into develop Oct 17, 2019

tcojean deleted the ci_improvements branch October 17, 2019 13:48

tcojean added 1:ST:ready-to-merge This PR is ready to merge. and removed 1:ST:ready-for-review This PR is ready for review labels Oct 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve the CI container setup #365

Improve the CI container setup #365

tcojean commented Oct 14, 2019 •

edited

Loading

codecov bot commented Oct 14, 2019 •

edited

Loading

yhmtsai left a comment

tcojean commented Oct 17, 2019

Improve the CI container setup #365

Improve the CI container setup #365

Conversation

tcojean commented Oct 14, 2019 • edited Loading

Overview

codecov bot commented Oct 14, 2019 • edited Loading

Codecov Report

yhmtsai left a comment

Choose a reason for hiding this comment

tcojean commented Oct 17, 2019

tcojean commented Oct 14, 2019 •

edited

Loading

codecov bot commented Oct 14, 2019 •

edited

Loading