[BEAM-14334] Remove remaining forkEvery 1 from all Spark tests and stop mixing unit tests with runner validations. #17662
Conversation
…op mixing unit tests with runner validations.
|
R: @aromanenko-dev |
|
|
||
| classpath = configurations.validatesRunner | ||
| testClassesDirs = files( | ||
| project(":sdks:java:core").sourceSets.test.output.classesDirs, | ||
| project(":runners:core-java").sourceSets.test.output.classesDirs, | ||
| ) | ||
| testClassesDirs += files(project.sourceSets.test.output.classesDirs) |
There was a problem hiding this comment.
All unit tests should be run as part of the test task
| maxParallelForks 4 | ||
| useJUnit { | ||
| includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner' | ||
| includeCategories 'org.apache.beam.runners.spark.UsesCheckpointRecovery' |
There was a problem hiding this comment.
as said above, tests of this custom category are normal unit tests and are already run during test. there's no runner validation for such a category
|
Run Spark ValidatesRunner |
|
Run Spark StructuredStreaming ValidatesRunner |
| @@ -48,6 +48,17 @@ configurations { | |||
| examplesJavaIntegrationTest | |||
| } | |||
|
|
|||
| def sparkTestProperties(overrides = [:]) { | |||
There was a problem hiding this comment.
Is it just refactoring (extract method)?
| @@ -113,10 +117,6 @@ test { | |||
| } | |||
|
|
|||
| maxParallelForks 4 | |||
| useJUnit { | |||
There was a problem hiding this comment.
Why it was removed?
There was a problem hiding this comment.
previously these unit tests were excluded during test runs because they failed. instead these were run as part of validatesRunnerStreaming.
I fixed them so they can be run as normal unit tests, so this exclusion isn't needed any more.
| maxParallelForks 4 | ||
| useJUnit { | ||
| includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner' | ||
| includeCategories 'org.apache.beam.runners.spark.UsesCheckpointRecovery' |
|
Run Spark ValidatesRunner |
1 similar comment
|
Run Spark ValidatesRunner |
|
Run Java PreCommit |
1 similar comment
|
Run Java PreCommit |
|
precommit issues are unrelated |
|
Agree, it's not related to this PR |
This is a follow up of #17406 fixing the same issue for all remaining testing tasks. That is basically the hadoop compatibility tests and the runner validations.
Additionally this removes the current intransparent mixing of unit tests and runner validations. All tests of the project are run as part of the
testtask (and the hadoop compatibility tests), whereas runner validations are defined intestingonly.Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.