Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup tests #177

Merged
merged 6 commits into from
Feb 13, 2020
Merged

Cleanup tests #177

merged 6 commits into from
Feb 13, 2020

Conversation

res0nance
Copy link
Contributor

  • Cleanup some tests with using the older now NOOPed doYank() method
  • Rewrite the delay calculation in RateLimitBranchPropertyTest to not be truncated.
  • Fix the delay to not be an overly bloated number

Running it in CI to try to reproduce all the test flakes

assertThat("At least the rate implied delay but no more than 500ms longer",
secondBuild.getStartTimeInMillis() - firstBuild.getStartTimeInMillis(),
allOf(
greaterThanOrEqualTo(60 * 60 / rate * 1000L - 200L),
lessThanOrEqualTo(60 * 60 / rate * 1000L * 500L)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The * in the less than condition from the comment seems to be a typo

@res0nance res0nance closed this Jan 31, 2020
@res0nance res0nance reopened this Jan 31, 2020
@res0nance res0nance closed this Jan 31, 2020
@res0nance res0nance reopened this Jan 31, 2020
@res0nance
Copy link
Contributor Author

I tested this locally and i got a mind boggling result

[ERROR] Failures:
[ERROR]   RateLimitBranchPropertyTest.rateLimitsBlockBuilds_maxRate:133->rateLimitsBlockBuilds:201 At least the rate implied delay but no more than 500ms longer
Expected: (a value equal to or greater than <3400L> and a value less than or equal to <4100L>)
     but: a value equal to or greater than <3400L> <1718L> was less than <3400L>
[ERROR]   RateLimitBranchPropertyTest.rateLimitsBlockBuilds_medRate:138->rateLimitsBlockBuilds:201 At least the rate implied delay but no more than 500ms longer
Expected: (a value equal to or greater than <7000L> and a value less than or equal to <7700L>)
     but: a value equal to or greater than <7000L> <5332L> was less than <7000L>
[INFO]
[ERROR] Tests run: 10, Failures: 2, Errors: 0, Skipped: 0

@@ -169,13 +170,14 @@ public void rateLimitsBlockBuilds(int rate) throws Exception {
);
assertThat(master.isInQueue(), is(false));
assertThat(master.getQueueItem(), nullValue());
assertThat(master.getBuilds().getLastBuild(), isNotNull());
long startTime = master.getBuilds().getLastBuild().getTimeInMillis();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably why it flakes all the time lastBuild is not null so it uses the time of the last build to throttle

Run lastBuild = job.getLastBuild();
if (lastBuild != null) {
// we don't mind if the project type allows concurrent builds
// we assume that the last build was started at a time that respects the rate
// thus the next build to start will start such that the time between starts matches the rate
// it doesn't matter if the duration of the build is longer than the time between starts
// only that we linearise the starts and keep a consistent minimum duration between *starts*
long timeSinceLastBuild = System.currentTimeMillis() - lastBuild.getTimeInMillis();
long betweenBuilds = property.getMillisecondsBetweenBuilds();
if (timeSinceLastBuild < betweenBuilds) {
LOGGER.log(Level.FINE, "{0} will be delayed for another {1}ms as it is {2}ms since "
+ "last build and ideal rate is {3}ms between builds",
new Object[]{
job.getFullName(),
betweenBuilds - timeSinceLastBuild,
timeSinceLastBuild,
betweenBuilds
}
);
return CauseOfBlockage.fromMessage(Messages._RateLimitBranchProperty_BuildBlocked(
new Date(lastBuild.getTimeInMillis() + betweenBuilds))
);
}
}

@res0nance res0nance marked this pull request as ready for review February 6, 2020 10:53
@res0nance res0nance closed this Feb 6, 2020
@res0nance res0nance reopened this Feb 6, 2020
@res0nance res0nance closed this Feb 6, 2020
@res0nance res0nance reopened this Feb 6, 2020
@res0nance res0nance closed this Feb 6, 2020
@res0nance res0nance reopened this Feb 6, 2020
@res0nance res0nance closed this Feb 6, 2020
@res0nance res0nance reopened this Feb 6, 2020
@res0nance res0nance closed this Feb 6, 2020
@res0nance res0nance reopened this Feb 6, 2020
@res0nance res0nance closed this Feb 7, 2020
@res0nance res0nance reopened this Feb 7, 2020
@bitwiseman
Copy link
Contributor

I watch with interest...

@res0nance
Copy link
Contributor Author

I watch with interest...

Now i always run into a different error. 😅

@res0nance
Copy link
Contributor Author

Huzzah it worked

Copy link
Contributor

@bitwiseman bitwiseman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, so you got it to succeed once. Does this mean the tests aren't flaky now?

@res0nance
Copy link
Contributor Author

It shouldn't be flaky as it stopped flaking on my local machine the one with

[ERROR] Failures:
[ERROR]   RateLimitBranchPropertyTest.rateLimitsBlockBuilds_maxRate:133->rateLimitsBlockBuilds:201 At least the rate implied delay but no more than 500ms longer
Expected: (a value equal to or greater than <3400L> and a value less than or equal to <4100L>)
     but: a value equal to or greater than <3400L> <1718L> was less than <3400L>
[ERROR]   RateLimitBranchPropertyTest.rateLimitsBlockBuilds_medRate:138->rateLimitsBlockBuilds:201 At least the rate implied delay but no more than 500ms longer
Expected: (a value equal to or greater than <7000L> and a value less than or equal to <7700L>)
     but: a value equal to or greater than <7000L> <5332L> was less than <7000L>
[INFO]
[ERROR] Tests run: 10, Failures: 2, Errors: 0, Skipped: 0

So at least for this flake it should stop, it was root caused in my previous comment as well. So I expect the flaking for this particular test to stop

@bitwiseman bitwiseman merged commit bd4003b into jenkinsci:master Feb 13, 2020
@res0nance res0nance deleted the test-fixes branch February 14, 2020 02:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants