Upload native build statistics #39784

zakkak · 2024-03-29T10:42:39Z

Since Quarkus CI already runs the native tests quite often with a fixed
Mandrel version, we can gather these data and increase our insight
regarding Quarkus changes that affects image build time statistics.

jerboaa · 2024-04-15T16:25:04Z

.github/import_stats.sh

+  # update timing info if present
+  if [ -e "$ts" ]; then
+    curl -s -w '\n' -H "Content-Type: application/json" -H "token: $TOKEN" \
+	    -X PUT --data "@$ts" "$URL/$stat_id" > /dev/null
+  fi


This shouldn't be needed as the build time is part of the latest build-output.jsons. It would be better to feed the quarkus version as an update with the new /update-runner-info endpoint.

👍 I am waiting for the new RunnerInfo to get deployed to collector.foci.life first (see Karm/collector#23 (comment))

OK. Either way the PUT call for timing info is redundant these days and should not be included.

@zakkak If you enable GHA on your Q fork, you can switch the endpoint to stage and run it of your own G account, right?

Yes, I will do this, and also enable the job to run on forks, it currently is supposed to run only on the main repo on the main branch.

gsmet · 2024-04-22T13:38:58Z

Note that you could use an approach similar to:

quarkus/.github/workflows/develocity-publish-build-scans.yml

Lines 33 to 39 in 824234e

    
                 - name: Output JSON file 
        
                   run: | 
        
                     if [ -f "${{ steps.setup.outputs.build-metadata-file-path }}" ]; then 
        
                       echo "```json" >> $GITHUB_STEP_SUMMARY 
        
                       jq '.' ${{ steps.setup.outputs.build-metadata-file-path }} >> $GITHUB_STEP_SUMMARY 
        
                       echo "\n```" >> $GITHUB_STEP_SUMMARY; 
        
                     fi

to also add it as a step summary (which allows to visualize things right away instead of downloading the file)

zakkak · 2024-04-23T12:32:46Z

to also add it as a step summary (which allows to visualize things right away instead of downloading the file)

That would be a json file per integration-test, wouldn't it be too much noise in the report? What do you think would make more sense to be shown in the report?

zakkak · 2024-04-24T20:07:20Z

This is blocked by Karm/collector#27

zakkak · 2024-05-27T13:01:37Z

This seems to work (tested with Main IT in https://github.com/zakkak/quarkus/actions/runs/9254781855)

@Karm please have a look.

jerboaa · 2024-05-27T13:38:34Z

.github/workflows/ci-actions-incremental.yml

+          {
+            "test_version": "$GITHUB_REF_NAME",
+            "quarkus_version": "$GITHUB_SHA",
+            "jdk_version": "$(java -version 2>&1 | head -n 2 | tail -n 1)",
+            "operating_system": "$RUNNER_OS",
+            "architecture": "$RUNNER_ARCH",
+            "memory_size_bytes": "$(awk '/MemTotal/ {print $2 * 1024}' /proc/meminfo)",
+            "memory_available_bytes": "$(awk '/MemAvailable/ {print $2 * 1024}' /proc/meminfo)",
+            "description": "Quarkus CI github runner on $GITHUB_REF_NAME branch/tag",
+            "triggered_by": "Quarkus CI"
+          }


Runner info extracted here might be different from the actual GHA runner that ran the native integration test, no? Should we not collect the info where the native tests ran?

The only thing I expect to actually change is memory_available_bytes as AFAIK all free-tier github runners have the same resources. So I am not sure if it's worth creating a new runner for each IT-category (since each category runs on its own runner). No strong opinions though.

So maybe we should keep the runner info empty (or some value indicating the values are somewhat meaningless), so as to avoid concluding something from the runner info. That's the only worry. The only info relevant for quarkus should be the test_version and quarkus_version, right?

So maybe we should keep the runner info empty (or some value indicating the values are somewhat meaningless), so as to avoid concluding something from the runner info.

That makes sense

The only info relevant for quarkus should be the test_version and quarkus_version, right?

Right, that should be enough for #40076

Done.

jerboaa

Looks OK to me.

.github/workflows/ci-actions-incremental.yml

Since Quarkus CI already runs the native tests quite often with a fixed Mandrel version, we can gather these data and increase our insight regarding Quarkus changes that affects image build time statistics.

quarkus-bot · 2024-05-28T09:05:35Z

Status for workflow `Quarkus CI`

This is the status report for running Quarkus CI on commit d470e57.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

You can consult the Develocity build scans.

Flaky tests - Develocity

⚙️ JVM Tests - JDK 21

📦 extensions/smallrye-reactive-messaging-kafka/deployment

✖ io.quarkus.smallrye.reactivemessaging.kafka.deployment.dev.KafkaDevServicesDevModeTestCase.sseStream - History

Assertion condition defined as a Lambda expression in io.quarkus.smallrye.reactivemessaging.kafka.deployment.dev.KafkaDevServicesDevModeTestCase Expecting size of: [] to be greater than or equal to 2 but was 0 within 10 seconds. - org.awaitility.core.ConditionTimeoutException

org.awaitility.core.ConditionTimeoutException: 
Assertion condition defined as a Lambda expression in io.quarkus.smallrye.reactivemessaging.kafka.deployment.dev.KafkaDevServicesDevModeTestCase 
Expecting size of:
  []
to be greater than or equal to 2 but was 0 within 10 seconds.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:119)
	at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:31)

📦 extensions/smallrye-reactive-messaging/deployment

✖ io.quarkus.smallrye.reactivemessaging.hotreload.ConnectorChangeTest.testUpdatingConnector - History

Expecting actual: ["-4","-5","-6","-7","-8","-9","-10","-11"] to start with: ["-3", "-4", "-5", "-6"] - java.lang.AssertionError

java.lang.AssertionError: 

Expecting actual:
  ["-4","-5","-6","-7","-8","-9","-10","-11"]
to start with:
  ["-3", "-4", "-5", "-6"]

	at io.quarkus.smallrye.reactivemessaging.hotreload.ConnectorChangeTest.testUpdatingConnector(ConnectorChangeTest.java:36)

📦 integration-tests/reactive-messaging-kafka

✖ io.quarkus.it.kafka.KafkaConnectorTest.testFruits - History

Assertion condition defined as a Lambda expression in io.quarkus.it.kafka.KafkaConnectorTest expected: <6> but was: <5> within 10 seconds. - org.awaitility.core.ConditionTimeoutException

org.awaitility.core.ConditionTimeoutException: Assertion condition defined as a Lambda expression in io.quarkus.it.kafka.KafkaConnectorTest expected: <6> but was: <5> within 10 seconds.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:119)
	at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:31)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1006)
	at org.awaitility.core.ConditionFactory.untilAsserted(ConditionFactory.java:790)
	at io.quarkus.it.kafka.KafkaConnectorTest.testFruits(KafkaConnectorTest.java:63)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)

Karm

Seems sane to me and consistent with this use case: https://github.com/Karm/mandrel-integration-tests/blob/bfb82325b3d3a4838a0e5567d77097ad03a22b36/aux_scripts/run_qts_host.sh

@zakkak The only nitpick would be some perhaps needlessly missing attributes from runner info, e.g.

  "operating_system": "$(uname -o)",
  "architecture": "$(uname -m)",
  "memory_size_bytes": $(awk '/MemTotal/ {print $2 * 1024}' /proc/meminfo),
  "memory_available_bytes": $(awk '/MemAvailable/ {print $2 * 1024}' /proc/meminfo),

zakkak · 2024-05-28T13:28:57Z

@zakkak The only nitpick would be some perhaps needlessly missing attributes from runner info, e.g.

Yes, we had that in place but as suggested by @jerboaa in #39784 (comment) I agree it is better we remove them given that they don't come from the actual runner (the runner uploading the data is different than the runner that did the actual runners).

If you think we really need this info I can modify it (in a follow up PR) to use a runner-info per IT-category and report the actual numbers.

Follow up to quarkusio#39784 Resolves: ``` jq: error: Could not open file /home/runner/work/quarkus/quarkus/./integration-tests/simple: No such file or directory ``` by treating each line returned by `find` as a single path instead of further breaking it down by space.

quarkus-bot bot added the area/infra-automation anything related to CI, bots, etc. that are used to automated our infrastructure label Mar 29, 2024

zakkak mentioned this pull request Apr 15, 2024

Rework the way we currently detect regressions in build time metrics #40076

Open

jerboaa reviewed Apr 15, 2024

View reviewed changes

zakkak force-pushed the 2024-03-29-upload-native-build-stats branch from dddcfe0 to 70ddd16 Compare April 22, 2024 13:17

zakkak force-pushed the 2024-03-29-upload-native-build-stats branch 3 times, most recently from 8fd0d3b to 9368eb7 Compare April 23, 2024 12:16

zakkak force-pushed the 2024-03-29-upload-native-build-stats branch 2 times, most recently from 6c87de6 to ec781e2 Compare April 24, 2024 05:53

This comment has been minimized.

Sign in to view

zakkak force-pushed the 2024-03-29-upload-native-build-stats branch 3 times, most recently from 724d42d to 3c4133b Compare April 24, 2024 06:11

This comment has been minimized.

Sign in to view

zakkak force-pushed the 2024-03-29-upload-native-build-stats branch 4 times, most recently from ad6c821 to 42d9ba9 Compare April 24, 2024 07:00

This comment has been minimized.

Sign in to view

zakkak force-pushed the 2024-03-29-upload-native-build-stats branch 3 times, most recently from ae8263e to d41c47c Compare May 13, 2024 13:21

zakkak force-pushed the 2024-03-29-upload-native-build-stats branch from d41c47c to c23eb7f Compare May 27, 2024 07:52

This comment has been minimized.

Sign in to view

zakkak force-pushed the 2024-03-29-upload-native-build-stats branch 2 times, most recently from 959576e to 5fa39fb Compare May 27, 2024 07:53

zakkak force-pushed the 2024-03-29-upload-native-build-stats branch 3 times, most recently from 661ee75 to 61ae566 Compare May 27, 2024 11:50

This comment has been minimized.

Sign in to view

zakkak force-pushed the 2024-03-29-upload-native-build-stats branch from 61ae566 to 874d587 Compare May 27, 2024 12:25

This comment has been minimized.

Sign in to view

zakkak force-pushed the 2024-03-29-upload-native-build-stats branch from 874d587 to 61380b8 Compare May 27, 2024 13:00

zakkak marked this pull request as ready for review May 27, 2024 13:00

jerboaa reviewed May 27, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

quarkus-bot bot added the triage/flaky-test label May 27, 2024

zakkak force-pushed the 2024-03-29-upload-native-build-stats branch from 61380b8 to 2512ca5 Compare May 27, 2024 15:27

zakkak requested a review from jerboaa May 27, 2024 15:27

jerboaa approved these changes May 27, 2024

View reviewed changes

jerboaa reviewed May 27, 2024

View reviewed changes

.github/workflows/ci-actions-incremental.yml Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

Upload native build statistics

d470e57

Since Quarkus CI already runs the native tests quite often with a fixed Mandrel version, we can gather these data and increase our insight regarding Quarkus changes that affects image build time statistics.

zakkak force-pushed the 2024-03-29-upload-native-build-stats branch from 2512ca5 to d470e57 Compare May 28, 2024 07:26

jerboaa approved these changes May 28, 2024

View reviewed changes

Karm approved these changes May 28, 2024

View reviewed changes

zakkak merged commit af19dfc into quarkusio:main May 28, 2024
52 checks passed

zakkak deleted the 2024-03-29-upload-native-build-stats branch May 28, 2024 13:29

quarkus-bot bot added this to the 3.12 - main milestone May 28, 2024

zakkak mentioned this pull request May 29, 2024

Fix native build stats uploading for "simple with space" IT #40877

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upload native build statistics #39784

Upload native build statistics #39784

zakkak commented Mar 29, 2024

jerboaa Apr 15, 2024

zakkak Apr 17, 2024

jerboaa Apr 17, 2024

Karm Apr 18, 2024 •

edited

Loading

zakkak Apr 18, 2024 •

edited

Loading

gsmet commented Apr 22, 2024

zakkak commented Apr 23, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

zakkak commented Apr 24, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

zakkak commented May 27, 2024

jerboaa May 27, 2024

zakkak May 27, 2024

jerboaa May 27, 2024

zakkak May 27, 2024 •

edited

Loading

This comment has been minimized.

jerboaa left a comment

This comment has been minimized.

quarkus-bot bot commented May 28, 2024 •

edited by github-actions bot

Loading

Karm left a comment

zakkak commented May 28, 2024

Upload native build statistics #39784

Upload native build statistics #39784

Conversation

zakkak commented Mar 29, 2024

jerboaa Apr 15, 2024

Choose a reason for hiding this comment

zakkak Apr 17, 2024

Choose a reason for hiding this comment

jerboaa Apr 17, 2024

Choose a reason for hiding this comment

Karm Apr 18, 2024 • edited Loading

Choose a reason for hiding this comment

zakkak Apr 18, 2024 • edited Loading

Choose a reason for hiding this comment

gsmet commented Apr 22, 2024

zakkak commented Apr 23, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

zakkak commented Apr 24, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

zakkak commented May 27, 2024

jerboaa May 27, 2024

Choose a reason for hiding this comment

zakkak May 27, 2024

Choose a reason for hiding this comment

jerboaa May 27, 2024

Choose a reason for hiding this comment

zakkak May 27, 2024 • edited Loading

Choose a reason for hiding this comment

This comment has been minimized.

jerboaa left a comment

Choose a reason for hiding this comment

This comment has been minimized.

quarkus-bot bot commented May 28, 2024 • edited by github-actions bot Loading

Status for workflow Quarkus CI

Flaky tests - Develocity

⚙️ JVM Tests - JDK 21

📦 extensions/smallrye-reactive-messaging-kafka/deployment

📦 extensions/smallrye-reactive-messaging/deployment

📦 integration-tests/reactive-messaging-kafka

Karm left a comment

Choose a reason for hiding this comment

zakkak commented May 28, 2024

Karm Apr 18, 2024 •

edited

Loading

zakkak Apr 18, 2024 •

edited

Loading

zakkak May 27, 2024 •

edited

Loading

quarkus-bot bot commented May 28, 2024 •

edited by github-actions bot

Loading

Status for workflow `Quarkus CI`