js/execution: Stage identifier

[codebien]

Feature Description

As part of the journey to achieve #796, we decided to start by giving the ability for doing it from the JS code. One of the main steps for doing it, it would be to detect which is the current Stage for the running iteration.

Proposal

The proposal is to add a Stage's identifier information. We would add a new stage object to the already existing scenario object as part of the js/execution module. It would export the exec.scenario.stage.id property that returns a zero-based value, that would track the index of the current running Stage.

The following example shows how to would be used for tracking the stage as a tag, invoking the already implemented tags setter #2172:

import http from 'k6/http';
import exec from 'k6/execution';

export const options = {
  stages: [
    { duration: '1m', target: 10 },
    { duration: '1m', target: 20 },
    { duration: '1m', target: 0 },
  ],
};

export default function () {
  exec.vu.tags["stage"] = 'stage#${exec.scenario.stage.id}'
  http.get('http://test.k6.io/');
}

Known problem

After a PoC, we have identified that the main problem for this, would be how to return an accurate value, especially for the ramping-vus executor. Because the current implementation of the ramping-vus doesn't have a specific event during execution where an iteration is going to be defined as part of a Stage. It pre-computes them and it will totally lose the connection with the defined Scenario's stages.

For this reason, the proposed workaround, it's to recognize the current Stage computing the elapsed time from the exec.scenario.startTime property then find the Stage that contains it. It has the benefit that the implementation is simple but in some circumstances, it could be not accurate (e.g when the iteration uses the gracefulRampDown period).

Alternatives

Currently, the identified alternatives are:

• An intermediate step (or as an experimental API) we could consider adding the same logic as a helper function being part of the jslib.
• Add an example to the documentation of how to compute it from JS then let users copy/customize/do it.

@grafana/k6-devs let's discuss if we have more alternatives and/or better solutions.

[codebien]

@mstoykov re-starting the discussion from #2199 (comment). The plan was to have this as a foundation for achieving #796, so the main con to not including it in the core or having it as a jslib func is the fact that we couldn't really close that feature. Do we have alternatives?

[mstoykov]

I am still very much against exec.scenario.stage.number==exec.scenario.stage.number being able to return false. IMO if we ask "in which the stage we are in", it should not be answered based on which stage is currently running. And if it is, it should be from a function so at least the above should be ... less surprising. But I don't think neither one of those is really necessary.

In the case that this is interesting this is easy to calculate with some js (possibly hosted in jslib), the stages, now and when the scenario started(as has been established). AFAIK the only thing that is currently hard to get is the stages ? Although even that isn't true as you can see in #2278, but also arguably shouldn't depend on having access to the options and the options value being accurate ;).

So to me it seems that working on #2259 to export the final value for options or on execution.scenario.stages being a thing(possibly different name and/or additional info as well) will be much better is the way forward. That has other benefits (as noted in that issue) and IMO won't be confusing as the currently proposed solution is ... again IMO.

And if we have execution.iteration.startTime (which value we already have as we time the iteration duration based on it) we can calculate the "in which stage did this start" with some amount of accuracy.

[codebien]

@mstoykov so what you're proposing should be what is provided by #796 (comment) with some additional improvements:

• provide it as js func hosted by jslib
• get the scenario's stages from the execution API for avoiding a data race
• compute the elapsed time from the scenario.startTime using a static iteration startTime instead of the "current time"

Is there a difference between execution.iteration.startTime and ⬇️ ? If not, we could avoid exporting it and ask the user to pass the iterStartTime.

export default function() {
    let iterStartTime = new Date()
    ...
}

[mstoykov]

Sorry, for the slow reply - yes this is more or less what I propose.

I guess startTime will be more accurate, and likely a thing we would like to add(I think), but we can start with requiring the time and move away from that later as well 🤷

[codebien]

In the case, we would add the stage's name then we could support an auto-tagging function:

export default function() {
    tagByStage(new Date())
    ...
}

function tagByStage(iterStartTime) { // or just tagByStage() in the case we add exec.iteration.startTime
   exec.vu.tags['stage'] = getStage(iterStartTime).name
}

It would require to add (in order of priority):

• execution API for exec.scenario.stages or exec.options.stages and exec.options.scenarios[i].stages
• getStage jslib function
• add name for stage
• tagByStage jslib function
• exec.iteration.startTime getter function

[na--]

I am still very much against exec.scenario.stage.number==exec.scenario.stage.number being able to return false. IMO if we ask "in which the stage we are in", it should not be answered based on which stage is currently running.

I think this is a mostly academic concern... (new Date()) == (new Date()) can also return false in JS, but nobody is making these comparisons, right? If the property name is something like exec.scenario.activeStage, then I think any potential user confusion will be quite minimal.

When I ask myself the question "in which stage is the second iteration of the first VU going to be?":

import http from 'k6/http';
import exec from 'k6/execution';
import { sleep } from 'k6';

export const options = {
    stages: [
        { duration: '5s', target: 10 },
        { duration: '1m', target: 10 },
    ],
};

export default function () {
    console.log(`[t=${(new Date()) - exec.scenario.startTime}ms] VU{${exec.vu.idInTest}} started iteration ${exec.scenario.iterationInTest}`);
    http.get(`https://httpbin.test.k6.io/delay/${(1 + 4 * Math.random()).toFixed(2)}`)
    http.get(`https://httpbin.test.k6.io/delay/${(1 + 2 * Math.random()).toFixed(2)}`)
    sleep(0.5);
}

My answer is "it depends on what you mean 🤷‍♂️"... It's almost a meaningless question unless you know precisely what you want to measure. To set the right threshold, you might care when the iteration starts, or you might care about how many other VUs are currently running (and making requests) at the exact time as the current request your VU is making.

Anyway, exec.scenario.stage.number (or whatever we call it) can basically be computed purely in JavaScript, so I am fine if we don't proceed with that approach.

And if we have execution.iteration.startTime (which value we already have as we time the iteration duration based on it) we can calculate the "in which stage did this start" with some amount of accuracy.

I think we have that value only here:

k6/js/runner.go

Line 790 in 737dfa7

startTime := time.Now()

And we don't have k6/execution.iteration yet... I think it may be worth trying to make a PoC PR to see how difficult it will be to thread it so it's accessible through k6/execution... Because if it turns out we need another Context layer, or a bunch of extra arguments everywhere, then it's definitely a 👎 from me. I am not sure exactly what you mean by "some amount of accuracy", since I the accuracy of the exec.iteration.startTime property will be basically the same as this, just slightly more convenient:

export default function() {
    var startTime = new Date();
    ...
}

So... yeah, if someone thinks a PoC will be quick, I am fine with trying it, but 👍 for starting by passing the JS time.

It would require to add (in order of priority):

• execution API for exec.scenario.stages or exec.options.stages and exec.options.scenarios[i].stages
• getStage jslib function
• add name for stage
• tagByStage jslib function
• exec.iteration.startTime getter function

This mostly LGTM, with a couple of comments:

• exec.scenario.stages seems better than exec.options.stages, however it might be even better if we work on k6/execution: add more information about the current test run #2259 and export the consolidated and derived test options through a test.options property in k6/execution... 🤔 Then a JS script can always get the stages array for ramping-vus and ramping-arrival-rate scenarios by exec.test.options.scenarios[exec.scenario.name].stages.
• if we allow users to specify a name for every stage but don't start automatically tagging metrics with it, then it will be a breaking change if we decide to start doing that in the future 😞 so we kind of have to decide this before we allow users to specify name for stages

[codebien]

This is not going to be implemented as part of the execution API so I'm closing this and we will discuss more details in an eventual PR. For the current plan check: #796 (comment)