Provide scaler for NATS Jetstream

[shaunc]

Proposal

Currently keda supports NATS streaming. However, NATS Streaming has been deprecated in favor of NATS Jetstream. NATS is simpler to run in-cluster than Kafka, and jetstream scales well. I hope that Keda will support it?

Use-Case

I hope to use in conjunction with fission

Anything else?

No response

[JorTurFer]

Hi @shaunc
Thanks for notify this. I think that KEDA should support NATS Jetstream being this the successor of NATS Streaming.
Are you willing to contribute with this?

[shaunc]

Thank you, @JorTurFer!

I'm not familiar with the codebase, and have many things to do. That said, getting this working would simplify a number of other things (like having to maintain a Kafka cluster). If you could give some pointers/a checklist, I can investigate, and depending on how much work there is make some contributions if not the full feature.

[JorTurFer]

hi @shaunc ,
As I understand this, it's basically another new scaler (because STAN is deprecated but still be there). This means that this guide could be useful.
Probably the majority of the code will be similar to already existing STAN scaler, also from unit/e2e tests
Thanks in advance in any case :)

[goku321]

I can take this up in coming weeks

[JorTurFer]

Thanks!!
I'll assign it to you just for having a reference person.
Don't be afraid about request the reassignment if you finally don't do it

[shaunc]

Very good! ... (currently very busy :( )....

[xgenvn]

@goku321 do you have a plan or task board, and repository to contribute on? I think there are some people having the same interest in this.

[goku321]

Hi @xgenvn I didn't plan anything yet. I believe it will be like any other scalers that we already have. I will start on this today/tomorrow and will try to keep you in loop.

[xgenvn]

@goku321 I did a check on the scalers of STAN and a bit on the JetStream metrics endpoint. It seems there are many differences with /jsz endpoint compared to /channelsz. If we can conclude the IsActive implementation which currently has hasPendingMessage, the implementation should be straight-forward.

[leopoldodonnell]

@xgenvn I'm interested in being able to scale using JetStream metrics as well. Is this something that is being considered, if so what is the timeframe?

Re the /jsz endpoint, which specific Nats server issue were you referencing. We might be willing to do some work here.

[goku321]

@leopoldodonnell I've started working on the scaler. Will try my best to conclude it in a couple of weeks. And, yes the scaler will consider the JetStream metrics

[leopoldodonnell]

@goku321 No pressure (really) Just checking to see if you were able to make any progress towards a JetStream scaler?

[goku321]

@leopoldodonnell @xgenvn Got delayed due to some pre-commit issues

Here's the commit: goku321@126130b (Didn't test it yet)

I'm not sure how accurate is getMaxMsgLag

Would really appreciate your input and help 🙂

[leopoldodonnell]

@goku321 I will build it and have a look - its not my top priority, but I'd like to see this working and really appreciate that someone is giving it a try.

[gcameo]

Any updates?

[tomkerkhove]

Let's see where @goku321 is at and if he wants to contribute it.

[goku321]

@gcameo @tomkerkhove I still want to contribute. I had already started with the tests couple of weeks back but a bit caught up with production deployments these days. Will try to wrap up in the coming weekends or so. Appreciate your patience :)

[JorTurFer]

hey @goku321
Any update? we aim to do a release next week

[goku321]

Hi @JorTurFer In the process of testing the scaler. Will try to create a PR by today.

[goku321]

Hi all. Can anyone from this thread with JetStream experience please review this scaler (#3468)? Would really appreciate the help :)