[CORE-2841] Transforms: Start consuming from an arbitrary offset (numeric from start/end or timestamp) #19975

oleiman · 2024-06-24T21:09:00Z

This PR wires up the ability to configure the start offset of a transform at deploy time. This can be either a unix timestamp (ms since epoch) or an offset delta (+oo from start offset of -oo from end).

Includes rpk experience.

Backports Required

Release Notes

Improvements

Adds the ability to start transform processing from an arbitrary offset on the input topic.

src/v/transform/transform_processor.cc

oleiman · 2024-06-27T04:28:13Z

/ci-repeat 1

oleiman · 2024-07-02T02:21:48Z

force push contents:

serde shenanigans
adjust feature gate to guard new offset alternatives only

oleiman · 2024-07-02T02:28:10Z

force push make legacy_transform_offset_options struct write-only.

rockwotj

One small edge case to redeem otherwise LGTM

src/v/model/transform.cc

rockwotj · 2024-07-02T02:51:42Z

src/v/transform/transform_processor.cc

+                  },
+                  [this, &latest](model::transform_from_end off) {
+                      vlog(_logger.debug, "starting at offset: {}", off);
+                      auto actual_offset = latest - off.delta;


sigh. goood catch. thought i made a note somewhere to address that 😕

rockwotj · 2024-07-02T02:52:01Z

src/v/transform/transform_processor.cc

+                  },
+                  [this](model::transform_from_start off) {
+                      vlog(_logger.debug, "starting at offset: {}", off);
+                      auto actual_offset = _source->start_offset() + off.delta;


model::offset_delta resolves arithmetic operator overloads that perform automatic conversions between kafka and model offsets. This commit introduces an offset_delta that is used specifically for applying a numeric delta to an existing kafka::offset without adjusting its type. Useful for transform start offset calculations. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Thin wrappers around a kafka::offset_delta meant for use in a sum type to indicate how the delta should be applied - i.e. added to the start offset of a topic or subtracted from the latest offset of a topic. Enclosed delta should always be greater than or equal to zero. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

As a consequence of this, to maintain compatibility during partial upgrades, introduce a legacy_transform_offset_options. This gives us the ability to feature gate use of the new offset alternatives and avoid writing non-forwards-compatible transform metadata in case a partial upgrade needs to be rolled back. This commit also adds transform_metadata::serde_write, which conditionally writes the legacy version of the offset options struct iff the position variant holds one of the legacy alternatives. This code can be removed after v24.3 ships. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

{ "format": enum[timestamp, from_start, from_end], "value": int64 } Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Specifically, we want to prevent the use of the new offset options alternatives before an upgrade has been finalized. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman · 2024-07-02T17:33:28Z

force push a dt test case

r-vasquez

LGTM, just one minor comment on the flag documentation and one question on the default value for the from-offset format.

src/go/rpk/pkg/cli/transform/deploy.go

r-vasquez · 2024-07-02T17:42:09Z

src/go/rpk/pkg/cli/transform/deploy.go

+	format := ""
+	switch pfx := formatted_offset[0:1]; pfx {
+	case "@":
+		format = "timestamp"
+	case "+":
+		format = "from_start"
+	case "-":
+		format = "from_end"
+	default:
+		return nil, fmt.Errorf("Bad prefix: expected one of ['@','+','-'], got: '%s'", pfx)
+	}


[Question] I do not have a strong opinion on this, but, can we default the format to from_start?

The flag reads:

Process an input topic partition from this offset

If I use the flag as:

rpk transform deploy --from-offset 30

It reads ok (I will deploy from offset 30) but it will fail with: Bad prefix: expected one of ['@','+','-'], got: '3'.

We could check if the prefix case is + or a number.

I'd caution against this because we really don't offer the ability to deploy from a particular offset, only incidentally if the start offset of every partition is 0.

I'm sort of okay with the UX speedbump if it forces a user back to the help text so they know exactly what to expect. I'm not exactly the ideal source of UX opinions, but I think this feature will be used rarely enough to benefit from high specificity.

If somebody does have a strong opinion about this though, I'm happy to make the change.

src/go/rpk/pkg/cli/transform/deploy.go

@t

--from-offset to start from this offset * @t: start from UNIX timestamp (ms from epoch) * +oo: start offset + oo * -oo: latest ofset - oo Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

- Consume records that were produced before the deploy - Specify offsets that run off the end of the input topic - Ill-formed offsets Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman · 2024-07-02T21:04:47Z

force push to address some CR comments on the rpk side

oleiman · 2024-07-03T15:12:18Z

CI Failure is #20574 (known, unrelated)

dotnwat · 2024-07-03T16:10:26Z

src/v/model/fundamental.h

+inline constexpr offset operator+(offset o, offset_delta d) {
+    if (o >= offset{0}) {
+        if (d() <= offset::max() - o) {
+            return offset{o() + d()};
+        } else {
+            return offset::max();
+        }
+    } else {
+        if (d() >= offset::min() - o) {
+            return offset{o() + d()};
+        } else {
+            return offset::min();
+        }
+    }
+}
+
+inline constexpr offset operator-(offset o, offset_delta d) { return o + (-d); }


@andijcr given the recent issue with named_type strictness (with the PR you had to address it), are these overloads scary?

I'm not familiar...do you have a pointer?

#15170 sorry i missed the tag.
i need to have a look, i think we already have some overloads for these operators and types...

Ah, got it thanks. That looks like a good change 👍

For context on this diff: kafka::offset_delta is new. the idea is to distinguish from model::offset_delta where the operator overloads do conversion back/forth between kafka::offset & model::offset.

as for the overload resolution itself, I think we're fine because the desired one is more specific? like this:
https://godbolt.org/z/EfG7Yx5vc

dotnwat · 2024-07-03T16:11:50Z

src/v/model/transform.cc

+struct legacy_transform_offset_options
+  : serde::envelope<
+      transform_offset_options,


the first envelope template parameter looks wrong (CRTP)

Good catch, thank you 🙏
#20828

oleiman self-assigned this Jun 24, 2024

github-actions bot added area/rpk area/redpanda area/wasm WASM Data Transforms labels Jun 24, 2024

rockwood-openai reviewed Jun 24, 2024

View reviewed changes

src/v/transform/transform_processor.cc Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

oleiman changed the title ~~Transforms: Start consuming from an arbitrary offset (numeric from start/end or timestamp)~~ [CORE-2841] Transforms: Start consuming from an arbitrary offset (numeric from start/end or timestamp) Jun 24, 2024

oleiman force-pushed the xform/core-2841/specify-start-offset branch 5 times, most recently from b177085 to 78e09c0 Compare June 25, 2024 17:26

oleiman marked this pull request as ready for review June 25, 2024 17:28

oleiman requested review from a team, twmb, r-vasquez, gene-redpanda and Deflaimun as code owners June 25, 2024 17:28

oleiman requested review from a team and michael-redpanda and removed request for a team June 25, 2024 17:29

oleiman force-pushed the xform/core-2841/specify-start-offset branch from 78e09c0 to 612cd16 Compare June 25, 2024 18:01

oleiman force-pushed the xform/core-2841/specify-start-offset branch from 612cd16 to 2d34443 Compare June 27, 2024 19:13

oleiman added doc-needed labels Jun 27, 2024

oleiman requested review from rockwood-openai and pgellert and removed request for rockwood-openai June 27, 2024 19:15

oleiman requested a review from rockwotj July 2, 2024 02:12

oleiman force-pushed the xform/core-2841/specify-start-offset branch from f743216 to f6f1e38 Compare July 2, 2024 02:27

rockwotj previously approved these changes Jul 2, 2024

View reviewed changes

oleiman added 6 commits July 1, 2024 22:36

x/api: Respect user-specified start offset at deploy time

97eae1b

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

admin/x: Add 'offset' to deploy request

1a99989

{ "format": enum[timestamp, from_start, from_end], "value": int64 } Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

features: introduce feature spec for new transform offset options

25f3852

Specifically, we want to prevent the use of the new offset options alternatives before an upgrade has been finalized. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman dismissed rockwotj’s stale review via e1c3d29 July 2, 2024 15:14

oleiman force-pushed the xform/core-2841/specify-start-offset branch from f6f1e38 to e1c3d29 Compare July 2, 2024 15:14

pgellert previously approved these changes Jul 2, 2024

View reviewed changes

oleiman dismissed pgellert’s stale review via 497bef4 July 2, 2024 17:32

oleiman force-pushed the xform/core-2841/specify-start-offset branch from e1c3d29 to 497bef4 Compare July 2, 2024 17:32

r-vasquez reviewed Jul 2, 2024

View reviewed changes

oleiman added 3 commits July 2, 2024 13:49

rpk/x/deploy: Add flag to read from an arbitrary offset

8734326

--from-offset to start from this offset * @t: start from UNIX timestamp (ms from epoch) * +oo: start offset + oo * -oo: latest ofset - oo Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

dt/rpk: Add offset parameter to deploy_wasm

6dd0eae

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

dt/xform: Add tests for 'from-offset' option

5d03998

- Consume records that were produced before the deploy - Specify offsets that run off the end of the input topic - Ill-formed offsets Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman force-pushed the xform/core-2841/specify-start-offset branch from 497bef4 to 5d03998 Compare July 2, 2024 21:04

oleiman requested review from rockwotj and r-vasquez July 2, 2024 21:12

r-vasquez approved these changes Jul 2, 2024

View reviewed changes

pgellert approved these changes Jul 3, 2024

View reviewed changes

aanthony-rp merged commit 64090fc into redpanda-data:dev Jul 3, 2024
23 of 26 checks passed

dotnwat reviewed Jul 3, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CORE-2841] Transforms: Start consuming from an arbitrary offset (numeric from start/end or timestamp) #19975

[CORE-2841] Transforms: Start consuming from an arbitrary offset (numeric from start/end or timestamp) #19975

oleiman commented Jun 24, 2024 •

edited

Loading

This comment was marked as resolved.

oleiman commented Jun 27, 2024

oleiman commented Jul 2, 2024

oleiman commented Jul 2, 2024

rockwotj left a comment

rockwotj Jul 2, 2024

oleiman Jul 2, 2024

rockwotj Jul 2, 2024

oleiman commented Jul 2, 2024

r-vasquez left a comment

r-vasquez Jul 2, 2024

oleiman Jul 2, 2024

oleiman commented Jul 2, 2024

oleiman commented Jul 3, 2024

dotnwat Jul 3, 2024

oleiman Jul 3, 2024

andijcr Jul 4, 2024

oleiman Jul 4, 2024

dotnwat Jul 3, 2024

oleiman Jul 3, 2024

[CORE-2841] Transforms: Start consuming from an arbitrary offset (numeric from start/end or timestamp) #19975

[CORE-2841] Transforms: Start consuming from an arbitrary offset (numeric from start/end or timestamp) #19975

Conversation

oleiman commented Jun 24, 2024 • edited Loading

Backports Required

Release Notes

Improvements

This comment was marked as resolved.

oleiman commented Jun 27, 2024

oleiman commented Jul 2, 2024

oleiman commented Jul 2, 2024

rockwotj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleiman commented Jul 2, 2024

r-vasquez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleiman commented Jul 2, 2024

oleiman commented Jul 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleiman commented Jun 24, 2024 •

edited

Loading