v23.3.19

Features

• #22600 Schema Registry: Support normalize=true by @BenPope in #22635

Bug Fixes

• Fix timequery failing when requested timestamp is in the future and local log is empty. by @nvartolomei in #21321
• Fixes a bug in the purger, which when used with ABS may attempt to download a directory Blob as a manifest. This would completely block the purger from purging partitions for a deleted topic. by @WillemKauf in #22543
• Fixes a bug in which the abs_parse_impl used in xml_sax_parser would throw an exception when processing a List Blobs response with a <BlobPrefix> element within <Blobs>, but without any <Blob> elements. by @WillemKauf in #22540
• Kafka client used by audit system will now properly handle DNS lookup failure errors by @michael-redpanda in #22526
• Schema Registry: Fix Swagger for POST /compatibility/subjects/{subject}/versions/{version} by @BenPope in #22647
• #21253 Fixed a crash during raft snapshot application (prefix truncation of the log). A race condition between raft snapshot application and lagging state machine apply fibers caused the state machine offset to move backwards incorrectly. by @bharathv in #21255

Improvements

• Schema Registry: remove the schema from memory when the last subject version referencing it is deleted. by @pgellert in #21258
• #21296 HTTP Proxy: Avoid large allocations during JSON serde of requests and responses by @BenPope in #21469
• #21296 Schema Registry: Avoid large allocations during JSON serde of requests and responses by @BenPope in #21469
• #21467 Schema Registry: Avoid large contiguous allocations whilst handling schema. by @BenPope in #21557
• #22600 Schema Registry: Improve AVRO Normalization by @BenPope in #22635
• #22626 rpk debug bundle: include the result of uname -a by @JFlath in #22627
• #22653 Schema Registry: Remove spurious log entry: No syntax specified for the proto file by @BenPope in #22654
• rpk: --mechanism flag is now required to update users when using rpk security user update. by @r-vasquez in #21515
• PR #18319 [v23.3.x] offline_log_viewer: fix get_control_record_type by @bharathv
• PR #20747 [v23.3.x] Do not set transaction timeout in tests that do not need it by @mmaslankaprv
• PR #20826 [v23.3.x] cpu_profiler: prealloc result buffers by @StephanDollberg
• PR #21290 [v23.3.x] pandaproxy: fix spelling of operation by @andijcr
• PR #21363 [v23.3.x] tests: increase tolerance in local retention movement test by @mmaslankaprv
• PR #21377 [v23.3.x] rptest: tolerate leadership changes in chunk read tests by @nvartolomei
• PR #21464 [v23.3.x] Maintenance mode test fixes by @ztlpn
• PR #21526 [v23.3.x] producer_state: add request::set_error and make it idempotent by @bharathv
• PR #21568 Backport for "Merged Guard from concurrent elections" #21361 by @bashtanov
• PR #22318 [v23.3.x] Fix http client segfault in metrics_reporter by @ztlpn
• PR #22536 [v23.3.x] cloud_storage: add error_outcome::operation_not_supported by @WillemKauf

Full Changelog: v23.3.18...v23.3.19

v24.2.1

Deprecation notice

We recommend v24.2.2 or newer. For more context, see this Technical Service Bulletin.

New Features

• Redpanda is now compliant with FIPS 140. To put Redpanda into FIPS mode, you will need to install the redpanda-fips package and set the fips_mode node config to true. For more information about Redpanda's security features and it's FIPS compliance, please see our documentation. by @michael-redpanda in #17616
• rpk: Add support to JSON Schema to rpk, now supported in rpk registry and rpk topic produce/consume by @r-vasquez in #20820
• PR #19796 [CORE-3180] json schema registry validate input schema by @andijcr
• PR #19818 [CORE-4180] schema_registry/json: JSON Schema support by @BenPope
• Support JavaScript and TypeScript as languages for Data Transforms. by @rockwotj in #18078
• rpk: now you can manage client quotas using rpk cluster quotas. by @r-vasquez in #18711
• rpk: introduce a command to import cluster client quotas: rpk cluster quotas import. by @r-vasquez in #21311
• Adds new cluster config tls_min_version that allows users of Redpanda to specify the minimum version of TLS Redpanda will support. By default, the value is TLS v1.2. by @michael-redpanda in #21372
• Schema Registry: Support /mode endpoints for READONLY by @BenPope in #17952
• Schema Registry: Support subjectPrefix for GET /subjects by @BenPope in #20145
• Schema Registry: Support for deleted=true query parameter on POST /subjects/<subject>. by @BenPope in #18391
• #17418 rpk: ability to transfer partition leadership by @daisukebe in #18026
• #2183 Adds path-style addressing to S3 client requests. by @WillemKauf in #17806
  This option is configurable through rpk cluster config edit: by @WillemKauf in #17806
• Adds a cloud storage check as part of the cluster's self test. by @WillemKauf in #17586
• Allow decreasing core count if node-local core assignment is enabled. by @ztlpn in #20312
• If the node_local_core_assignment flag is enabled, Redpanda will try to maintain balanced distribution of partition replicas across cores. by @ztlpn in #19864
• KIP-518 (ListGroups v4): adds redpanda and rpk support for printing consumer group states with rpk group list and filtering for specific group states with --states. by @pgellert in #21347
  #7867 rpk container start: now starts a Redpanda Console container connected with the cluster. by @r-vasquez in #18066
• rpk container now has a set of flags to specify ports for node to start on. by @r-vasquez in #17908
• rpk: rpk cluster partitions list now supports filtering with broker IDs. by @daisukebe in #17945
• Adds a read distribution histogram to the internal metrics that can be visualized using Prometheus and Grafana: by @WillemKauf in #18745
  • vectorized_kafka_fetch_read_distribution_bucket
  • vectorized_kafka_fetch_read_distribution_count
  • vectorized_kafka_fetch_read_distribution_sum

Improvements

• Adds a new fetch_read_strategy option non_polling_with_debounce. This option will add a fixed delay specified by fetch_reads_debounce_timeout to the start of every fetch. by @ballard26 in #20744
• Adds configuration options to trigger cache trim before the cache reaches its maximum size. by @jcipar in #18756
  • cloud_storage_cache_trim_threshold_size
  • cloud_storage_cache_trim_threshold_objects
    These mirror the options for controlling maximum size: cloud_storage_cache_size and cloud_storage_cache_max_objects
• Re-adds the fetch_read_strategy cluster config property to select between polling and non-polling fetch implementations. Uses the non-polling fetch implementation by default. by @StephanDollberg in #18090
• Split cache into buckets using cloud_storage_cache_num_buckets configuration parameter. by @Lazin in #18762
• The new default behavior, if these are not set, is to trigger a trim when the cache is 100% full. by @jcipar in #18756
• #2183* * --cloud-backoff-ms uint, the backoff in milliseconds for a cloud storage request. by @WillemKauf in #17586
• --cloud-timeout-ms uint, the timeout in milliseconds for a cloud storage request. by * redpanda will now self configure its cloud_storage_url_style for use with the internal S3 Client on start-up if it is left unspecified. by @WillemKauf in #18107
• Add an optional TLS config field for CRL (Certificate Revocation List) by @oleiman in #18708
• Add more cases to the rpk disk self-test to better probe write performance at various IO depths, and at 16K block sizes. Return more information about the specifics of the test in the output. by @travisdowns in #20590
• Add output batch compression for Data Transforms (configurable per deployed transform) by @oleiman in #18514
• Add schema registry support to experimental Data Transforms C++ SDK by @oleiman in #21292
• Adds the ability to pause a deployed data transform w/o removing it from the system by @oleiman in #18236
• Adds the ability to start transform processing from an arbitrary offset on the input topic. by @oleiman in #19975
• Client qroup quotas now support dynamic configuration of the rate. by @BenPope in #18693
• Data transforms better respect overall system memory usage and go faster when more memory is available. by @rockwotj in #17810
• Enable controller log eviction - now the snapshotted part of the controller log will be deleted. by @ztlpn in #18836
• Golang Transforms SDK (Serde): Adds the ability to specify subject name strategy for output topics. Transforms can also provide a function to be called after the subject name is derived - e.g. to ensure that the subject is created before emitting records. by @oleiman in #19823
• HTTP Proxy: Avoid large allocations during JSON serde of requests and responses by @BenPope in #20827
• Implemented node-local core assignment - nodes decide themselves on which core to put partitions instead of using global assignments. This feature is enabled by default for new clusters. by @ztlpn in #18581
• Improve rpk cluster self-test status output for the cloud test. by @WillemKauf in #21353
• Improve cloud storage cache to prevent readers from being blocked during cache eviction. by @Lazin in #18056
• Improved logging around create_topics and alter_configs requests by @graphcareful in #18789
• It is now possible to use rpk group seek to set commits on a group that does not yet exist by @twmb in #18667
• Made electing a leader faster by @mmaslankaprv in #18394
• Made leadership changes related with reconfiguration faster and less disruptive by @mmaslankaprv in #19966
• Modify connection limit reached log line at WARN instead of INFO level by @graphcareful in #18773
• Run directory walking during cache trimming concurrently. On some deployments it was observed that it can take hours for 600K objects with busy reactor during which...

v24.1.13

Bug fixes

• PR #21565 [v24.1.x] Guard from concurrent elections by @bashtanov
• PR #21612 [v24.1.x] Fix http client segfault in metrics_reporter by @ztlpn
• PR #21613 [v24.1.x] cst/cache: fix in mem trim termination condition by @abhijat

Full Changelog: v24.1.12...v24.1.13

v24.1.12

Improvements

• rpk: --mechanism flag is now required to update users when using rpk security user update. by @r-vasquez in #21514

Bug fixes

• PR #21521 [v24.1.x] archival: keep locks until outcome is known by @nvartolomei
• PR #21524 [v24.1.x] producer_state: add request::set_error and make it idempotent by @bharathv
• PR #21570 [v24.1.x] cst/config: Disable trim carryover by @abhijat
• PR #21579 [v24.1.x] Fix http client segfault in metrics_reporter by @ztlpn

Full Changelog: v24.1.11...v24.1.12

v24.1.11

Bug Fixes

• #21421 Speeds up the shutdown time of partition replicas when compactions are happening. by @andrwng in #21422
• fixed handling of delayed snapshot requests that might lead to an assertion by @mmaslankaprv in #20148

Improvements

• PR #21383 [v24.1.x] Optimize replicated_partition::validate_fetch_offset by @ballard26
• PR #21434 [v24.1.x] cst/ducktape: Change assertion for deleted chunks by @abhijat
• PR #21471 [v24.1.x] [CORE-5672] crypto/bench: Only load context once per run by @michael-redpanda
• PR #21480 [v24.1.x] schema_registry: Remove extraneous log by @BenPope
• PR #21502 [v24.1.x] cst/cache: tracker based trim by @abhijat

Full Changelog: v24.1.10...v24.1.11

v24.1.10

Features

• Adds configuration options to trigger cache trim before the cache reaches its maximum size. by @jcipar in #19850
  • cloud_storage_cache_trim_threshold_size
  • cloud_storage_cache_trim_threshold_objects
    These mirror the options for controlling maximum size: cloud_storage_cache_size and cloud_storage_cache_max_objects
• Schema Registry: Support subjectPrefix for GET /subjects by @BenPope in #20155
• The new default behavior, if these are not set, is to trigger a trim when the cache is 100% full. by @jcipar in #19850

Bug Fixes

• Fix an issue where transforms would miscalculate their initial start offset, leading to consuming the whole input topic. by @oleiman in #20139
• Fix timequery failing when requested timestamp is in the future and local log is empty. by @nvartolomei in #21320
• Schema Registry: Improve handling of deleted schema by @BenPope in #19973
• #20591 Avoid spurious step downs if the append_entries are blocked for longer than usual on the followers. by @bharathv in #20801
• #21252 Fixes incorrect updates to next offset to be applied when racing with applying raft snapshots. by @bharathv in #21254
• rpk: fixed a bug that prevented --any-port from working with Redpanda Console when using a cluster with more than 1 node. by @r-vasquez in #20138

Improvements

• Schema Registry: GET /subject/<subject>/versions/ now supports -1 as an alias for latest. by @BenPope in #20151
• Schema Registry: remove the schema from memory when the last subject version referencing it is deleted. by @pgellert in #21259
• Short description of how this PR improves existing behavior. by @jcipar in #19850
• #21297 HTTP Proxy: Avoid large allocations during JSON serde of requests and responses by @BenPope in #21450
• #21297 Schema Registry: Avoid large allocations during JSON serde of requests and responses by @BenPope in #21450
• #21466 Schema Registry: Avoid large contiguous allocations whilst handling schema. by @BenPope in #21468

Full Changelog: v24.1.9...v24.1.10

v23.3.18

Features

• Schema Registry: Support subjectPrefix for GET /subjects by @BenPope in #20156

Bug Fixes

• PR #20589 [CORE-4878] [v23.3.x] http: change stop() to shutdown() on TLS errors in client (manual backport) by @WillemKauf
• Fix timequery failing with exceptions when the queried partition is empty. by @nvartolomei in #19951
• Schema Registry: Improve handling of deleted schema by @BenPope in #19972

Improvements

• Run directory walking during cache trimming concurrently. On some deployments it was observed that it can take hours for 600K objects with busy reactor during which fetch operations that need to cache data are blocked. by @nvartolomei in #19816
• Schema Registry: Improve handling of Accept and Content-Type by @BenPope in #19879
• Schema Registry: GET /subject/<subject>/versions/ now supports -1 as an alias for latest. by @BenPope in #20150
• #19897 rpk: group describe supports --regex flag by @daisukebe in #19898
• new metric providing more insight into recovery process by @mmaslankaprv in #19874
• refined vectorized_raft_leadership_changes_total metric by @mmaslankaprv in #19892

Full Changelog: v23.3.17...v23.3.18

v24.1.9

Bug Fixes

• Fix timequery failing with exceptions when the queried partition is empty. by @nvartolomei in #19950

Improvements

• Schema Registry: Improve handling of Accept and Content-Type by @BenPope in #19880
• #19877 rpk debug bundle now fallback to loaded profile's admin API URLs if we fail to discover the cluster in the collection steps. by @r-vasquez in #19878
• #19881 #19882 #19883 #19884 rpk debug bundle mark the generated bundle file with the node's advertised_rpc address. by @r-vasquez in #19885
• #19881 #19882 #19883 #19884 rpk debug bundle now collects the log since yesterday by default, you can still change them with --logs-since. by @r-vasquez in #19885
• #19895 rpk: group describe supports --regex flag by @daisukebe in #19896
• refined vectorized_raft_leadership_changes_total metric by @mmaslankaprv in #19891

Full Changelog: v24.1.8...v24.1.9

v24.1.8

Bug Fixes

• fixed overflow that may lead to unnecessary moves by @mmaslankaprv in #19803
• rpk cluster config get: does not round float numbers anymore. by @r-vasquez in #18850
• PR #18835 [v24.1.x] Fixed possible log discrepancy when using forced reconfiguration by @mmaslankaprv
• PR #19313 [v24.1.x] cpu_profiler: prealloc result buffers by @StephanDollberg
• PR #19801 [v24.1.x] s/disk_log_impl: don't prefix-truncate empty segments by @ztlpn
• PR #19808 [v24.1.x] t/fetch_test: wait longer for leadership after stepping down by @mmaslankaprv

Improvements

• Run directory walking during cache trimming concurrently. On some deployments it was observed that it can take hours for 600K objects with busy reactor during which fetch operations that need to cache data are blocked. by @nvartolomei in #19815
• #19831 Don't try to transfer leadership to just restarted nodes when balancing leaders. by @ztlpn in #19832
• new metric providing more insight into recovery process by @mmaslankaprv in #18840
• reduced the amount of data required to transfer over the network by @mmaslankaprv in #19834
• PR #18854 [v24.1.x] cst: manual backport of chunk download changes PR 18278 by @abhijat

Full Changelog: v24.1.7...v24.1.8

v23.3.17

Features

• Adds configuration options to trigger cache trim before the cache reaches its maximum size. by @jcipar in #19624
  • cloud_storage_cache_trim_threshold_size
  • cloud_storage_cache_trim_threshold_objects
    These mirror the options for controlling maximum size: cloud_storage_cache_size and cloud_storage_cache_max_objects
• The new default behavior, if these are not set, is to trigger a trim when the cache is 100% full. by @jcipar in #19624
• #18739 Schema Registry: Support /mode endpoints for READONLY by @BenPope in #18742

Bug Fixes

• Fixes a bug where crashes within the redpanda http client could occur when encountering tls exceptions by @graphcareful in #18696
• #18633 rpk: fixes an error in rpk topic consume that prevented the usage of the --regex flag. by @r-vasquez in #18634
• #18734 Fixes incorrect ordering of arguments in the cloud cache trim admin endpoint. by @andrwng in #18764
• #18770 Fixes a bug that would allow requests to complete that created acls for topics with invalid kafka topic names by @graphcareful in #19791
• fixed overflow that may lead to unnecessary moves by @mmaslankaprv in #19805
• rpk cluster config get: does not round float numbers anymore. by @r-vasquez in #18849
• PR #18784 [v23.3.x] raft: fix node_id mismatch log message by @ztlpn
• PR #18855 [v23.3.x] Fixed possible log discrepancy when using forced reconfiguration by @mmaslankaprv
• PR #18573 rm_stm: couple of stability fixes noticed when down scaling max_concurrent_producer_ids by @bharathv

Improvements

• Short description of how this PR improves existing behavior. by @jcipar in #19624
• #18645 rpk: topic describe supports --regex flag by @daisukebe in #18646
• made fast partition movements easier to debug. by @mmaslankaprv in #18689
• reduced the amount of data required to transfer over the network by @mmaslankaprv in #19835
• PR #18741 [v23.3.x] cloud_storage_clients: check for BlobNotFound in abs_client::do_delete_path() by @WillemKauf
• PR #19838 [v23.3.x] s/disk_log_impl: don't prefix-truncate empty segments by @ztlpn

Full Changelog: v23.3.16...v23.3.17