[improve][broker] Fix non-persistent system topic schema compatibility

[Demogorgon314]

Motivation

When upgrading broker version from 3.0.x to 3.3.x with ExtensibleLoadManagerImpl enabled, it will have an Unable to read schema exception. And the broker will fail to start. This issue is caused by #22055 .

***************
*** 23,29 ****
      "default" : null
    }, {
      "name" : "bundleCount",
!     "type" : "int"
    }, {
      "name" : "cpu",
      "type" : [ "null", "org.apache.pulsar.policies.data.loadbalancer.ResourceUsage" ],
--- 23,29 ----
      "default" : null
    }, {
      "name" : "bundleCount",
!     "type" : "long"
    }, {
      "name" : "cpu",
      "type" : [ "null", "org.apache.pulsar.policies.data.loadbalancer.ResourceUsage" ],
***************
*** 59,65 ****
      "type" : "long"
    }, {
      "name" : "topics",
!     "type" : "int"
    }, {
      "name" : "updatedAt",
      "type" : "long"
--- 59,65 ----
      "type" : "long"
    }, {
      "name" : "topics",
!     "type" : "long"
    }, {
      "name" : "updatedAt",
      "type" : "long"

Modifications

Add a new class NonPersistentSystemTopic, and it will use for system non-persistent topic.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.54%. Comparing base (bbc6224) to head (dfcb47a).
Report is 578 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23286      +/-   ##
============================================
+ Coverage     73.57%   74.54%   +0.97%     
- Complexity    32624    34305    +1681     
============================================
  Files          1877     1928      +51     
  Lines        139502   145035    +5533     
  Branches      15299    15859     +560     
============================================
+ Hits         102638   108122    +5484     
+ Misses        28908    28652     -256     
- Partials       7956     8261     +305     

Flag	Coverage Δ
inttests	27.63% <100.00%> (+3.05%)	⬆️
systests	24.65% <0.00%> (+0.33%)	⬆️
unittests	73.92% <100.00%> (+1.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...rg/apache/pulsar/broker/service/BrokerService.java	82.02% <100.00%> (+1.24%)	⬆️
...ervice/nonpersistent/NonPersistentSystemTopic.java	100.00% <100.00%> (ø)

... and 558 files with indirect coverage changes

[codelipenghui]

@Demogorgon314
The fix looks good. But I still have concern about the compatibility.

• We have added systemTopicSchemaCompatibilityStrategy=ALWAYS_COMPATIBLE which is not by original design, because we don't have another way to resolve the compatibility issue.
• Even if we use ALWAYS_COMPATIBLE, that doesn't means we will ok for any data format change

So, I would like to suggest to add a real compatibility test for different version of data. For the code part, we should not remove the old format data struct directly. Instead, we can create v1, v2 ..., then we can test that we can publish data from v1, v2 and consume by v(latest), and vice versa.

[Demogorgon314]

@codelipenghui Did you mean to set the SchemaCompatibilityStrategy to BACKWARD? The default strategy is FULL, so int -> long and long -> int are incompatible.

[Demogorgon314]

@codelipenghui I have added a test to cover the BrokerLoadData format change.