Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat][schema] Support the schema type ProtoNativeSchema #1006

Merged
merged 11 commits into from
Apr 23, 2023

Conversation

gaoran10
Copy link
Contributor

Motivation

The pulsar-client-go supports ProtoSchema based on the Avro format, users must provide the schema definition string when using struct schema(JSON, Avro, Proto), it's hard to use and we can't get the entire original proto definition.

The difference between ProtoSchema and ProtoNativeSchema is the schema data, the ProtoNativeSchema will contain the entire original proto definition, and we can use the schema data of the ProtoNativeSchema to generate the corresponding Descriptor and get any field, we can leverage this feature in Pulsar SQL.

The schema data of the ProtoNativeSchema follow the Pulsar format, we can generate JSON data for ProtobufNativeSchemaData as the schema data.

public class ProtobufNativeSchemaData {

    private byte[] fileDescriptorSet;
    private String rootMessageTypeName;
    private String rootFileDescriptorName;

}

The field fileDescriptorSet is the serialized data of the message FileDescriptorSet, it contains all definition proto files data.

message FileDescriptorSet {
  repeated FileDescriptorProto file = 1;
}
message FileDescriptorProto {
  optional string name = 1;
  optional string package = 2;
  repeated string dependency = 3;
  repeated DescriptorProto message_type = 4;
  repeated EnumDescriptorProto enum_type = 5;
  repeated ServiceDescriptorProto service = 6;
  repeated FieldDescriptorProto extension = 7;
  optional FileOptions options = 8;
  optional SourceCodeInfo source_code_info = 9;
  repeated int32 public_dependency = 10;
  repeated int32 weak_dependency = 11;
  optional string syntax = 12;
} 

Modifications

Add a new schema type ProtoNative(20).
Add ProtoNativeSchema.
Add encode and decode methods for ProtoNativeSchema.

Verifying this change

Add an integration test to make sure the broker can deserialize the schema data of the ProtoNativeSchema.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (yes)
  • The public API: (yes)
  • The schema: (yes)
  • The default values of configurations: (no)
  • The wire protocol: (no)

Documentation

  • Does this pull request introduce a new feature? (yes)
  • If yes, how is the feature documented? (not applicable / docs / GoDocs / not documented)
  • If a feature is not applicable for documentation, explain why?
  • If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

@BewareMyPower
Copy link
Contributor

Could you fix lint error? You can use golangci-lint run to check lint errors in your local env.

@gaoran10 gaoran10 force-pushed the support-proto-native-schema branch from 6443073 to 7c8a10f Compare April 11, 2023 17:39
@RobertIndie
Copy link
Member

https://github.com/apache/pulsar-client-go/actions/runs/4670464787/jobs/8270224455?pr=1006#step:5:789

Copy link
Member

@nodece nodece left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! I left some comments.

pulsar/schema.go Outdated Show resolved Hide resolved
pulsar/schema.go Outdated Show resolved Hide resolved
pulsar/schema.go Outdated Show resolved Hide resolved
.gitignore Outdated Show resolved Hide resolved
pulsar/schema.go Outdated Show resolved Hide resolved
pulsar/schema.go Outdated Show resolved Hide resolved
pulsar/schema.go Outdated Show resolved Hide resolved
pulsar/schema.go Outdated Show resolved Hide resolved
pulsar/schema.go Outdated Show resolved Hide resolved
.gitignore Outdated Show resolved Hide resolved
pulsar/schema.go Outdated Show resolved Hide resolved
@BewareMyPower
Copy link
Contributor

@nodece Could you take a look again?

Copy link
Member

@nodece nodece left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nodece nodece merged commit fc4f289 into apache:master Apr 23, 2023
@shibd shibd added this to the v0.11.0 milestone May 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants