Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support NebulaGraph #5116

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Conversation

GG2002
Copy link
Contributor

@GG2002 GG2002 commented Sep 12, 2024

Which issue does this PR close?

Closes #4553.

Rationale for this change

What changes are included in this PR?

Support read, write, list, delete ops on NebulaGraph.

To access NebulaGraph, users must provide the username and password for NebulaGraph, as well as specify the space, tag, and the key field and value field of the tag.

Since each vertex in NebulaGraph must have a unique ID(called VID), but NebulaGraph lacks a self-incrementing ID, the unique IDs are generated using the Snowflake algorithm. Additionally, the set operation involves first delete existing original vertex, followed by inserting a new vertex.

NebulaGraph does not support the Blob type, hence Base64+string type is employed for file access. If it's supported in the future, just remove Base64.

Are there any user-facing changes?

No(?)

Signed-off-by: feathercyc <feathercyc@163.com>
@GG2002
Copy link
Contributor Author

GG2002 commented Sep 18, 2024

Test results are as follows:

$ OPENDAL_TEST=nebulagraph cargo test behavior --features tests -- --nocapture --test-threads=20
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.32s
     Running unittests src/lib.rs (target/debug/deps/opendal-20c211e669cef355)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 128 filtered out; finished in 0.00s

     Running tests/behavior/main.rs (target/debug/deps/behavior-419dea746c216121)

running 81 tests
test behavior::test_check                                     ... ok
test behavior::test_list_with_start_after                     ... ok
test behavior::test_delete_not_existing                       ... ok
test behavior::test_list_non_exist_dir                        ... ok
test behavior::test_delete_empty_dir                          ... ok
test behavior::test_create_dir                                ... ok
test behavior::test_create_dir_existing                       ... ok
test behavior::test_list_root_with_recursive                  ... ok
test behavior::test_list_sub_dir                              ... ok
test behavior::test_list_file_with_recursive                  ... ok
test behavior::test_list_empty_dir                            ... ok
test behavior::test_read_with_if_match                        ... ok
test behavior::test_read_with_if_none_match                   ... ok
test behavior::test_read_not_exist                            ... ok
test behavior::test_read_with_dir_path                        ... ok
test behavior::test_read_with_override_cache_control          ... ok
test behavior::test_read_with_override_content_disposition    ... ok
test behavior::test_read_with_override_content_type           ... ok
test behavior::test_list_dir_with_recursive                   ... ok
test behavior::test_list_dir_with_recursive_no_trailing_slash ... ok
test behavior::test_list_nested_dir                           ... ok
test behavior::test_stat_dir                                  ... ok
test behavior::test_stat_file                                 ... ok
test behavior::test_stat_not_exist                            ... ok
test behavior::test_stat_with_if_match                        ... ok
test behavior::test_stat_with_if_none_match                   ... ok
test behavior::test_stat_with_override_cache_control          ... ok
test behavior::test_stat_with_override_content_disposition    ... ok
test behavior::test_stat_with_override_content_type           ... ok
test behavior::test_stat_root                                 ... ok
test behavior::test_remove_all                                ... ok
test behavior::test_write_with_empty_content                  ... ok
test behavior::test_write_with_dir_path                       ... ok
test behavior::test_write_only                                ... ok
test behavior::test_write_with_cache_control                  ... ok
test behavior::test_write_with_content_type                   ... ok
test behavior::test_write_with_content_disposition            ... ok
test behavior::test_write_with_user_metadata                  ... ok
test behavior::test_writer_write                              ... ok
test behavior::test_write_with_special_chars                  ... ok
test behavior::test_writer_write_with_concurrent              ... ok
test behavior::test_writer_sink                               ... ok
test behavior::test_writer_sink_with_concurrent               ... ok
test behavior::test_list_rich_dir                             ... ok
test behavior::test_writer_abort                              ... ok
test behavior::test_writer_futures_copy                       ... ok
test behavior::test_writer_futures_copy_with_concurrent       ... ok
test behavior::test_stat_nested_parent_dir                    ... ok
test behavior::test_blocking_create_dir                       ... ok
test behavior::test_blocking_create_dir_existing              ... ok
test behavior::test_writer_abort_with_concurrent              ... ok
test behavior::test_stat_not_cleaned_path                     ... ok
test behavior::test_reader                                    ... ok
test behavior::test_blocking_remove_one_file                  ... ok
test behavior::test_blocking_read_not_exist                   ... ok
test behavior::test_list_dir_with_metakey                     ... ok
test behavior::test_blocking_stat_dir                         ... ok
test behavior::test_list_dir                                  ... ok
test behavior::test_blocking_stat_not_exist                   ... ok
test behavior::test_writer_write_with_overwrite               ... ok
test behavior::test_delete_file                               ... ok
test behavior::test_blocking_read_full                        ... ok
test behavior::test_read_full                                 ... ok
test behavior::test_blocking_write_with_dir_path              ... ok
test behavior::test_blocking_delete_file                      ... ok
test behavior::test_blocking_write_file                       ... ok
test behavior::test_blocking_read_range                       ... ok
test behavior::test_blocking_remove_all_basic                 ... ok
test behavior::test_read_range                                ... ok
test behavior::test_remove_all_basic                          ... ok
test behavior::test_list_prefix                               ... ok
test behavior::test_stat_with_special_chars                   ... ok
test behavior::test_delete_with_special_chars                 ... ok
test behavior::test_list_dir_with_file_path                   ... ok
test behavior::test_blocking_stat_file                        ... ok
test behavior::test_blocking_stat_with_special_chars          ... ok
test behavior::test_list_dir_with_metakey_complete            ... ok
test behavior::test_remove_one_file                           ... ok
test behavior::test_delete_stream                             ... ok
test behavior::test_read_with_special_chars                   ... ok
test behavior::test_blocking_write_with_special_chars         ... ok

test result: ok. 81 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.76s

Comment on lines +2863 to +2867
[[package]]
name = "futures"
version = "0.1.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a471a38ef8ed83cd6e40aa59c1ffe17db6855c18e3604d9c4ed8c08ebc28678"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Who is depending on futures 0.1?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's futures-util, fbthrift-* used in rust-nebula and bb8 depends on futures and futures depends on futures-util. futures-utils/dependencies displayed that futures=0.1.25.

cargo tree is as follows:

├── bb8 v0.8.5
│   ├── async-trait v0.1.82 (proc-macro) (*)
│   ├── futures-util v0.3.30
│   │   ├── futures v0.1.31
│   │   ├── futures-channel v0.3.30
│   │   │   ├── futures-core v0.3.30
│   │   │   └── futures-sink v0.3.30

Does it need to be removed? I just don't know how to do it:(

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems someone has enabled futures-util's compat feature in wrong.


Ok, I got it: https://github.com/bk-rs/fbthrift-git-rs/blob/36ddca22a4aca8bbf02482ca20ff3fe525a24a87/Cargo.toml#L17

Copy link
Contributor Author

@GG2002 GG2002 Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vkill has contributed almost all of fbThrift's packages, can we remove this without communicating with him😇?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure. Can you contact @vkill? Or can we obtain write permission for the bk-rs organization? Maybe @wey-gu can help a bit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps it's possible, but I don't have high hopes. If I can establish a stable connection with vkill, I won't have to release a new rust-nebula crate. I will make modifications directly on his nebula-rust crate:(

@@ -342,6 +343,11 @@ compio = { version = "0.11.0", optional = true, features = [
] }
# for services-s3
crc32c = { version = "0.6.6", optional = true }
# for services-nebula-graph
rust-nebula = { version = "0.1", optional = true, git = "https://github.com/nebula-contrib/rust-nebula", features = [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, this could prevent us from making a new release.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll publish rust-nebula crate in a few days, you can take a look at my code first.

@@ -163,6 +163,7 @@ services-moka = ["dep:moka"]
services-mongodb = ["dep:mongodb"]
services-monoiofs = ["dep:monoio", "dep:flume"]
services-mysql = ["dep:sqlx", "sqlx?/mysql"]
services-nebulagraph = ["dep:rust-nebula", "dep:bb8", "dep:snowflaked"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenDAL has strict naming pattern for service names.

Please either use:

  • nebulagraph / Nebulagraph

or

  • nebula_graph / NebulaGraph

In every place this name showed up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for NebulaGraph
2 participants