Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Merged by Bors] - feat: generate SmartModules using SMDK #2630

Conversation

EstebanBorai
Copy link
Contributor

@EstebanBorai EstebanBorai commented Sep 29, 2022

Provides support to generate new SmartModules using cargo-generate as a libary.

Resolve: #2621


Examples

Generates SmartModule Projects using smdk generate

➜  Desktop smdk generate log-filter
Generating new SmartModule project: log-filter
🔧   Destination: /Users/esteban/Desktop/log-filter ...
🔧   Generating template ...
✔ 🤷   Which type of SmartModule would you like? · filter
✔ 🤷   Want to use SmartModule parameters? · false
🤷   SmartModule Version [default: git = "https://github.com/infinyon/fluvio.git"]: git = "https://github.com/infinyon/fluvio.git"
✔ 🤷   Want to use SmartModule init? · true
Ignoring: /var/folders/0g/l702s31x2mj4zdlw2_43rpyr0000gn/T/.tmpwQ4Yql/cargo-generate.toml
[1/5]   Done: Cargo.toml
[2/5]   Done: README.md
[3/5]   Done: Smart.toml
[4/5]   Done: src/lib.rs
[5/5]   Done: src
🔧   Moving generated files into: `/Users/esteban/Desktop/log-filter`...
💡   Initializing a fresh Git repository
✨   Done! New project created /Users/esteban/Desktop/log-filter

Generated Project

total 24
drwxr-xr-x  7 esteban  staff   224B Oct  2 13:54 .
drwx------+ 6 esteban  staff   192B Oct  2 13:54 ..
drwxr-xr-x  9 esteban  staff   288B Oct  2 13:54 .git
-rw-r--r--  1 esteban  staff   358B Oct  2 13:54 Cargo.toml
-rw-r--r--  1 esteban  staff   1.8K Oct  2 13:54 README.md
-rw-r--r--  1 esteban  staff   152B Oct  2 13:54 Smart.toml
drwxr-xr-x  3 esteban  staff    96B Oct  2 13:54 src

Generated src/lib.rs with the provided options

use fluvio_smartmodule::dataplane::smartmodule::{SmartModuleExtraParams, SmartModuleInitError};

use once_cell::sync::OnceCell;
use fluvio_smartmodule::eyre;



use fluvio_smartmodule::{smartmodule, Result, Record};

#[smartmodule(filter)]
pub fn filter(record: &Record) -> Result<bool> {
    let string = std::str::from_utf8(record.value.as_ref())?;
    Ok(string.contains('a'))
}




static CRITERIA: OnceCell<String> = OnceCell::new();

#[smartmodule(init)]
fn init(params: SmartModuleExtraParams) -> Result<()> {
    // You can refer to the example SmartModules in Fluvio's GitHub Repository
    // https://github.com/infinyon/fluvio/tree/master/smartmodule
    if let Some(key) = params.get("key") {
        CRITERIA.set(key.clone()).map_err(|err| eyre!("failed setting key: {:#?}", err))
    } else {
        Err(SmartModuleInitError::MissingParam("key".to_string()).into())
    }
}

Building SMDK Project

➜  Desktop cd ./log-filter
➜  log-filter git:(main) ✗ smdk build
    Updating git repository `https://github.com/infinyon/fluvio.git`
    Updating crates.io index
   Compiling proc-macro2 v1.0.46

Testing SM Project using smdk test

➜  log-filter git:(main) ✗ smdk test --params "key=a" --text "testing"
project name: "log-filter"
loading module at: target/wasm32-unknown-unknown/release-lto/log_filter.wasm
0 records outputed
➜  log-filter git:(main) ✗ smdk test --params "key=a" --text "tasty"
project name: "log-filter"
loading module at: target/wasm32-unknown-unknown/release-lto/log_filter.wasm
1 records outputed
tasty

_out: Arc<O>,
_fluvio: &Fluvio,
) -> Result<()> {
let template_path = TemplatePath {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neither this struct nor GenerateArgs implements Default trait. So I had to provide initial values for both.
I'll open a PR to the generate crate to see if they allow us to have a Default implementation.

@EstebanBorai
Copy link
Contributor Author

Addresses #2631

@sehz
Copy link
Contributor

sehz commented Sep 29, 2022

see: #2632

@sehz
Copy link
Contributor

sehz commented Sep 29, 2022

Updated #2631 per new Smdk spec

@EstebanBorai
Copy link
Contributor Author

Updated #2631 per new Smdk spec

Awesome! Here I'm also taking care of the build step by spawning a cargo build command.
Heres the file:

https://github.com/infinyon/fluvio/pull/2630/files#diff-953ad69fcfe3250bce68a4d31f2e069b45e53b9448c737fc0d22b6fe77762527

@EstebanBorai
Copy link
Contributor Author

see: #2632

Great! Will migrate to crates/smdk/ then. Thanks for sharing!

@EstebanBorai EstebanBorai force-pushed the feat/#2621-generate-new-smart-module branch from e0fd9fd to 154a5c9 Compare September 30, 2022 13:50
@EstebanBorai EstebanBorai marked this pull request as draft September 30, 2022 14:00
@EstebanBorai EstebanBorai marked this pull request as ready for review September 30, 2022 17:05
@EstebanBorai EstebanBorai changed the title feat: generate smart modules support feat: build and generate SmartModules using SMDK Sep 30, 2022
@EstebanBorai
Copy link
Contributor Author

@sehz now we have support for testing the SmartModule build using smdk build command.
An example on usage is in the PR comment.

crates/smdk/src/build.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@sehz sehz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also attach running output of working?

crates/smdk/src/build.rs Outdated Show resolved Hide resolved
crates/smdk/src/build.rs Outdated Show resolved Hide resolved
crates/smdk/src/build.rs Outdated Show resolved Hide resolved
crates/smdk/src/generate.rs Outdated Show resolved Hide resolved
@EstebanBorai
Copy link
Contributor Author

@sehz code related to building SmartModules lives now on this PR: #2638

@EstebanBorai EstebanBorai changed the title feat: build and generate SmartModules using SMDK feat: generate SmartModules using SMDK Sep 30, 2022
Provides support to generate new SmartModules using `cargo-generate`
as a libary.

Resolve: infinyon#2621
@EstebanBorai EstebanBorai force-pushed the feat/#2621-generate-new-smart-module branch from a8b337e to 54a1733 Compare October 1, 2022 16:36
@EstebanBorai EstebanBorai force-pushed the feat/#2621-generate-new-smart-module branch from 37e5fd2 to f8e2e84 Compare October 1, 2022 19:35
Copy link
Contributor

@sehz sehz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, can you add output from generating->build->test including init?
I believe template need handling for init

crates/smdk/src/generate.rs Show resolved Hide resolved
@EstebanBorai
Copy link
Contributor Author

EstebanBorai commented Oct 2, 2022

Also, can you add output from generating->build->test including init?

I believe template need handling for init

Current template has support for init:

[placeholders.smartmodule-init]

But I dont find any effect from such option in the lib.rs from the same template.

https://github.com/infinyon/fluvio/blob/master/smartmodule/cargo_template/src/lib.rs

@sehz
Copy link
Contributor

sehz commented Oct 2, 2022

That needs to be added. See regex project

@EstebanBorai EstebanBorai force-pushed the feat/#2621-generate-new-smart-module branch from b6511c1 to 4752f3d Compare October 2, 2022 15:51
@EstebanBorai
Copy link
Contributor Author

Also, can you add output from generating->build->test including init?
I believe template need handling for init

Handling for init is now provided

@EstebanBorai
Copy link
Contributor Author

@sehz im stick to the RegExp project version instead of using the current cargo-generate template for this generation. Also added once_cell to provide an example on a filter smart module with init.

@EstebanBorai EstebanBorai requested a review from sehz October 2, 2022 17:27
Copy link
Contributor

@sehz sehz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nice work!

@sehz
Copy link
Contributor

sehz commented Oct 2, 2022

bors r+

bors bot pushed a commit that referenced this pull request Oct 2, 2022
Provides support to generate new SmartModules using `cargo-generate` as a libary.

Resolve: #2621

---

## Examples

### Generates SmartModule Projects using `smdk generate`

```bash
➜  Desktop smdk generate log-filter
Generating new SmartModule project: log-filter
🔧   Destination: /Users/esteban/Desktop/log-filter ...
🔧   Generating template ...
✔ 🤷   Which type of SmartModule would you like? · filter
✔ 🤷   Want to use SmartModule parameters? · false
🤷   SmartModule Version [default: git = "https://github.com/infinyon/fluvio.git"]: git = "https://github.com/infinyon/fluvio.git"
✔ 🤷   Want to use SmartModule init? · true
Ignoring: /var/folders/0g/l702s31x2mj4zdlw2_43rpyr0000gn/T/.tmpwQ4Yql/cargo-generate.toml
[1/5]   Done: Cargo.toml
[2/5]   Done: README.md
[3/5]   Done: Smart.toml
[4/5]   Done: src/lib.rs
[5/5]   Done: src
🔧   Moving generated files into: `/Users/esteban/Desktop/log-filter`...
💡   Initializing a fresh Git repository
✨   Done! New project created /Users/esteban/Desktop/log-filter
```

### Generated Project

```bash
total 24
drwxr-xr-x  7 esteban  staff   224B Oct  2 13:54 .
drwx------+ 6 esteban  staff   192B Oct  2 13:54 ..
drwxr-xr-x  9 esteban  staff   288B Oct  2 13:54 .git
-rw-r--r--  1 esteban  staff   358B Oct  2 13:54 Cargo.toml
-rw-r--r--  1 esteban  staff   1.8K Oct  2 13:54 README.md
-rw-r--r--  1 esteban  staff   152B Oct  2 13:54 Smart.toml
drwxr-xr-x  3 esteban  staff    96B Oct  2 13:54 src
```

### Generated `src/lib.rs` with the provided options

```rust
use fluvio_smartmodule::dataplane::smartmodule::{SmartModuleExtraParams, SmartModuleInitError};

use once_cell::sync::OnceCell;
use fluvio_smartmodule::eyre;



use fluvio_smartmodule::{smartmodule, Result, Record};

#[smartmodule(filter)]
pub fn filter(record: &Record) -> Result<bool> {
    let string = std::str::from_utf8(record.value.as_ref())?;
    Ok(string.contains('a'))
}




static CRITERIA: OnceCell<String> = OnceCell::new();

#[smartmodule(init)]
fn init(params: SmartModuleExtraParams) -> Result<()> {
    // You can refer to the example SmartModules in Fluvio's GitHub Repository
    // https://github.com/infinyon/fluvio/tree/master/smartmodule
    if let Some(key) = params.get("key") {
        CRITERIA.set(key.clone()).map_err(|err| eyre!("failed setting key: {:#?}", err))
    } else {
        Err(SmartModuleInitError::MissingParam("key".to_string()).into())
    }
}
```

### Building SMDK Project

```bash
➜  Desktop cd ./log-filter
➜  log-filter git:(main) ✗ smdk build
    Updating git repository `https://github.com/infinyon/fluvio.git`
    Updating crates.io index
   Compiling proc-macro2 v1.0.46
```

### Testing SM Project using `smdk test`

```
➜  log-filter git:(main) ✗ smdk test --params "key=a" --text "testing"
project name: "log-filter"
loading module at: target/wasm32-unknown-unknown/release-lto/log_filter.wasm
0 records outputed
➜  log-filter git:(main) ✗ smdk test --params "key=a" --text "tasty"
project name: "log-filter"
loading module at: target/wasm32-unknown-unknown/release-lto/log_filter.wasm
1 records outputed
tasty
```
@bors
Copy link

bors bot commented Oct 2, 2022

Build failed:

@sehz
Copy link
Contributor

sehz commented Oct 2, 2022

Mac VM failures

@sehz
Copy link
Contributor

sehz commented Oct 2, 2022

bors r+

bors bot pushed a commit that referenced this pull request Oct 2, 2022
Provides support to generate new SmartModules using `cargo-generate` as a libary.

Resolve: #2621

---

## Examples

### Generates SmartModule Projects using `smdk generate`

```bash
➜  Desktop smdk generate log-filter
Generating new SmartModule project: log-filter
🔧   Destination: /Users/esteban/Desktop/log-filter ...
🔧   Generating template ...
✔ 🤷   Which type of SmartModule would you like? · filter
✔ 🤷   Want to use SmartModule parameters? · false
🤷   SmartModule Version [default: git = "https://github.com/infinyon/fluvio.git"]: git = "https://github.com/infinyon/fluvio.git"
✔ 🤷   Want to use SmartModule init? · true
Ignoring: /var/folders/0g/l702s31x2mj4zdlw2_43rpyr0000gn/T/.tmpwQ4Yql/cargo-generate.toml
[1/5]   Done: Cargo.toml
[2/5]   Done: README.md
[3/5]   Done: Smart.toml
[4/5]   Done: src/lib.rs
[5/5]   Done: src
🔧   Moving generated files into: `/Users/esteban/Desktop/log-filter`...
💡   Initializing a fresh Git repository
✨   Done! New project created /Users/esteban/Desktop/log-filter
```

### Generated Project

```bash
total 24
drwxr-xr-x  7 esteban  staff   224B Oct  2 13:54 .
drwx------+ 6 esteban  staff   192B Oct  2 13:54 ..
drwxr-xr-x  9 esteban  staff   288B Oct  2 13:54 .git
-rw-r--r--  1 esteban  staff   358B Oct  2 13:54 Cargo.toml
-rw-r--r--  1 esteban  staff   1.8K Oct  2 13:54 README.md
-rw-r--r--  1 esteban  staff   152B Oct  2 13:54 Smart.toml
drwxr-xr-x  3 esteban  staff    96B Oct  2 13:54 src
```

### Generated `src/lib.rs` with the provided options

```rust
use fluvio_smartmodule::dataplane::smartmodule::{SmartModuleExtraParams, SmartModuleInitError};

use once_cell::sync::OnceCell;
use fluvio_smartmodule::eyre;



use fluvio_smartmodule::{smartmodule, Result, Record};

#[smartmodule(filter)]
pub fn filter(record: &Record) -> Result<bool> {
    let string = std::str::from_utf8(record.value.as_ref())?;
    Ok(string.contains('a'))
}




static CRITERIA: OnceCell<String> = OnceCell::new();

#[smartmodule(init)]
fn init(params: SmartModuleExtraParams) -> Result<()> {
    // You can refer to the example SmartModules in Fluvio's GitHub Repository
    // https://github.com/infinyon/fluvio/tree/master/smartmodule
    if let Some(key) = params.get("key") {
        CRITERIA.set(key.clone()).map_err(|err| eyre!("failed setting key: {:#?}", err))
    } else {
        Err(SmartModuleInitError::MissingParam("key".to_string()).into())
    }
}
```

### Building SMDK Project

```bash
➜  Desktop cd ./log-filter
➜  log-filter git:(main) ✗ smdk build
    Updating git repository `https://github.com/infinyon/fluvio.git`
    Updating crates.io index
   Compiling proc-macro2 v1.0.46
```

### Testing SM Project using `smdk test`

```
➜  log-filter git:(main) ✗ smdk test --params "key=a" --text "testing"
project name: "log-filter"
loading module at: target/wasm32-unknown-unknown/release-lto/log_filter.wasm
0 records outputed
➜  log-filter git:(main) ✗ smdk test --params "key=a" --text "tasty"
project name: "log-filter"
loading module at: target/wasm32-unknown-unknown/release-lto/log_filter.wasm
1 records outputed
tasty
```
@bors
Copy link

bors bot commented Oct 2, 2022

Build failed:

@sehz
Copy link
Contributor

sehz commented Oct 2, 2022

bors r+

bors bot pushed a commit that referenced this pull request Oct 2, 2022
Provides support to generate new SmartModules using `cargo-generate` as a libary.

Resolve: #2621

---

## Examples

### Generates SmartModule Projects using `smdk generate`

```bash
➜  Desktop smdk generate log-filter
Generating new SmartModule project: log-filter
🔧   Destination: /Users/esteban/Desktop/log-filter ...
🔧   Generating template ...
✔ 🤷   Which type of SmartModule would you like? · filter
✔ 🤷   Want to use SmartModule parameters? · false
🤷   SmartModule Version [default: git = "https://github.com/infinyon/fluvio.git"]: git = "https://github.com/infinyon/fluvio.git"
✔ 🤷   Want to use SmartModule init? · true
Ignoring: /var/folders/0g/l702s31x2mj4zdlw2_43rpyr0000gn/T/.tmpwQ4Yql/cargo-generate.toml
[1/5]   Done: Cargo.toml
[2/5]   Done: README.md
[3/5]   Done: Smart.toml
[4/5]   Done: src/lib.rs
[5/5]   Done: src
🔧   Moving generated files into: `/Users/esteban/Desktop/log-filter`...
💡   Initializing a fresh Git repository
✨   Done! New project created /Users/esteban/Desktop/log-filter
```

### Generated Project

```bash
total 24
drwxr-xr-x  7 esteban  staff   224B Oct  2 13:54 .
drwx------+ 6 esteban  staff   192B Oct  2 13:54 ..
drwxr-xr-x  9 esteban  staff   288B Oct  2 13:54 .git
-rw-r--r--  1 esteban  staff   358B Oct  2 13:54 Cargo.toml
-rw-r--r--  1 esteban  staff   1.8K Oct  2 13:54 README.md
-rw-r--r--  1 esteban  staff   152B Oct  2 13:54 Smart.toml
drwxr-xr-x  3 esteban  staff    96B Oct  2 13:54 src
```

### Generated `src/lib.rs` with the provided options

```rust
use fluvio_smartmodule::dataplane::smartmodule::{SmartModuleExtraParams, SmartModuleInitError};

use once_cell::sync::OnceCell;
use fluvio_smartmodule::eyre;



use fluvio_smartmodule::{smartmodule, Result, Record};

#[smartmodule(filter)]
pub fn filter(record: &Record) -> Result<bool> {
    let string = std::str::from_utf8(record.value.as_ref())?;
    Ok(string.contains('a'))
}




static CRITERIA: OnceCell<String> = OnceCell::new();

#[smartmodule(init)]
fn init(params: SmartModuleExtraParams) -> Result<()> {
    // You can refer to the example SmartModules in Fluvio's GitHub Repository
    // https://github.com/infinyon/fluvio/tree/master/smartmodule
    if let Some(key) = params.get("key") {
        CRITERIA.set(key.clone()).map_err(|err| eyre!("failed setting key: {:#?}", err))
    } else {
        Err(SmartModuleInitError::MissingParam("key".to_string()).into())
    }
}
```

### Building SMDK Project

```bash
➜  Desktop cd ./log-filter
➜  log-filter git:(main) ✗ smdk build
    Updating git repository `https://github.com/infinyon/fluvio.git`
    Updating crates.io index
   Compiling proc-macro2 v1.0.46
```

### Testing SM Project using `smdk test`

```
➜  log-filter git:(main) ✗ smdk test --params "key=a" --text "testing"
project name: "log-filter"
loading module at: target/wasm32-unknown-unknown/release-lto/log_filter.wasm
0 records outputed
➜  log-filter git:(main) ✗ smdk test --params "key=a" --text "tasty"
project name: "log-filter"
loading module at: target/wasm32-unknown-unknown/release-lto/log_filter.wasm
1 records outputed
tasty
```
@bors
Copy link

bors bot commented Oct 2, 2022

Build failed:

@sehz
Copy link
Contributor

sehz commented Oct 2, 2022

jeez. Murphy's law strikes...

@sehz
Copy link
Contributor

sehz commented Oct 2, 2022

bors r+

bors bot pushed a commit that referenced this pull request Oct 2, 2022
Provides support to generate new SmartModules using `cargo-generate` as a libary.

Resolve: #2621

---

## Examples

### Generates SmartModule Projects using `smdk generate`

```bash
➜  Desktop smdk generate log-filter
Generating new SmartModule project: log-filter
🔧   Destination: /Users/esteban/Desktop/log-filter ...
🔧   Generating template ...
✔ 🤷   Which type of SmartModule would you like? · filter
✔ 🤷   Want to use SmartModule parameters? · false
🤷   SmartModule Version [default: git = "https://github.com/infinyon/fluvio.git"]: git = "https://github.com/infinyon/fluvio.git"
✔ 🤷   Want to use SmartModule init? · true
Ignoring: /var/folders/0g/l702s31x2mj4zdlw2_43rpyr0000gn/T/.tmpwQ4Yql/cargo-generate.toml
[1/5]   Done: Cargo.toml
[2/5]   Done: README.md
[3/5]   Done: Smart.toml
[4/5]   Done: src/lib.rs
[5/5]   Done: src
🔧   Moving generated files into: `/Users/esteban/Desktop/log-filter`...
💡   Initializing a fresh Git repository
✨   Done! New project created /Users/esteban/Desktop/log-filter
```

### Generated Project

```bash
total 24
drwxr-xr-x  7 esteban  staff   224B Oct  2 13:54 .
drwx------+ 6 esteban  staff   192B Oct  2 13:54 ..
drwxr-xr-x  9 esteban  staff   288B Oct  2 13:54 .git
-rw-r--r--  1 esteban  staff   358B Oct  2 13:54 Cargo.toml
-rw-r--r--  1 esteban  staff   1.8K Oct  2 13:54 README.md
-rw-r--r--  1 esteban  staff   152B Oct  2 13:54 Smart.toml
drwxr-xr-x  3 esteban  staff    96B Oct  2 13:54 src
```

### Generated `src/lib.rs` with the provided options

```rust
use fluvio_smartmodule::dataplane::smartmodule::{SmartModuleExtraParams, SmartModuleInitError};

use once_cell::sync::OnceCell;
use fluvio_smartmodule::eyre;



use fluvio_smartmodule::{smartmodule, Result, Record};

#[smartmodule(filter)]
pub fn filter(record: &Record) -> Result<bool> {
    let string = std::str::from_utf8(record.value.as_ref())?;
    Ok(string.contains('a'))
}




static CRITERIA: OnceCell<String> = OnceCell::new();

#[smartmodule(init)]
fn init(params: SmartModuleExtraParams) -> Result<()> {
    // You can refer to the example SmartModules in Fluvio's GitHub Repository
    // https://github.com/infinyon/fluvio/tree/master/smartmodule
    if let Some(key) = params.get("key") {
        CRITERIA.set(key.clone()).map_err(|err| eyre!("failed setting key: {:#?}", err))
    } else {
        Err(SmartModuleInitError::MissingParam("key".to_string()).into())
    }
}
```

### Building SMDK Project

```bash
➜  Desktop cd ./log-filter
➜  log-filter git:(main) ✗ smdk build
    Updating git repository `https://github.com/infinyon/fluvio.git`
    Updating crates.io index
   Compiling proc-macro2 v1.0.46
```

### Testing SM Project using `smdk test`

```
➜  log-filter git:(main) ✗ smdk test --params "key=a" --text "testing"
project name: "log-filter"
loading module at: target/wasm32-unknown-unknown/release-lto/log_filter.wasm
0 records outputed
➜  log-filter git:(main) ✗ smdk test --params "key=a" --text "tasty"
project name: "log-filter"
loading module at: target/wasm32-unknown-unknown/release-lto/log_filter.wasm
1 records outputed
tasty
```
@bors
Copy link

bors bot commented Oct 2, 2022

Pull request successfully merged into master.

Build succeeded:

@bors bors bot changed the title feat: generate SmartModules using SMDK [Merged by Bors] - feat: generate SmartModules using SMDK Oct 2, 2022
@bors bors bot closed this Oct 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SMDK: Generate new Smart module project
3 participants