Feat/v0.8 (#6)

* add new resource secret, fix bugs, add tests and misc performance improvements

* bugfix: squash error, and cleanup code

* misc fixes and features
- cli / admin api bug fixes
- add secret resource
- performance
- raft updates (snapshots disabled)
- add metrics

* fix nil deref

* add more verbose metrics
improve metric error handling
add reverse proxy tests

* add new async sleep
add imple retry timeout
add disable metrics opt
refactor module logic

.gitignore

@@ -13,3 +13,4 @@ dist/
 .dgate*/
 cov.out
 go.work.sum
+.env

README.md

@@ -9,7 +9,7 @@
 ![GitHub Release](https://img.shields.io/github/v/release/dgate-io/dgate)
 
 
-DGate is a distributed API Gateway built for developers. DGate allows you to use JavaScript/TypeScript to modify request/response data(L7). Inpired by [k6](https://github.com/grafana/k6) and [kong](https://github.com/Kong/kong).
+DGate is a distributed API Gateway built for developers. DGate allows you to use JavaScript/TypeScript to modify request/response data(L7). Inspired by [k6](https://github.com/grafana/k6) and [kong](https://github.com/Kong/kong).
 
 > DGate is currently in development and is not ready for production use. Please use at your own discretion.
 
@@ -30,6 +30,10 @@ go install github.com/dgate-io/dgate/cmd/dgate-server@latest
 
 DGate Server is proxy and admin server bundled into one. the admin server is responsible for managing the state of the proxy server. The proxy server is responsible for routing requests to upstream servers. The admin server can also be used to manage the state of the cluster using the Raft Consensus Algorithm.
 
+### DGate CLI (dgate-cli)
+
+DGate CLI is a command-line interface that can be used to interact with the DGate Server. It can be used to deploy modules, manage the state of the cluster, and more.
+
 #### Proxy Modules
 
 - Fetch Upstream Module (`fetchUpstream`) - executed before the request is sent to the upstream server. This module is used to decided which upstream server to send the current request to. (Essentially a custom load balancer module)

TODO.md

@@ -11,11 +11,6 @@
 
 ## Replace zerolog with slog
 
-## Improve async function performance
-
-There is a pretty significant difference in performance when using async function.
-
-
 ## Add Module Tests
 
 - Test multiple modules being used at the same time
@@ -44,6 +39,10 @@ Also to execute code on specific events, like when a new route is added, or when
   - resource CRUD operations (namespace/domain/service/module/route/collection/document)
 - execute cron jobs: @every 1m, @cron 0 0 * * *, @daily, @weekly, @monthly, @yearly
 
+At a higher level, background jobs can be used to enable features like health checks, which can periodically check the health of the upstream servers and disable/enable them if they are not healthy.
+
+Other features include: automatic service discovery, ping-based load balancing, 
+
 # Metrics
 
 -- add support for prometheus, datadog, sentry, etc.
@@ -89,38 +88,6 @@ DGate Runtime is a JavaScript/TypeScript runtime that can be used to test module
 
 RuntimePool is a pool of runtimes that can be used to execute modules. It can be used to manage the different modules and clean up resources when they are no longer needed or idle for a certain amount of time.
 
-## TCP Proxy/Gateway (L4LB) (low priority)
-
-Using the same architecture as the HTTP Proxy/Gateway, create a TCP Proxy/Gateway that can be used to proxy TCP connections to upstream servers.
-
-A 'Custom Protocols API'can  allow users to define custom protocols that can be used to proxy TCP connections to upstream servers or handle the connections themselves.
-
-The custom protocols can be defined using JavaScript/TypeScript function or using protocol definitions (API) which will allow these values to be passed to the JavaScript/TypeScript code.
-
-```
-{
-  "name": "custom_protocol",
-  "version": "1",
-  "description": "Custom Protocol",
-  "modules": ["module_x"]
-  "format_definitions": [
-    {
-      "name": "command",
-      "type": "uint8"
-    }
-    {
-      "name": "data_len",
-      "type": "int16"
-    }
-    {
-      "name": "data",
-      "type": "string",
-      "length": "variable.data_len.length"
-    }
-  ]
-}
-```
-
 ## Server Tags
 
 No special characters are allowed in the tag name or value
@@ -149,3 +116,38 @@ time based tags
 ## Module Permissions
 
 - Allow users to define permissions for modules to access certain dgate resources/apis and/or OS resources.
+  - resource:document:read
+  - resource:document:write
+  - os:net:(http/tcp/udp)
+  - os:file:read
+  - os:env:read
+
+# Bundles
+
+- Add support for bundles that can be used to extend the functionality of DGate. Bundles are a grouping of resources that can be used to extend the functionality of DGate. Bundles can be used to add new modules, resources, and more.
+A good example of a bundle would be a bundle that adds support for OAuth2 authentication. It would need to setup the necessary routes, modules, and configurations to enable OAuth2 authentication. 
+
+## Module/Plugin Variables
+
+- Allow users to define variables that can be used in modules/plugins. These variables can be set by the user, eventually the Admin Console should allow these variables to be set, and the variables can be used in the modules/plugins.
+
+## Mutual TLS Support (low priority)
+
+## Versioning Modules
+
+Differing from common resource versioning, modules can have multiple versions that can be used at the same time. This can be used to test new versions of modules before deploying them to the cluster.
+
+
+## DGate CLI - argument variable suggestions
+
+For example, if the user types an argument that is not recognized, the CLI can suggest the correct argument by search the available arguments and finding the closest match.
+```
+dgate-cli ns mk my-ns nmae=my-ns
+Variable 'nmae' is not recognized. Did you mean 'name'?
+```
+
+## Improve Module Debugability
+
+Make it easier to debug modules by adding more logging and error handling. This can be done by adding more logging to the modules and making it easier to see the logs in the Admin Console.
+
+Add stack tracing for typescript modules.

cmd/dgate-cli/commands/collection.go → cmd/dgate-cli/commands/collection_commands.go

@@ -57,7 +57,13 @@ func CollectionCommand(client *dgclient.DGateClient) *cli.Command {
 				Aliases: []string{"ls"},
 				Usage:   "list collections",
 				Action: func(ctx *cli.Context) error {
-					col, err := client.ListCollection()
+					nsp, err := createMapFromArgs[dgclient.NamespacePayload](
+						ctx.Args().Slice(),
+					)
+					if err != nil {
+						return err
+					}
+					col, err := client.ListCollection(nsp.Namespace)
 					if err != nil {
 						return err
 					}

cmd/dgate-cli/commands/document.go → cmd/dgate-cli/commands/document_commands.go

@@ -37,18 +37,37 @@ func DocumentCommand(client *dgclient.DGateClient) *cli.Command {
 				Name:    "delete",
 				Aliases: []string{"rm"},
 				Usage:   "delete a document",
+				Flags: []cli.Flag{
+					&cli.BoolFlag{
+						Name:  "all",
+						Usage: "delete all documents",
+					},
+				},
 				Action: func(ctx *cli.Context) error {
-					doc, err := createMapFromArgs[spec.Document](
-						ctx.Args().Slice(), "id",
-					)
-					if err != nil {
-						return err
-					}
-					err = client.DeleteDocument(
-						doc.ID, doc.NamespaceName,
-					)
-					if err != nil {
-						return err
+					if ctx.Bool("all") {
+						doc, err := createMapFromArgs[spec.Document](
+							ctx.Args().Slice())
+						if err != nil {
+							return err
+						}
+						err = client.DeleteAllDocument(
+							doc.NamespaceName, doc.CollectionName)
+						if err != nil {
+							return err
+						}
+						return nil
+					} else {
+						doc, err := createMapFromArgs[spec.Document](
+							ctx.Args().Slice(), "id",
+						)
+						if err != nil {
+							return err
+						}
+						err = client.DeleteDocument(doc.ID,
+							doc.NamespaceName, doc.CollectionName)
+						if err != nil {
+							return err
+						}
 					}
 					return nil
 				},
@@ -58,7 +77,14 @@ func DocumentCommand(client *dgclient.DGateClient) *cli.Command {
 				Aliases: []string{"ls"},
 				Usage:   "list documents",
 				Action: func(ctx *cli.Context) error {
-					doc, err := client.ListDocument()
+					d, err := createMapFromArgs[spec.Document](
+						ctx.Args().Slice(), "collection",
+					)
+					if err != nil {
+						return err
+					}
+					doc, err := client.ListDocument(
+						d.NamespaceName, d.CollectionName)
 					if err != nil {
 						return err
 					}
@@ -75,9 +101,8 @@ func DocumentCommand(client *dgclient.DGateClient) *cli.Command {
 					if err != nil {
 						return err
 					}
-					doc, err = client.GetDocument(
-						doc.ID, doc.NamespaceName,
-					)
+					doc, err = client.GetDocument(doc.ID,
+						doc.NamespaceName, doc.CollectionName)
 					if err != nil {
 						return err
 					}

cmd/dgate-cli/commands/domain.go → cmd/dgate-cli/commands/domain_commands.go

@@ -57,7 +57,13 @@ func DomainCommand(client *dgclient.DGateClient) *cli.Command {
 				Aliases: []string{"ls"},
 				Usage:   "list domains",
 				Action: func(ctx *cli.Context) error {
-					dom, err := client.ListDomain()
+					nsp, err := createMapFromArgs[dgclient.NamespacePayload](
+						ctx.Args().Slice(),
+					)
+					if err != nil {
+						return err
+					}
+					dom, err := client.ListDomain(nsp.Namespace)
 					if err != nil {
 						return err
 					}

cmd/dgate-cli/commands/module.go → cmd/dgate-cli/commands/module_commands.go

@@ -57,7 +57,13 @@ func ModuleCommand(client *dgclient.DGateClient) *cli.Command {
 				Aliases: []string{"ls"},
 				Usage:   "list modules",
 				Action: func(ctx *cli.Context) error {
-					mod, err := client.ListModule()
+					nsp, err := createMapFromArgs[dgclient.NamespacePayload](
+						ctx.Args().Slice(),
+					)
+					if err != nil {
+						return err
+					}
+					mod, err := client.ListModule(nsp.Namespace)
 					if err != nil {
 						return err
 					}

cmd/dgate-cli/commands/route.go → cmd/dgate-cli/commands/route_commands.go

@@ -58,7 +58,13 @@ func RouteCommand(client *dgclient.DGateClient) *cli.Command {
 				Aliases: []string{"ls"},
 				Usage:   "list routes",
 				Action: func(ctx *cli.Context) error {
-					rt, err := client.ListRoute()
+					nsp, err := createMapFromArgs[dgclient.NamespacePayload](
+						ctx.Args().Slice(),
+					)
+					if err != nil {
+						return err
+					}
+					rt, err := client.ListRoute(nsp.Namespace)
 					if err != nil {
 						return err
 					}

cmd/dgate-cli/commands/secret_commands.go

@@ -0,0 +1,95 @@
+package commands
+
+import (
+	"github.com/dgate-io/dgate/pkg/dgclient"
+	"github.com/dgate-io/dgate/pkg/spec"
+	"github.com/urfave/cli/v2"
+)
+
+func SecretCommand(client *dgclient.DGateClient) *cli.Command {
+	return &cli.Command{
+		Name:      "secret",
+		Aliases:   []string{"sec"},
+		Args:      true,
+		ArgsUsage: "<command> <name>",
+		Usage:     "secret commands",
+		Subcommands: []*cli.Command{
+			{
+				Name:    "create",
+				Aliases: []string{"mk"},
+				Usage:   "create a secret",
+				Action: func(ctx *cli.Context) error {
+					sec, err := createMapFromArgs[spec.Secret](
+						ctx.Args().Slice(), "name", "data",
+					)
+					if err != nil {
+						return err
+					}
+					err = client.CreateSecret(sec)
+					if err != nil {
+						return err
+					}
+					// redact the data field
+					sec.Data = "**redacted**"
+					return jsonPrettyPrint(sec)
+				},
+			},
+			{
+				Name:    "delete",
+				Aliases: []string{"rm"},
+				Usage:   "delete a secret",
+				Action: func(ctx *cli.Context) error {
+					sec, err := createMapFromArgs[spec.Secret](
+						ctx.Args().Slice(), "name",
+					)
+					if err != nil {
+						return err
+					}
+					err = client.DeleteSecret(
+						sec.Name, sec.NamespaceName)
+					if err != nil {
+						return err
+					}
+					return nil
+				},
+			},
+			{
+				Name:    "list",
+				Aliases: []string{"ls"},
+				Usage:   "list services",
+				Action: func(ctx *cli.Context) error {
+					nsp, err := createMapFromArgs[dgclient.NamespacePayload](
+						ctx.Args().Slice(),
+					)
+					if err != nil {
+						return err
+					}
+					sec, err := client.ListSecret(nsp.Namespace)
+					if err != nil {
+						return err
+					}
+					return jsonPrettyPrint(sec)
+				},
+			},
+			{
+				Name:  "get",
+				Usage: "get a secret",
+				Action: func(ctx *cli.Context) error {
+					s, err := createMapFromArgs[spec.Secret](
+						ctx.Args().Slice(), "name",
+					)
+					if err != nil {
+						return err
+					}
+					sec, err := client.GetSecret(
+						s.Name, s.NamespaceName,
+					)
+					if err != nil {
+						return err
+					}
+					return jsonPrettyPrint(sec)
+				},
+			},
+		},
+	}
+}

cmd/dgate-cli/commands/service.go → cmd/dgate-cli/commands/service_commands.go

@@ -57,7 +57,13 @@ func ServiceCommand(client *dgclient.DGateClient) *cli.Command {
 				Aliases: []string{"ls"},
 				Usage:   "list services",
 				Action: func(ctx *cli.Context) error {
-					svc, err := client.ListService()
+					nsp, err := createMapFromArgs[dgclient.NamespacePayload](
+						ctx.Args().Slice(),
+					)
+					if err != nil {
+						return err
+					}
+					svc, err := client.ListService(nsp.Namespace)
 					if err != nil {
 						return err
 					}