Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added compatibility with Neptune Analytics #241

Merged
merged 19 commits into from
Mar 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
**/node_modules/
**/coverage/
**/.DS_Store
**/.vs/
**/.idea/
**/.vs/
4 changes: 4 additions & 0 deletions Changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ This release includes the following feature enhancements and bug fixes:
- Bumped `vite` to `4.5.2` (<https://github.com/aws/graph-explorer/pull/233>)
- Searching for text containing quotes now works.

**Features**

- Added compatibility with Neptune Analitycs.

## Release 1.5.0

This release includes the following feature enhancements and bug fixes:
Expand Down
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ You can create and manage connections to graph databases using this feature. Con
- **Using proxy server:** Check this box if using a proxy endpoint.
- **Graph connection URL:** Provide the endpoint for the graph database
- **AWS IAM Auth Enabled:** Check this box if connecting to Amazon Neptune using IAM Auth and SigV4 signed requests
- **Service Type:** Choose the service type
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's explain inline (so it is easier for folks to understand) - choose service time neptune-db for Neptune database, neptune-graph for Neptune Analytics or empty when not using IAM authentication. ?

- **AWS Region:** Specify the AWS region where the Neptune cluster is hosted (e.g., us-east-1)
- **Fetch Timeout:** Specify the timeout for the fetch request

Expand Down Expand Up @@ -122,6 +123,7 @@ To provide a default connection such that initial loads of the graph explorer al
- `GRAPH_CONNECTION_URL` - `None` - See [Add a New Connection](#connections-ui)
- Required if `USING_PROXY_SERVER=True` and `IAM=True`
- `AWS_REGION` - `None` - See [Add a New Connection](#connections-ui)
- `SERVICE_TYPE` - `neptune-db`, Set this as `neptune-db` for Neptune database or `neptune-graph` for Neptune Analytics.
alexey-temnikov marked this conversation as resolved.
Show resolved Hide resolved

#### JSON Configuration Approach

Expand All @@ -133,6 +135,7 @@ First, create a `config.json` file containing values for the connection attribut
"GRAPH_CONNECTION_URL": "https://cluster-cqmizgqgrsbf.us-west-2.neptune.amazonaws.com:8182",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"GRAPH_CONNECTION_URL": "https://cluster-cqmizgqgrsbf.us-west-2.neptune.amazonaws.com:8182",
"GRAPH_CONNECTION_URL": "https://cluster-somecluster.us-west-2.neptune.amazonaws.com:8182",

"USING_PROXY_SERVER": true, (Can be string or boolean)
"IAM": true, (Can be string or boolean)
"SERVICE_TYPE": "neptune-db",
"AWS_REGION": "us-west-2",
"GRAPH_TYPE": "gremlin" (Possible Values: "gremlin", "sparql", "opencypher"),
"GRAPH_EXP_HTTPS_CONNECTION": true (Can be string or boolean),
Expand Down Expand Up @@ -160,6 +163,7 @@ docker run -p 80:80 -p 443:443 \
--env IAM=false \
--env GRAPH_CONNECTION_URL=https://cluster-cqmizgqgrsbf.us-west-2.neptune.amazonaws.com:8182 \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
--env GRAPH_CONNECTION_URL=https://cluster-cqmizgqgrsbf.us-west-2.neptune.amazonaws.com:8182 \
--env GRAPH_CONNECTION_URL=https://cluster-somecluster.us-west-2.neptune.amazonaws.com:8182 \

Exact endpoints should not be used in the documentation.

--env AWS_REGION=us-west-2 \
--env SERVICE_TYPE=neptune-db \
--env PROXY_SERVER_HTTPS_CONNECTION=true \
--env GRAPH_EXP_FETCH_REQUEST_TIMEOUT=240000 \
graph-explorer
Expand Down
5 changes: 5 additions & 0 deletions additionaldocs/ecs/ECS_FARGATE_DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,10 @@ After the request is processed, the console will return you to your certificate
"name": "HOST",
"value": "localhost"
},
{
"name": "SERVICE_TYPE",
"value": "neptune-db"
},
{
"name": "GRAPH_CONNECTION_URL",
"value": "https://{NEPTUNE_ENDPOINT}:8182"
Expand Down Expand Up @@ -157,6 +161,7 @@ After the request is processed, the console will return you to your certificate
- `IAM`: Set this to `true` to use SigV4 signed requests, if your Neptune cluster has IAM db authentication enabled.
- `GRAPH_CONNECTION_URL`: Set this as `https://{NEPTUNE_ENDPOINT}:8182`.
- `PUBLIC_OR_PROXY_ENDPOINT`: Set this as `https://{Domain name set in Step 5 of "Request an ACM Public Certificate"}`.
- `SERVICE_TYPE`: Set this as `neptune-db` for Neptune database or `neptune-graph` for Neptune Analytics.
6. Click **Create**.

### Create a Fargate Service
Expand Down
7 changes: 7 additions & 0 deletions additionaldocs/sagemaker/graph-explorer-policy.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,13 @@
"arn:aws:neptune-db:[AWS_REGION]:[AWS_ACCOUNT_ID]:[NEPTUNE_CLUSTER_RESOURCE_ID]/*"
]
},
{
"Effect": "Allow",
"Action": "neptune-graph:*",
"Resource": [
"arn:aws:neptune-graph:[AWS_REGION]:[AWS_ACCOUNT_ID]:[NEPTUNE_CLUSTER_RESOURCE_ID]/*"
]
},
{
"Effect": "Allow",
"Action": "sagemaker:DescribeNotebookInstance",
Expand Down
1 change: 1 addition & 0 deletions additionaldocs/sagemaker/install-graph-explorer-lc.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ sudo -u ec2-user -i <<'EOF'

echo "export GRAPH_NOTEBOOK_AUTH_MODE=DEFAULT" >> ~/.bashrc # set to IAM instead of DEFAULT if cluster is IAM enabled
echo "export GRAPH_NOTEBOOK_HOST=CHANGE-ME" >> ~/.bashrc
echo "export GRAPH_NOTEBOOK_SERVICE=neptune-db" >> ~/.bashrc # set to `neptune-db` for Neptune database or `neptune-graph` for Neptune Analytics
echo "export GRAPH_NOTEBOOK_PORT=8182" >> ~/.bashrc
echo "export AWS_REGION=us-west-2" >> ~/.bashrc # modify region if needed

Expand Down
27 changes: 0 additions & 27 deletions packages/graph-explorer-proxy-server/RequestSig.js

This file was deleted.

38 changes: 25 additions & 13 deletions packages/graph-explorer-proxy-server/node-server.js
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ const aws4 = require("aws4");
// Load environment variables from .env file.
dotenv.config({ path: "../graph-explorer/.env" });

const DEFAULT_SERVICE_TYPE = "neptune-db";
const NEPTUNE_ANALYTICS_SERVICE_TYPE = "neptune-graph";

// Create a logger instance with pino.
const proxyLogger = pino({
level: process.env.LOG_LEVEL || "info",
Expand Down Expand Up @@ -69,6 +72,7 @@ const retryFetch = async (
options,
isIamEnabled,
region,
serviceType,
retryDelay = 10000,
refetchMaxRetries = 1
) => {
Expand All @@ -78,7 +82,7 @@ const retryFetch = async (
host: url.hostname,
port: url.port,
path: url.pathname + url.search,
service: "neptune-db",
service: serviceType,
region,
method: options.method,
body: options.body ?? undefined,
Expand All @@ -88,7 +92,7 @@ const retryFetch = async (
host: url.hostname,
port: url.port,
path: url.pathname + url.search,
service: "neptune-db",
service: serviceType,
region,
method: options.method,
body: options.body ?? undefined,
Expand All @@ -99,7 +103,7 @@ const retryFetch = async (
host: url.hostname,
port: url.port,
path: url.pathname + url.search,
service: "neptune-db",
service: serviceType,
method: options.method,
body: options.body ?? undefined,
headers: options.headers,
Expand Down Expand Up @@ -129,13 +133,14 @@ const retryFetch = async (
};

// Function to fetch data from the given URL and send it as a response.
async function fetchData(res, next, url, options, isIamEnabled, region) {
async function fetchData(res, next, url, options, isIamEnabled, region, serviceType) {
try {
const response = await retryFetch(
new URL(url),
options,
isIamEnabled,
region
region,
serviceType
);
const data = await response.json();
res.send(data);
Expand Down Expand Up @@ -185,8 +190,9 @@ async function fetchData(res, next, url, options, isIamEnabled, region) {
};
const isIamEnabled = !!req.headers["aws-neptune-region"];
const region = isIamEnabled ? req.headers["aws-neptune-region"] : "";
const serviceType = isIamEnabled ? (req.headers["service-type"] ?? DEFAULT_SERVICE_TYPE) : "";

fetchData(res, next, rawUrl, requestOptions, isIamEnabled, region);
fetchData(res, next, rawUrl, requestOptions, isIamEnabled, region, serviceType);
});

// POST endpoint for Gremlin queries.
Expand All @@ -209,8 +215,9 @@ async function fetchData(res, next, url, options, isIamEnabled, region) {

const isIamEnabled = !!req.headers["aws-neptune-region"];
const region = isIamEnabled ? req.headers["aws-neptune-region"] : "";
const serviceType = isIamEnabled ? (req.headers["service-type"] ?? DEFAULT_SERVICE_TYPE) : "";

fetchData(res, next, rawUrl, requestOptions, isIamEnabled, region);
fetchData(res, next, rawUrl, requestOptions, isIamEnabled, region, serviceType);
});

// POST endpoint for openCypher queries.
Expand All @@ -233,22 +240,26 @@ async function fetchData(res, next, url, options, isIamEnabled, region) {

const isIamEnabled = !!req.headers["aws-neptune-region"];
const region = isIamEnabled ? req.headers["aws-neptune-region"] : "";
const serviceType = isIamEnabled ? (req.headers["service-type"] ?? DEFAULT_SERVICE_TYPE) : "";

fetchData(res, next, rawUrl, requestOptions, isIamEnabled, region);
fetchData(res, next, rawUrl, requestOptions, isIamEnabled, region, serviceType);
});

// GET endpoint to retrieve PostgreSQL statistics summary.
// GET endpoint to retrieve statistics summary.
app.get("/pg/statistics/summary", async (req, res, next) => {
const rawUrl = `${req.headers["graph-db-connection-url"]}/pg/statistics/summary?mode=detailed`;
const isIamEnabled = !!req.headers["aws-neptune-region"];
const serviceType = isIamEnabled ? (req.headers["service-type"] ?? DEFAULT_SERVICE_TYPE) : "";
const rawUrl = serviceType === NEPTUNE_ANALYTICS_SERVICE_TYPE
? `${req.headers["graph-db-connection-url"]}/summary?mode=detailed`
: `${req.headers["graph-db-connection-url"]}/pg/statistics/summary?mode=detailed`;

const requestOptions = {
method: "GET",
};

const isIamEnabled = !!req.headers["aws-neptune-region"];
const region = isIamEnabled ? req.headers["aws-neptune-region"] : "";

fetchData(res, next, rawUrl, requestOptions, isIamEnabled, region);
fetchData(res, next, rawUrl, requestOptions, isIamEnabled, region, serviceType);
});

// GET endpoint to retrieve RDF statistics summary.
Expand All @@ -261,8 +272,9 @@ async function fetchData(res, next, url, options, isIamEnabled, region) {

const isIamEnabled = !!req.headers["aws-neptune-region"];
const region = isIamEnabled ? req.headers["aws-neptune-region"] : "";
const serviceType = isIamEnabled ? (req.headers["service-type"] ?? DEFAULT_SERVICE_TYPE) : "";

fetchData(res, next, rawUrl, requestOptions, isIamEnabled, region);
fetchData(res, next, rawUrl, requestOptions, isIamEnabled, region, serviceType);
});

app.get("/logger", (req, res, next) => {
Expand Down
3 changes: 3 additions & 0 deletions packages/graph-explorer/.env
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,6 @@
LOG_LEVEL=info
## Client side fetch request timeout
GRAPH_EXP_FETCH_REQUEST_TIMEOUT=240000
## Service
SERVICE_TYPE=neptune-db

Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ const fetchVerticesAttributes = async (
const response = await openCypherFetch<RawVerticesSchemaResponse>(verticesTemplate);

const vertex = response.results[0]?.object as OCVertex;
if (!vertex) return;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: avoid adding a return statement, and wrap vertices.push({ within the condition, to avoid fragmented logic with multiple returns.

Suggested change
if (!vertex) return;
if (vertex) {
};

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's return early pattern

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @vkagamlyk in this case. Returning early when checking assertions is good practice.

The only nit I would have is to not use inline if statements. While it is more code, it is also more readable. It's much easier to find the places in a method that return early if the return call is in a predictable place. Having it inline makes the placement depend on the condition expression length.

if (!vertex) {
  return;
}

const label = vertex["~labels"][0] as string;
const properties = vertex["~properties"];
vertices.push({
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ const useOpenCypher = () => {
...ops
});

summary = response.payload.graphSummary as GraphSummary || undefined;
summary = (response.payload ? response.payload.graphSummary as GraphSummary : response.graphSummary as GraphSummary) || undefined;
} catch (e) {
if (import.meta.env.DEV) {
console.error("[Summary API]", e);
Expand Down
2 changes: 2 additions & 0 deletions packages/graph-explorer/src/connector/useGEFetch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import { useCallback } from 'react';
import localforage from "localforage";
import { CacheItem } from './useGEFetchTypes';
import { useConfiguration, type ConnectionConfig } from '../core';
import { DEFAULT_SERVICE_TYPE } from "../utils/constants";

// 10 minutes
const CACHE_TIME_MS = 10 * 60 * 1000;
Expand Down Expand Up @@ -46,6 +47,7 @@ const useGEFetch = () => {
}
if (connection?.awsAuthEnabled) {
headers["aws-neptune-region"] = connection.awsRegion || "";
headers["service-type"] = connection.serviceType || DEFAULT_SERVICE_TYPE;
}

return { ...headers, ...typeHeaders };
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,10 @@ export type ConnectionConfig = {
* If it is Neptune, it could need authentication.
*/
awsAuthEnabled?: boolean;
/**
* If it is Neptune, it could need authentication.
*/
serviceType?: 'neptune-db' | 'neptune-graph',
alexey-temnikov marked this conversation as resolved.
Show resolved Hide resolved
/**
* AWS Region where the Neptune cluster is deployed.
* It is needed to sign requests.
Expand Down
2 changes: 2 additions & 0 deletions packages/graph-explorer/src/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import App from "./App";
import { RawConfiguration } from "./core";
import ConnectedProvider from "./core/ConnectedProvider";
import "./index.css";
import { DEFAULT_SERVICE_TYPE } from "./utils/constants";

const grabConfig = async (): Promise<RawConfiguration | undefined> => {
const defaultConnectionPath = `${location.origin}/defaultConnection`;
Expand Down Expand Up @@ -62,6 +63,7 @@ const grabConfig = async (): Promise<RawConfiguration | undefined> => {
graphDbUrl: defaultConnectionData.GRAPH_EXP_CONNECTION_URL || "",
awsAuthEnabled: !!defaultConnectionData.GRAPH_EXP_IAM,
awsRegion: defaultConnectionData.GRAPH_EXP_AWS_REGION || "",
serviceType: defaultConnectionData.SERVICE_TYPE || DEFAULT_SERVICE_TYPE,
fetchTimeoutMs:
defaultConnectionData.GRAPH_EXP_FETCH_REQUEST_TIMEOUT || 240000,
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -210,13 +210,14 @@ const ConnectionDetail = ({ isSync, onSyncChange }: ConnectionDetailProps) => {
<CreateConnection
onClose={() => setEdit(false)}
configId={config.id}
disabledFields={config.__fileBase ? ["type", "url"] : undefined}
disabledFields={config.__fileBase ? ["type", "url", "serviceType"] : undefined}
initialData={{
...(config.connection || {}),
name: config.displayLabel || config.id,
url: config.connection?.url,
type: config.connection?.queryEngine,
fetchTimeMs: config.connection?.fetchTimeoutMs,
serviceType: config.connection?.serviceType,
alexey-temnikov marked this conversation as resolved.
Show resolved Hide resolved
}}
/>
</Modal>
Expand Down
Loading
Loading