forked from cisagov/Malcolm
-
Notifications
You must be signed in to change notification settings - Fork 58
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
support both ElasticSearch and OpenSearch output #258
Labels
enhancement
New feature or request
install.py
Relating to the install.py configuration script
opensearch
Relating to Malcolm's use of OpenSearch
Milestone
Comments
mmguero
added
opensearch
Relating to Malcolm's use of OpenSearch
install.py
Relating to the install.py configuration script
enhancement
New feature or request
labels
Sep 7, 2023
this would be very useful, thank you. |
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Sep 18, 2023
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Sep 18, 2023
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Sep 18, 2023
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Sep 18, 2023
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Sep 19, 2023
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Oct 3, 2023
I've got this mostly working in my development fork. At this point it's a lot of heavy testing and writing documentation, but it should work in the next release. |
things I still need to figure out:
things I'm going to live with:
|
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Oct 11, 2023
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Oct 11, 2023
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Oct 11, 2023
This all seems to be working. I'll release for v23.10.0 and then open individual issues if they crop up later. |
This was referenced Oct 25, 2023
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
enhancement
New feature or request
install.py
Relating to the install.py configuration script
opensearch
Relating to Malcolm's use of OpenSearch
We've had a number of interested parties wanting to be able to store logs in both OpenSearch and ElasticSearch.
The index store that is managed by Malcolm will always (for the foreseeable future) be OpenSearch. This is due 1) to the licensing changes that initially caused us to go this direction and 2) to the features in Elasticsearch that would only be available to "premium" subscribers that we can get in open-source OpenSearch.
However, because we already support opensearch instances that are "local" vs. "remote" the code is already in place in Malcolm to use another data store into which to index the data. As of Q3 2023, the APIs between the two projects are still pretty similar.
So what I'm envisioning is this:
Should they choose Elasticsearch, Malcolm won't run its opensearch container (the same way it doesn't today if they choose "OpenSearch (remote)") and it also won't run the dashboards container (since OpenSearch Dashboards isn't going to run against ElasticSearch).
Let's be clear: malcolm won't run an instance of ElasticSearch or Kibana. But it will be able to index the network data into an ElasticSearch instance and the user would be able to see that data in their own Kibana.
Here are a few areas we'd need to look at:
We shouldn't need to worry about:
Honestly it's not too much that deals directly with the opensearch API. It's pretty easy to determine where those places are: since we're already handling remote opensearch instances, just look anywhere in the docker-compose file where we're referencing
.opensearch.primary.curlrc
and that is a place that is talking to OpenSearch, and as such a place we'd need to support ElasticSearch.The text was updated successfully, but these errors were encountered: