Skip to content

Commit

Permalink
fix: add logs in traps server for situations when incoming traps have…
Browse files Browse the repository at this point in the history 
… wrong credentials

fix: delete unwanted files

fix: add logging of source device ip

fix: log only necessary info

fix: run pre-commit

fix: update the troubleshooting documentation

fix: add comment in code
  • Loading branch information
@wojtekzyla
wojtekzyla committed Feb 7, 2024
1 parent 8d9399f commit 9c5d43c
Show file tree
Hide file tree
Showing 2 changed files with 145 additions and 1 deletion.
123 changes: 122 additions & 1 deletion docs/bestpractices.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -103,4 +103,125 @@ If the changes are required to be applied immedietly, the previous inventory job
```
microk8s kubectl delete job/snmp-splunk-connect-for-snmp-inventory -n sc4snmp
```
The upgrade command can be executed again.
The upgrade command can be executed again.
### Traps messages don't appear in Splunk
The first possible answer to why traps aren't sent to Splunk is that SNMP agents send trap messages to the wrong IP
address or port. To check what is the correct address of traps server, run the following command:
```
microk8s kubectl -n sc4snmp get services
```
This command should output similar data:
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
snmp-redis-headless ClusterIP None <none> 6379/TCP 113s
snmp-mibserver ClusterIP 10.152.183.163 <none> 80/TCP 113s
snmp-mongodb ClusterIP 10.152.183.118 <none> 27017/TCP 113s
snmp-redis-master ClusterIP 10.152.183.61 <none> 6379/TCP 113s
snmp-mongodb-metrics ClusterIP 10.152.183.50 <none> 9216/TCP 113s
snmp-splunk-connect-for-snmp-trap LoadBalancer 10.152.183.190 114.241.233.134 162:32180/UDP 113s
```
Check the `EXTERNAL-IP` of `snmp-splunk-connect-for-snmp-trap` and the second port number for this service. In this case
the full `snmp-splunk-connect-for-snmp-trap` address will be `114.241.233.134:32180`.
In case agents send traps to the correct address, but there is still no data in the `netops` index, there might some
issues with credentials. These errors can be seen in logs if the `snmp-splunk-connect-for-snmp-trap` pod. Logs can be
accessed either in Splunk in case of using [sck-otel](gettingstarted/sck-installation.md) or directly in kubernetes
using the terminal.
#### Accessing logs via Splunk
If [sck-otel](gettingstarted/sck-installation.md) is installed, run the following search query in Splunk
`index=em_logs sourcetype="kube:container:splunk-connect-for-snmp-traps" error`
#### Accessing logs in kubernetes
to access logs directly in kubernetes, first run `microk8s kubectl -n sc4snmp get pods`. This will output all pods:
```
NAME READY STATUS RESTARTS AGE
snmp-splunk-connect-for-snmp-worker-trap-99f49c557-j9jwx 1/1 Running 0 29m
snmp-splunk-connect-for-snmp-trap-56f75f9754-kmlgb 1/1 Running 0 29m
snmp-splunk-connect-for-snmp-scheduler-7bb8c79855-rgjkj 1/1 Running 0 29m
snmp-mibserver-784bd599fd-6xzfj 1/1 Running 0 29m
snmp-splunk-connect-for-snmp-worker-poller-78b46d668f-59mv4 1/1 Running 0 29m
snmp-splunk-connect-for-snmp-worker-sender-6f8496bfbf-cvt9l 1/1 Running 0 29m
snmp-mongodb-7579dc7867-mlnst 2/2 Running 0 29m
snmp-redis-master-0 1/1 Running 0 29m
```
Now get logs of the `snmp-splunk-connect-for-snmp-trap`:
```
microk8s kubectl -n sc4snmp logs pod/snmp-splunk-connect-for-snmp-trap-56f75f9754-kmlgb
```
#### Error messages for different authentication problems
##### Wrong community string
In case of using community string for authentication purposes, the following error should be expected if the arriving trap
has a community string not configured in SC4SNMP:
```
2024-02-06 15:42:14,885 ERROR Security Model failure for device ('18.226.181.199', 42514): Unknown SNMP community name encountered
```
If this error occurs, check if the appropriate community is defined under `traps.communities` in `values.yaml`. Example
`public` community configuration:
```yaml
traps:
communities:
public:
communityIndex:
contextEngineId:
contextName:
tag:
securityName:
```

##### Wrong security name or snmp engine id

While sending SNMP v3 traps in case of wrong username or engine id configuration, the following error should be expected:
```
2024-02-06 15:42:14,091 ERROR Security Model failure for device ('18.226.181.199', 46066): Unknown SNMP security name encountered
```

If this error occurs, make sure that the kubernetes secret with the correct username has been created ([SNMPv3 configuration](configuration/snmpv3-configuration.md)).
After creating the secret, add it under `traps.usernameSecrets` in `values.yaml`. Also make sure, that the correct snmp engine id
is configured under `traps.securityEngineId`. Example of `values.yaml` with configured secret and engine id:
```yaml
traps:
usernameSecrets:
- my-secret-name
securityEngineId:
- "090807060504030201"
```
##### Wrong authentication protocol or password
While sending SNMP v3 traps in case of wrong authentication protocol or password configuration, the following error should be expected:
```
2024-02-06 15:42:14,642 ERROR Security Model failure for device ('18.226.181.199', 54806): Authenticator mismatched
```
If this error occurs, make sure that the kubernetes secret with the correct authentication protocol and password has been created ([SNMPv3 configuration](configuration/snmpv3-configuration.md)).
After creating the secret, add it under `traps.usernameSecrets` in `values.yaml`. Example of `values.yaml` with configured secret:
```yaml
traps:
usernameSecrets:
- my-secret-name
```

##### Wrong privacy protocol or password
While sending SNMP v3 traps in case of wrong privacy protocol or password configuration, the following error should be expected:
```
2024-02-06 15:42:14,780 ERROR Security Model failure for device ('18.226.181.199', 48249): Ciphering services not available or ciphertext is broken
```
If this error occurs, make sure that the kubernetes secret with the correct privacy protocol and password has been created ([SNMPv3 configuration](configuration/snmpv3-configuration.md)).
After creating the secret, add it under `traps.usernameSecrets` in `values.yaml`. Example of `values.yaml` with configured secret:
```yaml
traps:
usernameSecrets:
- my-secret-name
```
23 changes: 23 additions & 0 deletions splunk_connect_for_snmp/traps.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@

import asyncio
import os
from typing import Any, Dict

import yaml
from celery import Celery, chain
Expand All @@ -44,9 +45,13 @@
from splunk_connect_for_snmp.snmp.tasks import trap
from splunk_connect_for_snmp.splunk.tasks import prepare, send

# from pysnmp import debug

provider = TracerProvider()
trace.set_tracer_provider(provider)

# debug.setLogger(debug.Debug('secmod'))

CONFIG_PATH = os.getenv("CONFIG_PATH", "/app/config/config.yaml")
SECURITY_ENGINE_ID_LIST = os.getenv("SNMP_V3_SECURITY_ENGINE_ID", "80003a8c04").split(
","
Expand Down Expand Up @@ -96,6 +101,13 @@ def cbFun(snmpEngine, stateReference, contextEngineId, contextName, varBinds, cb
_ = my_chain.apply_async()


def authentication_observer_cb_fun(snmpEngine, execpoint, variables, contexts):
logging.error(
f"Security Model failure for device {variables.get('transportAddress', None)}: "
f"{variables.get('statusInformation', {}).get('errorIndication', None)}"
)


app.autodiscover_tasks(
packages=[
"splunk_connect_for_snmp",
Expand All @@ -111,9 +123,20 @@ def main():
# Get the event loop for this thread
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)

# Create SNMP engine with autogenernated engineID and pre-bound
# to socket transport dispatcher
snmpEngine = engine.SnmpEngine()

# Register callback function to log errors with traps authentication
observer_context: Dict[Any, Any] = {}
snmpEngine.observer.registerObserver(
authentication_observer_cb_fun,
"rfc2576.prepareDataElements:sm-failure",
"rfc3412.prepareDataElements:sm-failure",
cbCtx=observer_context,
)

# UDP over IPv4, first listening interface/port
config.addTransport(
snmpEngine,
Expand Down

0 comments on commit 9c5d43c

Please sign in to comment.