[BUG] ProcessEvent method in the Integration connector is being driven twice for one event

[davidradl]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

in the sample lineage connector the processEvent method is driven twice for the same event. This can be seen in the audit log. The processEvent does not throw an Exception in either run, so Kafka should not be retrying the event.

Expected Behavior

the processEvent method should only be driven once

Steps To Reproduce

No response

Environment

- Egeria:
- OS:
- Java:
- Browser (for UI issues):
- Additional connectors and integration:

Any Further Information?

IK need to investigate further into the Kafka connector to understand what is happening. Maybe there is a Kafka trigger for this due to something in my kafka config that I could change.

[mandy-chessell]

This is not necessarily a problem. Multiple delivery of the same event is possible - this is often called at "least once" delivery. How do you have kafka configured?

[planetf1]

In a stable single system, single partition kafka 'cluster' we'd expect to get very rare cases of multiple delivery

From time to time there may be a cluster rebalance - this can occur when a topic is added for example, and it's possible the kafka connector is not correctly committing offset when this happens
At startup we also rewind the offset intentially, but only when there is no offset saved

If nothing obvious comes up in your code/configuration or inspection of the framework code .I would recommend turning on DEBUG (LOGGING_LEVEL_ROOT ). This helps us to understand kafka's view of the world - when the rebalance happens etc, but it can be very noisy. You can try just doing the connector, which should log salient info

But ultimately the code will need to handle the potential for never receiving an event, or duplicates unless configured to guarantee once only delivery (which we don't do -- overhead)

[planetf1]

I noticed #7149 -- something I plan to look into

[davidradl]

Thanks @planetf1 I will look into this, if it looks like the same issue - we can talk about who will fix it.

[davidradl]

@planetf1 - this is an artifact of the audit logging config I have. It is putting out 2 entries to the console for each log message. I suspect it is using he default log and then the destination defined in the connection configuration

[davidradl]

closing

[planetf1]

Thanks for clarifying. That's good to know it wasn't a functional issue