-
Notifications
You must be signed in to change notification settings - Fork 292
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
trace-load-limits: restrict trace load for applications #671
Open
alexmohr
wants to merge
5
commits into
COVESA:master
Choose a base branch
from
mercedes-benz:add-trace-load-limits
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Commits on Aug 8, 2024
-
trace-load-limits: restrict trace load for applications
In a shared system many developers log as much as they want into DLT. This can lead into overloading the logging system resulting in bad performance and dropped logs. This commit introduces trace load limits to restrict applications to a certain log volume measured in bytes/s. It is based on COVESA#134 but extends this heavily. Trace load limits are configured via a space separted configuraiton file. The format of the file follows: APPID [CTXID] SOFT_LIMIT HARD_LIMIT The most matching entry will be selected for each log, meaning that either app and context must match or at least the app id, for which a default is created when not configured. This allows to configure trace load for single contexts which can be used for example to limit different applications in the qnx slog to a given budget without affecting others or to give a file transfer unlimited bandwidth. It is recommended to always specify a budget for the application id without the contexts to ensure new contexts and internal logs like DLTL can be logged. Applications are starting up with a default limit defined via CMake variables TRACE_LOAD_USER_HARD_LIMIT, TRACE_LOAD_USER_SOFT_LIMIT. As soon as the connection to the daemon was succesull each configuration entry matching the app id will be sent to the client via an application message. If no configuration is found TRACE_LOAD_DAEMON_HARD_LIMIT and TRACE_LOAD_USER_SOFT_LIMIT will be used instead. The two staged configuration process makes sure that the daemon default can be set to 0, forcing developers to define a limit for their application while making sure that applications are able to log before they received the log levels. Measuring the trace load is done in the daemon and the client. Dropping messages on the client side is the primary mechanism and prevents sending logs to the daemon only to be dropped there, which reduces the load of the IPC. Measuring again on daemon side makes sure that rouge clients are still limited to the trace load defined. Exceeding the limit soft will produce the following logs: ECU1- DEMO DLTL log warn V 1 [Trace load exceeded trace soft limit on apid: DEMO. (soft limit: 2000 bytes/sec, current: 2414 bytes/sec)] ECU1- DEMO DLTL log warn V 1 [Trace load exceeded trace soft limit on apid: DEMO, ctid TEST.(soft limit: 150 bytes/sec, current: 191 bytes/sec)] Exceeding the hard limit will produce the same message but the text '42 messages discarded.' is appended and messages will be dropped. Dropped messages are lost and cannot be recovered, which forces developers to get their logging volume under control. As debug and trace load are usually disabled for production and thus do not impact the performance of actual systems these logs are not accounted for in trace load limits. In practice this turned out to be very usefull to improve developer experience while maintaining good performance, as devs know that debug and trace logs are only available during development and important information has to be logged on a higher level. To simplify creating a trace limit base line the script 'utils/calculate-load.py' is provided which makes suggestions for the limits based on actual log volume. Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
Configuration menu - View commit details
-
Copy full SHA for 398ac45 - Browse repository at this point
Copy the full SHA 398ac45View commit details
Commits on Aug 15, 2024
-
trace-looad: fix review findings
* Add documentation * remove broken statistics * set defaults for unit tests differently.
Configuration menu - View commit details
-
Copy full SHA for 6aaae63 - Browse repository at this point
Copy the full SHA 6aaae63View commit details
Commits on Sep 12, 2024
-
trace-load: protect settings with rwlock
Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
Configuration menu - View commit details
-
Copy full SHA for 7906ffc - Browse repository at this point
Copy the full SHA 7906ffcView commit details
Commits on Sep 19, 2024
-
review: remove redundant ifdef
Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
Configuration menu - View commit details
-
Copy full SHA for da50a13 - Browse repository at this point
Copy the full SHA da50a13View commit details -
Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
Configuration menu - View commit details
-
Copy full SHA for cf63c53 - Browse repository at this point
Copy the full SHA cf63c53View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.