Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

trace-load-limits: restrict trace load for applications #671

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Commits on Aug 8, 2024

  1. trace-load-limits: restrict trace load for applications

    In a shared system many developers log as much as they
    want into DLT. This can lead into overloading the logging
    system resulting in bad performance and dropped logs.
    This commit introduces trace load limits to restrict
    applications to a certain log volume measured in bytes/s.
    It is based on COVESA#134 but extends this heavily.
    
    Trace load limits are configured via a space separted
    configuraiton file.
    The format of the file follows:
    
    APPID [CTXID] SOFT_LIMIT HARD_LIMIT
    The most matching entry will be selected for each log, meaning that
    either app and context must match or at least the app id, for which a
    default is created when not configured.
    This allows to configure trace load for single contexts which can be
    used for example to limit different applications in the qnx slog to a
    given budget without affecting others or to give a file transfer
    unlimited bandwidth.
    It is recommended to always specify a budget for the application id
    without the contexts to ensure new contexts and internal logs like DLTL
    can be logged.
    
    Applications are starting up with a default limit defined
    via CMake variables TRACE_LOAD_USER_HARD_LIMIT, TRACE_LOAD_USER_SOFT_LIMIT.
    As soon as the connection to the daemon was succesull each configuration
    entry matching the app id will be sent to the client via an
    application message.
    If no configuration is found TRACE_LOAD_DAEMON_HARD_LIMIT and
    TRACE_LOAD_USER_SOFT_LIMIT will be used instead.
    The two staged configuration process makes sure that the daemon default
    can be set to 0, forcing developers to define a limit for their
    application while making sure that applications are able to log before
    they received the log levels.
    
    Measuring the trace load is done in the daemon and the client.
    Dropping messages on the client side is the primary mechanism and
    prevents sending logs to the daemon only to be dropped there, which
    reduces the load of the IPC. Measuring again on daemon side makes
    sure that rouge clients are still limited to the trace load defined.
    
    Exceeding the limit soft will produce the following logs:
    ECU1- DEMO DLTL log warn V 1 [Trace load exceeded trace soft limit on apid: DEMO. (soft limit: 2000 bytes/sec, current: 2414 bytes/sec)]
    ECU1- DEMO DLTL log warn V 1 [Trace load exceeded trace soft limit on apid: DEMO, ctid TEST.(soft limit: 150 bytes/sec, current: 191 bytes/sec)]
    
    Exceeding the hard limit will produce the same message but the text
    '42 messages discarded.' is appended and messages will be dropped.
    Dropped messages are lost and cannot be recovered, which forces
    developers to get their logging volume under control.
    
    As debug and trace load are usually disabled for production and thus do
    not impact the performance of actual systems these logs are not
    accounted for in trace load limits. In practice this turned out to be
    very usefull to improve developer experience while maintaining good
    performance, as devs know that debug and trace logs are only available
    during development and important information has to be logged on a
    higher level.
    
    To simplify creating a trace limit base line the script
    'utils/calculate-load.py' is provided which makes suggestions
    for the limits based on actual log volume.
    
    Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
    alexmohr committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    398ac45 View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. trace-looad: fix review findings

    * Add documentation
    * remove broken statistics
    * set defaults for unit tests differently.
    alexmohr committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    6aaae63 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. trace-load: protect settings with rwlock

    Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
    alexmohr committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    7906ffc View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. review: remove redundant ifdef

    Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
    alexmohr committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    da50a13 View commit details
    Browse the repository at this point in the history
  2. review: fix malloc findings

    Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
    alexmohr committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    cf63c53 View commit details
    Browse the repository at this point in the history