cmonitor_collector developments to improve the Docker-monitoring-Dock…

…er scenario

cmonitor_collector:
* formalize list of monitored files and dump it as debug log after initialization using new helper get_list_monitored_files()
* create CMonitorSystem::sample_memory() using newer FastFileReader method instead of proc_read_numeric_stats_from()
* remove slow proc_read_numeric_stats_from() using fopen()+sscanf()
* move all checks against "m_nCollectFlags" into each sample_*() function to be coherent with new get_list_monitored_files()
* move info FastFileReader some generic parsers to read simple stats from the file
* move into "deep" collection mode several per-process KPIs to make the output JSON more compact in general case
* fix regression where the first samples would have CPU close to 100% int he "cgroup_tasks" JSON section
* share the PID reader between cgroup processes/network stats collector code
* get rid of lscpu dependency, not available in the alpine docker image and rather read more kernel files about CPU min/max frequency, NUMA topology, etc
* add docker-collecting-docker-stats example 

cmonitor_chart:
* remove use of "lscpu" section from input JSON and instead process the "cpuinfo" section with some logic to produce results similar to the ones of "lscpu"
* remove use of "cpu_tot" KPI that was removed

README.md

@@ -37,10 +37,8 @@ Table of contents of this README:
     - [Step 2: plot stats collected as JSON](#step-2-plot-stats-collected-as-json)
     - [Usage scenarios and HTML result examples](#usage-scenarios-and-html-result-examples)
       - [Monitoring the baremetal server (no containers)](#monitoring-the-baremetal-server-no-containers)
-      - [Monitoring the baremetal server from a Docker container](#monitoring-the-baremetal-server-from-a-docker-container)
-      - [Monitoring a Docker container launching cmonitor_collector on the baremetal](#monitoring-a-docker-container-launching-cmonitor_collector-on-the-baremetal)
-      - [Monitoring a Docker container that embeds cmonitor_collector](#monitoring-a-docker-container-that-embeds-cmonitor_collector)
-      - [Monitoring a Kubernetes POD launching cmonitor_collector on the worker node](#monitoring-a-kubernetes-pod-launching-cmonitor_collector-on-the-worker-node)
+      - [Monitoring your Docker container](#monitoring-your-docker-container)
+      - [Monitoring your Kubernetes POD](#monitoring-your-kubernetes-pod)
     - [Connecting with InfluxDB and Grafana](#connecting-with-influxdb-and-grafana)
     - [Reference Manual](#reference-manual)
   - [Project History](#project-history)
@@ -147,6 +145,7 @@ yum install -y cmonitor-collector cmonitor-tools
 Note that the RPM `cmonitor-collector` has no dependencies from Python and has very small set of dependencies (GNU libc and few others)
 so can be installed easily everywhere. The RPM `cmonitor-tools` instead requires Python3.
 
+
 ### Debian package (for Debian, Ubuntu, etc)
 
 You can get started with cmonitor by installing it as a Debian package.
@@ -168,14 +167,29 @@ release in my PPA.
 
 ### Docker
 
-If you want to simply use a out-of-the-box Docker container to monitor your baremetal performances you can run:
+Alternatively to native packages, you can use `cmonitor_collector` utility as a Docker:
 
 ```
-docker pull f18m/cmonitor
+docker run -d \
+    --rm \
+    --name=cmonitor-baremetal-collector \
+    --network=host \
+    --pid=host \
+    --volume=/sys:/sys:ro \
+    --volume=/etc/os-release:/etc/os-release:ro \
+    --volume=$(pwd):/perf:rw \
+    f18m/cmonitor:latest \
+    --sampling-interval=1  ...
 ```
 
-which downloads the Docker image for this project from [Docker Hub](https://hub.docker.com/r/f18m/cmonitor).
-See below for examples on how to run the Docker image.
+which runs the Docker image for this project from [Docker Hub](https://hub.docker.com/r/f18m/cmonitor).
+Note that the Dockerfile entrypoint is `cmonitor_collector` and thus any [supported CLI option](#reference-manual) can be provided
+at the end of the `docker run` command.
+The volume mount of `/etc/os-release` and the `--network=host` option are required to allow `cmonitor_collector` to correctly identify the real host
+being monitored (otherwise `cmonitor_collector` reports the randomly-generated hash by Docker for the hostname).
+The `--pid=host` option is required to allow `cmonitor_collector` to monitor processes generated by other containers.
+Finally, the volume mount of `/sys` exposes all the cgroups of baremetal into the `cmonitor_collector` docker and thus enables the collector
+utility to access the stats of all other running containers; this is required by similar tools as well like [cAdvisor](https://github.com/google/cadvisor).
 
 
 ## How to build from sources
@@ -210,21 +224,40 @@ sudo make install DESTDIR=/usr/local BINDIR=bin   # to install in /usr/local/bin
 
 ### Step 1: collect stats
 
-The RPM installs a single utility, `cmonitor_collector` inside your container; launch it like that:
+The RPM/Debian packages "cmonitor-collector" install a single utility, named `cmonitor_collector`.
+It can be launched as simply as:
 
 ```
 cmonitor_collector --sampling-interval=3 --output-directory=/home
 ```
 
+(on baremetal) or as a Docker:
+
+```
+docker run -d \
+    --rm \
+    --name=cmonitor-baremetal-collector \
+    --network=host \
+    --pid=host \
+    --volume=/sys:/sys:ro \
+    --volume=/etc/os-release:/etc/os-release:ro \
+    --volume=/home:/perf:rw \
+    f18m/cmonitor:latest \
+    --sampling-interval=1  ...
+```
+
 to produce in the `/home` folder a JSON with CPU/memory/disk/network stats for the container
 sampling all supported performance statistics every 3 seconds.
-Whenever you want you can either:
+
+Once the JSON is produced, next steps are either:
 
 - inject that JSON inside InfluxDB (mostly useful for **persistent** containers that you want to monitor in real-time);
   see section "Connecting with InfluxDB and Grafana" below;
 - or use the `cmonitor_chart` utility to convert that JSON into a self-contained HTML file (mostly useful for **ephemeral** containers);
   see below for practical examples.
 
+See [supported CLI option](#reference-manual) section for complete list of accepted options.
+
 
 
 ### Step 2: plot stats collected as JSON
@@ -245,43 +278,34 @@ Note that to save space/bandwidth you can also gzip the JSON file and pass it gz
 
 #### Monitoring the baremetal server (no containers)
 
-In this case you can simply install cmonitor as RPM or APT package following instructions in [How to install](#section-id-65)
-and then launch the cmonitor collector as any other Linux daemon.
+If you want to monitor the performances of an entire server (or worker node in Kubernetes terminology), you can either:
+a) install `cmonitor_collector` as RPM or APT package following instructions or
+b) use `cmonitor_collector` as a Docker;
+See [How to install](#how-to-install) for more information.
+
 Example results:
 
 1) [baremetal1](https://f18m.github.io/cmonitor/examples/baremetal1.html): 
-   example of graph generated with the performance stats collected from a physical (baremetal) server; 
-   note that despite the absence of any kind of container, the `cmonitor_collector` utility (like just any other software in modern Linux distributions) was running inside the default "user.slice" cgroup and collected both the stats of that cgroup and all baremetal stats (which in this case mostly coincide since the "user.slice" cgroup contains almost all running processes of the server);
+   example of graph generated with the performance stats collected from a physical (baremetal) server by running `cmonitor_collector` installed as RPM; 
+   note that despite the absence of any kind of container, the `cmonitor_collector` utility (like just any other software in 
+   modern Linux distributions) was running inside the default "user.slice" cgroup and collected both the stats of that cgroup 
+   and all baremetal stats (which in this case mostly coincide since the "user.slice" cgroup contains almost all running processes of the server);
 
 2) [baremetal2](https://f18m.github.io/cmonitor/examples/baremetal2.html): 
-   This is a longer example of collected statistics (results in a larger file, may take some time to download) generated with 9 hours of performance stats collected from a physical server running Centos7 and with 56 CPUs (!!); 
-   the `cmonitor_collector` utility was running inside the default "user.slice" cgroup so both "CGroup" and "Baremetal"
+   This is a longer example of collected statistics (results in a larger file, may take some time to download) generated
+   with 9 hours of performance stats collected from a physical server running Centos7 and with 56 CPUs (!!); 
+   the `cmonitor_collector` utility was installed as RPM and running inside the default "user.slice" cgroup so both "CGroup" and "Baremetal"
    graphs are present;
 
+3) [docker_collecting_baremetal_stats](https://f18m.github.io/cmonitor/examples/docker-collecting-baremetal-stats.html): 
+   example of graph generated with the performance stats collected from a physical server from `cmonitor_collector` Docker container;
+   in this case cgroup stat collection was explicitely disabled so that only baremetal performance graphs are present;
+   see [Docker installation](#docker) information as reference how the docker was started.
 
-#### Monitoring the baremetal server from a Docker container
 
-In this case you can install cmonitor Docker using the official DockerHub image, see [Docker](#section-id-88); the Docker container
-will collect all performance stats of the baremetal. Just run it
+#### Monitoring your Docker container
 
-```
-docker run -d \
-    --name=cmonitor-baremetal-collector
-    -v /root:/perf \
-    f18m/cmonitor
-```
-
-Example results:
-
-1) [docker_collecting_baremetal_stats](https://f18m.github.io/cmonitor/examples/docker-collecting-baremetal-stats.html): 
-   example of graph generated with the performance stats collected from a physical server from inside a Docker container;
-   in this case cgroup stat collection was explicitely disabled so that only baremetal performance graphs are present;
-
-      
-
-#### Monitoring a Docker container launching cmonitor_collector on the baremetal
-
-In this case you can simply install cmonitor as RPM or APT package following instructions in [How to install](#section-id-65)
+In this case you can simply install cmonitor as RPM or APT package following instructions in [How to install](#how-to-install)
 and then launch the `cmonitor_collector` utility as any other Linux daemon, specifying the name of the cgroup associated with
 the docker container to monitor.
 Finding out the cgroup associated with a Docker container can require some detailed information about your OS / runtime configuration;
@@ -304,48 +328,59 @@ CGROUP_NAME=system.slice/docker-${DOCKER_ID}.scope    # when 'systemd' driver is
 cmonitor_collector \
    --num-samples=until-cgroup-alive \
    --cgroup-name=${CGROUP_NAME} \
-   --collect=cgroup_threads,cgroup_cpu,cgroup_memory --score-threshold=0 \
+   --collect=cgroup_threads,cgroup_cpu,cgroup_memory,cgroup_network --score-threshold=0 \
+   --custom-metadata=cmonitor_chart_name:userapp \
+   --sampling-interval=3 \
+   --output-filename=docker-userapp.json
+```
+
+Alternatively the Redis Docker container (or any other one) can be monitored from `cmonitor_collector` running as a Docker itself:
+
+```
+docker run -d \
+    --rm \
+    --name=cmonitor-collector \
+    --network=host \
+    --pid=host \
+    --volume=/sys:/sys:ro \
+    --volume=/etc/os-release:/etc/os-release:ro \
+    --volume=/home:/perf:rw \
+    f18m/cmonitor:latest \
+   --num-samples=until-cgroup-alive \
+   --cgroup-name=${CGROUP_NAME} \
+   --collect=cgroup_threads,cgroup_cpu,cgroup_memory,cgroup_network --score-threshold=0 \
    --custom-metadata=cmonitor_chart_name:userapp \
    --sampling-interval=3 \
-   --output-filename docker-userapp.json
+   --output-filename=docker-userapp.json
 ```
 
+See [Docker usage](#docker) paragraph for more info about the "docker run" options required.
+See example #4 below to view the results produced by using the `cmonitor_collector` Docker with the command above.
+
 Example results:
 
-1)[docker_userapp](https://f18m.github.io/cmonitor/examples/docker-userapp.html): example of the chart generated by monitoring
+1) [docker_userapp](https://f18m.github.io/cmonitor/examples/docker-userapp.html): example of the chart generated by monitoring
    from the baremetal a simple Redis docker simulating a simple application, doing some CPU and I/O. In this example the 
    `--collect=cgroup_threads` is further used to show Redis CPU usage by-thread.
-
-
-#### Monitoring a Docker container that embeds cmonitor_collector
 
-If you can easily modify the Dockerfile of your container, you can embed cmonitor so that it runs inside your container and
-monitor your dockerized-application.
-Example of the *Dockerfile* modified for this purpose:
+2) [docker_stress_test_cpu](https://f18m.github.io/cmonitor/examples/docker-stress-test-cpu.html): example of the chart generated
+   by monitoring from the baremetal a [stress-ng](https://hub.docker.com/r/alexeiled/stress-ng/) Docker with a CPU limit set. 
+   This example shows how cmonitor will report "CPU throttling" and how useful it is to detect cases where a Docker is trying to
+   use too much CPU.
 
-```
-...
-COPY cmonitor_collector /usr/bin/   # first you need to grabthe cmonitor binary for your Docker base image
-CMD /usr/bin/cmonitor_collector \
-      --sampling-interval=3 \
-      --output-filename=mycontainer.json \
-      --output-directory /perf ; \
-    myapplication
-...
-```
-
-Example results:
+3) [docker_stress_test_memory](https://f18m.github.io/cmonitor/examples/docker-stress-test-mem.html): example of the chart generated
+   by monitoring from the baremetal a [stress-ng](https://hub.docker.com/r/alexeiled/stress-ng/) Docker with a MEMORY limit set. 
+   This example shows how cmonitor will report "Memory allocation failures" and how useful it is to detect cases where a Docker is trying to
+   use too much memory.
 
-1) [docker_userapp_with_embedded_collector](https://f18m.github.io/cmonitor/examples/docker-userapp-with-embedded-collector.html): 
-   example of graph generated with the performance stats collected from inside a Docker container using Ubuntu as base image; this is a practical example
-   where the Docker container is actually deploying something that simulates your target application, together with an embedded
-   `cmonitor_collector` instance that monitors the performance of the Docker container itself;
-   in this case both cgroup stats and baremetal performance graphs are present.
+4) [docker_collecting_docker_stats](https://f18m.github.io/cmonitor/examples/docker-collecting-docker-stats.html): example of the chart generated
+   by monitoring from  `cmonitor_collector` Docker a Redis Docker.
+   This example shows how cmonitor is able to run as a Docker-monitoring-other-Dockers.
 
 
-#### Monitoring a Kubernetes POD launching cmonitor_collector on the worker node
+#### Monitoring your Kubernetes POD
 
-In this case you can simply install cmonitor as RPM or APT package following instructions in [How to install](#section-id-65)
+In this case you can simply install cmonitor as RPM or APT package following instructions in [How to install](#how-to-install)
 on all the worker nodes where Kubernetes might be scheduling the POD you want to monitor.
 Then, you can launch the `cmonitor_collector` utility as any other Linux daemon, specifying the name of the cgroup associated
 with the Kubernetes POD (or more precisely, associated with one of the containers inside the Kubernetes POD, in case it contains
@@ -365,7 +400,7 @@ cmonitor_collector \
    --collect=cgroup_threads,cgroup_cpu,cgroup_memory --score-threshold=0 \
    --custom-metadata=cmonitor_chart_name:${PODNAME} \
    --sampling-interval=3 \
-   --output-filename pod-performances.json
+   --output-filename=pod-performances.json
 ```
 
 

collector/Makefile

@@ -108,26 +108,35 @@ endif
 
 cmonitor_musl: # build cmonitor inside a Docker from Alpine distro, to build a cmonitor_collector musl-linked
 	@# use a Docker to compile inside an Alpine environment:
-	@# NOTE: by default debug symbols are embededed in the binary (to support COPR builds) but
+	docker build \
+		--tag f18m/cmonitor_builder:latest \
+		-f docker/Dockerfile.builder \
+		docker/
+	@# NOTE: by default debug symbols are embedded in the binary (to support COPR builds) but
 	#        for docker builds, we strip them away to reduce docker size:
-	docker run --rm -it -v "${ROOT_DIR}":"/opt/src" -e "MUSL_BUILD=1" -e "DISABLE_BENCHMARKS_BUILD=1" \
-		radupopescu/musl-builder \
-		sh -c "cd /opt/src/collector && apk add gtest-dev fmt-dev && gcc --version && make clean && make -j && make strip"
+	docker run \
+		--rm -it \
+		-v "${ROOT_DIR}":"/opt/src" \
+		-e "MUSL_BUILD=1" -e "DISABLE_BENCHMARKS_BUILD=1" \
+		f18m/cmonitor_builder:latest \
+		sh -c "cd /opt/src/collector && gcc --version && make clean && make -j && make strip"
 
 docker_image: cmonitor_musl
 	@cp -fv bin/musl/cmonitor_collector docker
 	docker build \
 		--tag f18m/cmonitor:$(DOCKER_TAG) \
 		--build-arg sampling_interval=3 \
 		--build-arg num_samples=0 \
+		-f docker/Dockerfile \
 		docker
 
 docker_run:
 	@docker rm cmonitor-baremetal-collector >/dev/null 2>&1 || true
 	@docker run -it \
 		--rm \
 		--name=cmonitor-baremetal-collector \
-		-v /root:/perf \
+		--volume /etc/os-release:/etc/os-release \
+		--volume /root:/perf \
 		f18m/cmonitor:$(DOCKER_TAG)
 
 docker_push:

collector/docker/Dockerfile

@@ -1,22 +1,13 @@
-FROM alpine:3.14
+# NOTE: the version of Alpine used here must be aligned with the one in Dcokerfile.builder
+FROM alpine:3.15
 
 # make sure you did run the "cmonitor_musl" target before building this image:
-RUN apk add libstdc++ libc6-compat
+RUN apk add libstdc++ libc6-compat fmt-dev
 COPY cmonitor_collector /usr/bin/
 
-ARG SAMPLING_INTERVAL_SEC=60
-ENV SAMPLING_INTERVAL_SEC=${SAMPLING_INTERVAL_SEC}
-
 # finally run the cmonitor collector 
 #  - in foreground since Docker does not like daemons
-#  - collect "all_baremetal": in this way we just collect ONLY baremetal performance stats, not the cgroup where cmonitor_collector itself is running!
-#  - by default collect every 1min
-#  - put resulting files in /perf folder which is actually a volume shared with the host (see docker run command)
-CMD /usr/bin/cmonitor_collector \
-    --foreground \
-    --sampling-interval=$SAMPLING_INTERVAL_SEC \
-    --num-samples=0 \
-    --collect=all_baremetal \
-    --output-directory /perf
+#  - put resulting files in /perf folder which is actually a volume shared with the host (see README.md for the docker run command)
+ENTRYPOINT ["/usr/bin/cmonitor_collector", "--foreground", "--output-directory", "/perf"]
 
 LABEL GIT_REPO_URL="https://github.com/f18m/cmonitor"

collector/docker/Dockerfile.builder

@@ -0,0 +1,4 @@
+# This is just a small docker used to BUILD the cmonitor_collector utility
+
+FROM alpine:3.15
+RUN apk update && apk add --no-cache binutils make libgcc musl-dev gcc g++ fmt-dev gtest-dev

collector/src/cgroups.h

@@ -67,8 +67,6 @@ typedef struct {
 
 typedef std::map<std::string /* controller type */, std::string /* path */> cgroup_paths_map_t;
 
-typedef std::map<std::string /* KPI name */, uint64_t /* value */> key_value_map_t;
-
 typedef struct {
     uint64_t v1_failcnt;
     key_value_map_t v2_events;
@@ -100,6 +98,7 @@ class CMonitorCgroups : public CMonitorAppHelper {
         const std::string& cgroup_prefix_for_test = "", // force newline
         const std::string& proc_prefix_for_test = "", // force newline
         uint64_t my_own_pid_for_test = UINT64_MAX);
+    void get_list_monitored_files(std::set<std::string>& list);
 
     // one-shot configuration info
     void output_config();
@@ -108,6 +107,8 @@ class CMonitorCgroups : public CMonitorAppHelper {
     void sample_cpuacct(double elapsed_sec);
     void sample_memory(
         const std::set<std::string>& allowedStatsNames_v1, const std::set<std::string>& allowedStatsNames_v2);
+
+    void sample_process_list(); // call before sample_network_interfaces() and sample_processes()
     void sample_network_interfaces(double elapsed_sec, OutputFields output_opts);
     void sample_processes(double elapsed_sec, OutputFields output_opts);
 
@@ -213,17 +214,21 @@ class CMonitorCgroups : public CMonitorAppHelper {
     FastFileReader m_cgroup_memory_v2_events;
     memory_events_t m_memory_prev_values;
 
+    //------------------------------------------------------------------------------
+    // shared variables between cgroup network/process tracker
+    //------------------------------------------------------------------------------
+    FastFileReader m_cgroup_processes_reader_pids;
+    std::vector<pid_t> m_cgroup_all_pids; // this is the continuosly-updated list of PIDs/TIDs inside cgroup
+
     //------------------------------------------------------------------------------
     // cgroup network
     //------------------------------------------------------------------------------
-    FastFileReader m_cgroup_network_reader_pids;
     // previous values for network interfaces inside cgroup
     netinfo_map_t m_previous_netinfo;
 
     //------------------------------------------------------------------------------
     // cgroup processes tracking
     //------------------------------------------------------------------------------
-    FastFileReader m_cgroup_processes_reader_pids;
     bool m_cgroup_processes_include_threads = false;
     std::map<pid_t, procsinfo_t> m_pid_databases[2];
     unsigned int m_pid_database_current_index = 0; // will be alternatively 0 and 1