Got different metric scopes when access /metrics starting from v0.26.1

[maptile]

My computer running Ubuntu 17.10, and only 1 container is running (PostgresSQL).

Then I pull and run cAdvisor image from docker using the following command.

docker run \
    --volume=/:/rootfs:ro \
    --volume=/var/run:/var/run:rw \
    --volume=/sys:/sys:ro \
    --volume=/var/lib/docker/:/var/lib/docker:ro \
    --volume=/dev/disk:/dev/disk:ro \
    --publish=8080:8080 \
    --restart=always \
    --privileged=true \
    --detach=true \
    google/cadvisor:v0.27.2

After that, I have 2 running containers, one is PostgreSQL and one is cAdvisor

Then I navigate lo http://localhost:8080/metrics in browser.

I got the following data, you can see there is no container info.

container_cpu_load_average_10s{id="/"} 0
container_cpu_load_average_10s{id="/docker"} 0
container_cpu_load_average_10s{id="/init.scope"} 0
container_cpu_load_average_10s{id="/system.slice"} 0
container_cpu_load_average_10s{id="/system.slice/ModemManager.service"} 0
container_cpu_load_average_10s{id="/system.slice/NetworkManager.service"} 0
container_cpu_load_average_10s{id="/system.slice/accounts-daemon.service"} 0
container_cpu_load_average_10s{id="/system.slice/acpid.service"} 0
container_cpu_load_average_10s{id="/system.slice/alsa-restore.service"} 0
container_cpu_load_average_10s{id="/system.slice/apport.service"} 0
container_cpu_load_average_10s{id="/system.slice/atop.service"} 0
container_cpu_load_average_10s{id="/system.slice/atopacct.service"} 0
container_cpu_load_average_10s{id="/system.slice/avahi-daemon.service"} 0
container_cpu_load_average_10s{id="/system.slice/binfmt-support.service"} 0
container_cpu_load_average_10s{id="/system.slice/clamav-freshclam.service"} 0
container_cpu_load_average_10s{id="/system.slice/colord.service"} 0
container_cpu_load_average_10s{id="/system.slice/console-setup.service"} 0
container_cpu_load_average_10s{id="/system.slice/cron.service"} 0
container_cpu_load_average_10s{id="/system.slice/cups.service"} 0
container_cpu_load_average_10s{id="/system.slice/dbus.service"} 0
container_cpu_load_average_10s{id="/system.slice/dev-disk-by\\x2did-ata\\x2dSanDisk_SD7SB6S256G1122_154001801699\\x2dpart3.swap"} 0
container_cpu_load_average_10s{id="/system.slice/dev-disk-by\\x2did-wwn\\x2d0x5001b44ee6cdfde3\\x2dpart3.swap"} 0
container_cpu_load_average_10s{id="/system.slice/dev-disk-by\\x2dpartuuid-da11ff03\\x2df3ef\\x2d4285\\x2dbba7\\x2d15ae6becd2f6.swap"} 0
container_cpu_load_average_10s{id="/system.slice/dev-disk-by\\x2dpath-pci\\x2d0000:00:1f.2\\x2data\\x2d1\\x2dpart3.swap"} 0
container_cpu_load_average_10s{id="/system.slice/dev-disk-by\\x2duuid-9193ee16\\x2d5f82\\x2d49c7\\x2d90e2\\x2d8e2226c0f6e1.swap"} 0
container_cpu_load_average_10s{id="/system.slice/dev-sda3.swap"} 0
container_cpu_load_average_10s{id="/system.slice/dnsmasq.service"} 0
container_cpu_load_average_10s{id="/system.slice/docker.service"} 0
container_cpu_load_average_10s{id="/system.slice/fwupd.service"} 0
container_cpu_load_average_10s{id="/system.slice/gdm.service"} 0
container_cpu_load_average_10s{id="/system.slice/grub-common.service"} 0
container_cpu_load_average_10s{id="/system.slice/irqbalance.service"} 0
container_cpu_load_average_10s{id="/system.slice/kerneloops.service"} 0
container_cpu_load_average_10s{id="/system.slice/keyboard-setup.service"} 0
container_cpu_load_average_10s{id="/system.slice/kmod-static-nodes.service"} 0
container_cpu_load_average_10s{id="/system.slice/lm-sensors.service"} 0
container_cpu_load_average_10s{id="/system.slice/openvpn.service"} 0
container_cpu_load_average_10s{id="/system.slice/packagekit.service"} 0
container_cpu_load_average_10s{id="/system.slice/polipo.service"} 0
container_cpu_load_average_10s{id="/system.slice/polkit.service"} 0
container_cpu_load_average_10s{id="/system.slice/resolvconf.service"} 0
container_cpu_load_average_10s{id="/system.slice/rsyslog.service"} 0
container_cpu_load_average_10s{id="/system.slice/rtkit-daemon.service"} 0
container_cpu_load_average_10s{id="/system.slice/setvtrgb.service"} 0
container_cpu_load_average_10s{id="/system.slice/shadowsocks.service"} 0
container_cpu_load_average_10s{id="/system.slice/snapd.service"} 0
container_cpu_load_average_10s{id="/system.slice/speech-dispatcher.service"} 0
container_cpu_load_average_10s{id="/system.slice/ssh.service"} 0
container_cpu_load_average_10s{id="/system.slice/strongswan.service"} 0
container_cpu_load_average_10s{id="/system.slice/system-getty.slice"} 0
container_cpu_load_average_10s{id="/system.slice/system-systemd\\x2dbacklight.slice"} 0
container_cpu_load_average_10s{id="/system.slice/system-systemd\\x2dfsck.slice"} 0
container_cpu_load_average_10s{id="/system.slice/systemd-journal-flush.service"} 0
container_cpu_load_average_10s{id="/system.slice/systemd-journald.service"} 0
container_cpu_load_average_10s{id="/system.slice/systemd-logind.service"} 0
container_cpu_load_average_10s{id="/system.slice/systemd-modules-load.service"} 0
container_cpu_load_average_10s{id="/system.slice/systemd-random-seed.service"} 0
container_cpu_load_average_10s{id="/system.slice/systemd-remount-fs.service"} 0
container_cpu_load_average_10s{id="/system.slice/systemd-resolved.service"} 0
container_cpu_load_average_10s{id="/system.slice/systemd-sysctl.service"} 0
container_cpu_load_average_10s{id="/system.slice/systemd-timesyncd.service"} 0
container_cpu_load_average_10s{id="/system.slice/systemd-tmpfiles-setup-dev.service"} 0
container_cpu_load_average_10s{id="/system.slice/systemd-tmpfiles-setup.service"} 0
container_cpu_load_average_10s{id="/system.slice/systemd-udev-trigger.service"} 0
container_cpu_load_average_10s{id="/system.slice/systemd-udevd.service"} 0
container_cpu_load_average_10s{id="/system.slice/systemd-update-utmp.service"} 0
container_cpu_load_average_10s{id="/system.slice/systemd-user-sessions.service"} 0
container_cpu_load_average_10s{id="/system.slice/thermald.service"} 0
container_cpu_load_average_10s{id="/system.slice/tinc.service"} 0
container_cpu_load_average_10s{id="/system.slice/udisks2.service"} 0
container_cpu_load_average_10s{id="/system.slice/ufw.service"} 0
container_cpu_load_average_10s{id="/system.slice/unattended-upgrades.service"} 0
container_cpu_load_average_10s{id="/system.slice/upower.service"} 0
container_cpu_load_average_10s{id="/system.slice/vboxautostart-service.service"} 0
container_cpu_load_average_10s{id="/system.slice/vboxballoonctrl-service.service"} 0
container_cpu_load_average_10s{id="/system.slice/vboxdrv.service"} 0
container_cpu_load_average_10s{id="/system.slice/vboxweb-service.service"} 0
container_cpu_load_average_10s{id="/system.slice/vmware-USBArbitrator.service"} 0
container_cpu_load_average_10s{id="/system.slice/vmware.service"} 0
container_cpu_load_average_10s{id="/system.slice/vpnagentd.service"} 0
container_cpu_load_average_10s{id="/system.slice/whoopsie.service"} 0
container_cpu_load_average_10s{id="/system.slice/wpa_supplicant.service"} 0
container_cpu_load_average_10s{id="/user.slice"} 0
container_cpu_system_seconds_total{id="/"} 624.76
container_cpu_system_seconds_total{id="/docker"} 33.17
container_cpu_system_seconds_total{id="/init.scope"} 2.33
...

Then I refresh the page about 35 times, I got the following data. You can see only container info.

container_cpu_load_average_10s{id="/docker/96379a7b317847228ac42c85a6d542520869afb32e9c6b77270468846036ee2d",image="google/cadvisor:v0.27.2",name="thirsty_bhabha"} 0
container_cpu_load_average_10s{id="/docker/cb1e58b91e919e7640ac10ea32e1df59496d17f615c4f629ece755c729060647",image="postgres:9.4",name="db"} 0
container_cpu_system_seconds_total{id="/docker/96379a7b317847228ac42c85a6d542520869afb32e9c6b77270468846036ee2d",image="google/cadvisor:v0.27.2",name="thirsty_bhabha"} 0.87
container_cpu_system_seconds_total{id="/docker/cb1e58b91e919e7640ac10ea32e1df59496d17f615c4f629ece755c729060647",image="postgres:9.4",name="db"} 0.74
...

It behaves strange, and I do the same test from 0.26.1 to 0.27.2, all versions have this problem. Only version <= v0.25.0 behave normally.

This file is the first time I get data
0.27.2.first.time.txt

This file is after refresh the page 35 times
0.27.2.after.refresh.txt

[dashpole]

This is likely #1704.

[maptile]

Yes, and sorry for the duplicate issue. Should I close it?

I've searched before open this issue, but nothing related was found. May be the keyword I'm using is wrong :)

[dashpole]

No problem.

[dashpole]

The fix is released in version v0.28.3