Expose kprobe's maxactive parameter in bcc #2224
Conversation
pchaigno
requested review from
4ast,
brendangregg,
drzaeus77,
goldshtn and
yonghong-song
as code owners
When a kretprobe is installed on a kernel function, there is a limit on
how many parallel calls it can catch. You can change that limit with the
maxactive parameter. Since commit 696ced4 ("tracing/kprobes: expose
maxactive for kretprobe in kprobe_events") in the Linux kernel, debugfs
exposes that parameter.
This commit exposes that parameter in bcc as well, through
bpf_attach_kprobe in libbcc and BPF.attach_kretprobe at the Python level.
If a maxactive value is given, we fallback to using debugfs to install the
kprobe event, instead of the usual perf_event_open.
For kernels without maxactive supports in debugfs, the error resolution is
not straightforward. The kernel will still install a probe, but under a
different name. We therefore need to check for our expected name, delete
the probe with the different name, and try to attach the probe again,
without the maxactive value.
Signed-off-by: Paul Chaignon <paul.chaignon@orange.com>
|
[buildbot, test this please] |
1 similar comment
|
[buildbot, test this please] |
| else if (maxactive > 0 && attach_type == BPF_PROBE_RETURN) | ||
| snprintf(buf, buf_size, "r%d:kprobes/%s %s", | ||
| maxactive, ev_alias, fn_name); | ||
| else |
There was a problem hiding this comment.
Does maxactive do anything in k(entry)probe? If not, maybe we should check that the value should be 0?
There was a problem hiding this comment.
maxactive doesn't do anything for kprobes. Do you mean check that the value is 0 at the beginning of bpf_attach_kprobe?
| strerror(errno)); | ||
| return -1; | ||
| } | ||
| snprintf(fname, sizeof(fname), "-:kprobes/%s_0", ev_name); |
There was a problem hiding this comment.
Is it possible to re-use detach kprobe logic here?
There was a problem hiding this comment.
I considered it, but bpf_detach_probe also contains some code to handle the case where we attached the probe using perf_event_open (it iterates on the list of events). We could break bpf_detach_probe down to refactor, but I want to try and add support for maxactive in perf_event_open, so I thought I'd refactor after that.
There was a problem hiding this comment.
Sorry to bug a closed issue.
But where is the documentation for maxactive as referred in "See the kprobes documentation for its default value."?
We run into issue similar to #1920 where bpf_trace_printk() states that the perf_submit() succeeded, but user-space did not read the events from the perf buffer.
There was a problem hiding this comment.
The default value is discussed in the kprobes documentation in the kernel:
If maxactive <= 0, it is
set to a default value. If CONFIG_PREEMPT is enabled, the default
is max(10, 2*NR_CPUS). Otherwise, the default is NR_CPUS.
| close(kfd); | ||
|
|
||
| // Re-creating kprobe event without maxactive. | ||
| if (create_kprobe_event(buf, ev_name, attach_type, fn_name, |
There was a problem hiding this comment.
So... ideally here we should go back and try use Perf Event based approach?
There was a problem hiding this comment.
probably not needed. pref event based approach is supported in 4.17 and maxactive is supported in 4.12...
There was a problem hiding this comment.
Hm. I missed that. I should've called bpf_attach_kprobe with maxactive=0 instead. I could refactor if you think it's worth it.
Also, I might be missing something, but is there any reason the perf_event_open API is preferred to the debugfs'?
Context: iovisor/bcc#2224 Set `maxActive` to <= 0 to use the kernel default. See: https://www.kernel.org/doc/Documentation/kprobes.txt
bcc changed the signature of bpf_attach_kprobe to accept a new argument: `maxactive`, which determines the maximum limit of how many calls in parallel a kretprobe attached to a kernel function can catch. The new signature is not backwards compatible, thus our build broke. Since there's no way to feature detect the existance of an argument in a function using cmake, we use a function pointer with the long signature and cast bpf_attach_kprobe to it. If we're on an older bcc version, the last argument will be allocated in a register (according to the platform's calling convention), but should be ignored inside the function. Ref: torvalds/linux@696ced4 Ref: iovisor/bcc#2224 Fixes: bpftrace#453
When a kretprobe is installed on a kernel function, there is a limit on
how many parallel calls it can catch. You can change that limit with the
maxactive parameter. Since commit 696ced4 ("tracing/kprobes: expose
maxactive for kretprobe in kprobe_events") in the Linux kernel, debugfs
exposes that parameter.
This commit exposes that parameter in bcc as well, through
bpf_attach_kprobe in libbcc and BPF.attach_kretprobe at the Python level.
If a maxactive value is given, we fallback to using debugfs to install the
kprobe event, instead of the usual perf_event_open.
For kernels without maxactive supports in debugfs, the error resolution is
not straightforward. The kernel will still install a probe, but under a
different name. We therefore need to check for our expected name, delete
the probe with the different name, and try to attach the probe again,
without the maxactive value.
Signed-off-by: Paul Chaignon <paul.chaignon@orange.com>
When a kretprobe is installed on a kernel function, there is a limit on
how many parallel calls it can catch. You can change that limit with the
maxactive parameter. Since commit 696ced4 ("tracing/kprobes: expose
maxactive for kretprobe in kprobe_events") in the Linux kernel, debugfs
exposes that parameter.
This commit exposes that parameter in bcc as well, through
bpf_attach_kprobe in libbcc and BPF.attach_kretprobe at the Python level.
If a maxactive value is given, we fallback to using debugfs to install the
kprobe event, instead of the usual perf_event_open.
For kernels without maxactive supports in debugfs, the error resolution is
not straightforward. The kernel will still install a probe, but under a
different name. We therefore need to check for our expected name, delete
the probe with the different name, and try to attach the probe again,
without the maxactive value.
Signed-off-by: Paul Chaignon <paul.chaignon@orange.com>
When a kretprobe is installed on a kernel function, there is a limit on
how many parallel calls it can catch. You can change that limit with the
maxactive parameter. Since commit 696ced4 ("tracing/kprobes: expose
maxactive for kretprobe in kprobe_events") in the Linux kernel, debugfs
exposes that parameter.
This commit exposes that parameter in bcc as well, through
bpf_attach_kprobe in libbcc and BPF.attach_kretprobe at the Python level.
If a maxactive value is given, we fallback to using debugfs to install the
kprobe event, instead of the usual perf_event_open.
For kernels without maxactive supports in debugfs, the error resolution is
not straightforward. The kernel will still install a probe, but under a
different name. We therefore need to check for our expected name, delete
the probe with the different name, and try to attach the probe again,
without the maxactive value.
Signed-off-by: Paul Chaignon <paul.chaignon@orange.com>
When a kretprobe is installed on a kernel function, there is a limit on how many parallel calls it can catch. You can change that limit with the
maxactiveparameter. Since commit696ced4("tracing/kprobes: expose maxactive for kretprobe in kprobe_events") in the Linux kernel, debugfs exposes that parameter.This pull request exposes that parameter in bcc as well, through
bpf_attach_kprobein libbcc andBPF.attach_kretprobeat the Python level. If amaxactivevalue is given, we fallback to using debugfs to install the kprobe event, instead of the usualperf_event_open.For kernels without
maxactivesupports in debugfs, the error resolution is not straightforward. The kernel will still install a probe, but under a different name. We therefore need to check for our expected name, delete the probe with the different name, and try to attach the probe again, without themaxactivevalue.Fixes #1072 #1250
/cc @alban @anisse