errors: remove verifier truncation heuristic, rely on ENOSPC

[ti-mo]

In order to capture full verifier errors of large programs, the caller can
choose to repeatedly retry loading the program with progressively larger log
buffers, since the important info is always at the end of the log.

In doing so, it's incredibly likely for the last character of the buffer to
be a newline, resulting in the truncated flag being set to false when it should
instead be set to true, regardless of the kernel's ENOSPC return code.

This commit removes the newline heuristic that currently aims to detect log
truncation. Instead, we now require ENOSPC to be sent by the kernel in order
for the Truncated flag to be set in the VerifierError.

The heuristic was initially introduced for parsing BTF verifier errors,
since there's a latent kernel bug that causes ENOSPC only being returned
on a successful BTF load. Unfortunately, it turns out that not all BTF
verifier errors end in a newline either (see calls to btf_verifier_log()),
making the newline heuristic ineffective for BTF logs as well.

ErrorWithLog now takes an ancillary error from a second load operation that
is used to determine truncation status. This second error will not be wrapped
into the VerifierError.

[lmb]

There is no indication that kernels 5.18 and up don't return ENOSPC for BTF
loads.

Have you actually managed to get ENOSPC from loading BTF? In my experience an EINVAL or similar will override ENOSPC when loading invalid BTF. Maybe you're talking about ENOSPC from valid BTF with a small log?

Agreed re bringing the check for ENOSPC back, but I think we might still need the heuristic for BTF. We could fall back if the logErr is nil, or we could factor it out into a separate function and call that from the BTF loading code.

[ti-mo]

Have you actually managed to get ENOSPC from loading BTF? In my experience an EINVAL or similar will override ENOSPC when loading invalid BTF. Maybe you're talking about ENOSPC from valid BTF with a small log?

I see, I misinterpreted your syscall will never return ENOSPC as of 5.18-rc4 comment to be 'as of 5.18-rc4...', implying something changed in 5.18. This should probably be ENOSPC until at least 6.0, as this is still the case on master. In btf.c, at the end of btf_parse(), ENOSPC is only returned if the log space check is hit, which requires a successful BTF parse.

I'll think of something else.

[ti-mo]

We could fall back if the logErr is nil, or we could factor it out into a separate function and call that from the BTF loading code.

This feels clearer to me. BPF and BTF verifier logs have different styles and their APIs have different semantics.

[lmb]

In order to capture full verifier errors of large programs, the caller can
choose to repeatedly retry loading the program with progressively larger log
buffers, since the important info is always at the end of the log.

I think this is the first time I understand what you are trying to achieve, and why you need .Truncated to be accurate. I didn't consider that use case when introducing .Truncated, so I didn't mind that the heuristic isn't bomb proof.

You're trying to make .Truncated "reliable". Have you considered implementing resize-and-retry in the lib instead? Seems like a useful feature to have.