Leak detection doesn't work

[mikea]

Leak detection is disabled because it always gives a false positive.

To reproduce:

docker run -ti -e ASAN_OPTIONS=detect_leaks=1 ossfuzz/expat run expat_parse_fuzzer -runs=100

This will generate lots of leaks like:

Indirect leak of 1 byte(s) in 1 object(s) allocated from:
    #0 0x50d000 in operator new(unsigned long) /src/llvm/projects/compiler-rt/lib/asan/asan_new_delete.cc:82
    #1 0x53fd89 in std::__1::__allocate(unsigned long) /usr/local/bin/../include/c++/v1/new:171:10
    #2 0x53fd89 in std::__1::allocator<unsigned char>::allocate(unsigned long, void const*) /usr/local/bin/../include/c++/v1/memory:1771
    #3 0x53fd89 in std::__1::allocator_traits<std::__1::allocator<unsigned char> >::allocate(std::__1::allocator<unsigned char>&, unsigned long) /usr/local/bin/../include/c++/v1/memory:1526
    #4 0x53fd89 in std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >::allocate(unsigned long) /usr/local/bin/../include/c++/v1/vector:923
    #5 0x53f2a4 in _ZNSt3__16vectorIhNS_9allocatorIhEEE6assignIPhEENS_9enable_ifIXaasr21__is_forward_iteratorIT_EE5valuesr16is_constructibleIhNS_15iterator_traitsIS7_E9referenceEEE5valueEvE4typeES7_S7_ /usr/local/bin/../include/c++/v1/vector:1403:9
    #6 0x52d096 in std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >::operator=(std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > const&) /usr/local/bin/../include/c++/v1/vector:1348:9
    #7 0x52d096 in fuzzer::InputCorpus::AddToCorpus(std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > const&, unsigned long, bool) /src/libfuzzer/FuzzerCorpus.h:71
    #8 0x583213 in fuzzer::Fuzzer::ShuffleAndMinimize(std::__1::vector<std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >, std::__1::allocator<std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > >*) /src/libfuzzer/FuzzerLoop.cpp:427:14
    #9 0x52059f in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:519:5
    #10 0x5aac38 in main /src/libfuzzer/FuzzerMain.cpp:20:10
    #11 0x7f9b61f2982f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

[kcc]

Solution: add --cap-add SYS_PTRACE to docker run command line.
lsan needs ptrace and by default docker denies it.

[mikea]

I have verified that ClusterFuzz has detect_leaks=1 (@oliverchang). We'll keep detect_leaks=0 meanwhile in docker images.

lsan issue also filed: google/sanitizers#728

[kcc]

Could you please double-check again?
We haven't reported a single leak yet. This is very, very strange.

[inferno-chromium]

It is explicitly disabled in all job types. Oliver might know why. We can turn it on easily, we should just make sure it does not cause startup crashes, might cause some noise with fixed testcases. But CF should blacklist them automatically over time.

[oliverchang]

I just enabled this after doing some quick tests. Let's see if we get any reports tomorrow morning. Did something in LSan change recently? Last I tried (>1 month ago) this didn't work on our bots.

[kcc]

I don't remember any changes in LSan.
What didn't work?

[inferno-chromium]

No, changes in LSan. Just that we had this disabled in ClusterFuz via a job type environment variable. I think this might be done due to overcaution since both asan and lsan are in same job type. On chromium, we had these in different job type. But now that we have well tested automatic blacklisting and deblacklisting, this will work smoothly. And we see new leaks, see https://clusterfuzz-external.appspot.com/v2/testcases?q=leak&showall=1

[kcc]

lovely leaks!
Two of them have short stack traces, most likely because some of
the deps are built w/o -fno-omit-frame-pointers.
In the long run we'll solve this by rebuilding the world (needed for msan too).
In short term, for those two, we'll just rerun manually with fast_unwind_on_malloc=0

[mikea]

Can CF use fast_unwind_on_malloc=0 for reproducer only?

[oliverchang]

Unfortunately that might break CF, since we expect the crash state to be the same/similar during fuzzing and running a testcase for reproducibility purposes.

[kcc]

can we make lsan work in the docker? (by enabling --cap-add SYS_PTRACE)
It's pretty confusing when leaks are not reported inside the docker but then are reported by CF (e.g. #168)

[mikea]

Do you know an easy way to check from the shell if ptrace is available? I'd like it to fail gracefully.

[kcc]

Run a simple program with a leak under lsan :)
But why do you want to fail gracefully if what we need is to pass (to make lsan work)?

[mikea]

I can add a flag to the helper.py, it is trivial. But people who run it without helper.py manually will see lsan report and will be confused.

[kcc]

ok... then indeed just run a simple test with a leak and verify that the leak is reported.

[inferno-chromium]

Well it is working inside docker.

(ENV) aarya@aarya-linux2:/build/oss-fuzz$ python infra/helper.py run_fuzzer $PROJECT_NAME gnutls_x509_parser_fuzzer
Running: docker build --pull -t ossfuzz/libfuzzer-runner infra/base-images/libfuzzer-runner
Sending build context to Docker daemon 2.56 kB
Step 1 : FROM ossfuzz/base-runner
latest: Pulling from ossfuzz/base-runner
Digest: sha256:292e0c2da06869c6c9fc83b87d05327b064c55219eaf172084fa0f56ce35c2fc
Status: Image is up to date for ossfuzz/base-runner:latest
---> fb0b1a3b7bbd
Step 2 : MAINTAINER mike.aizatsky@gmail.com
---> Using cache
---> c0d835c656a0
Step 3 : RUN apt-get install -y gdb zip
---> Using cache
---> df638e632437
Successfully built df638e632437
Running: docker run --rm -i -v /usr/local/google/home/aarya/build/oss-fuzz/build/out/gnutls:/out -e ASAN_OPTIONS=coverage=0,detect_leaks=1 -t ossfuzz/libfuzzer-runner run_fuzzer /out/gnutls_x509_parser_fuzzer
Using seed corpus: /out/gnutls_x509_parser_fuzzer_seed_corpus.zip
/out/gnutls_x509_parser_fuzzer /tmp/seed_corpus/
INFO: Seed: 1052878858
INFO: Loaded 0 modules (0 guards):
Loading corpus dir: /tmp/seed_corpus/
Loaded 1024/1504 files from /tmp/seed_corpus/
INFO: -max_len is not provided, using 8668
#0 READ units: 1504

=================================================================
==10==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 16 byte(s) in 1 object(s) allocated from:
#0 0x4d4f20 in calloc /src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:72
#1 0x550ef8 in gnutls_subject_alt_names_init /src/gnutls/lib/x509/x509_ext.c:53:10
#2 0x528588 in print_altname /src/gnutls/lib/x509/output.c:683:8
#3 0x51e8b6 in print_extension /src/gnutls/lib/x509/output.c:1009:4
#4 0x521bb7 in print_extensions /src/gnutls/lib/x509/output.c:1153:3
#5 0x515adc in print_cert /src/gnutls/lib/x509/output.c:1504:3
#6 0x51299a in gnutls_x509_crt_print /src/gnutls/lib/x509/output.c:1903:3
#7 0x511c55 in LLVMFuzzerTestOneInput /src/gnutls_x509_parser_fuzzer.cc:40:15
#8 0xa7e7cd in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:536:13
#9 0xa7f4d4 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:488:3
#10 0xa9d6ea in fuzzer::Fuzzer::RunOne(std::__1::vector<unsigned char, std::__1::allocator > const&) /src/libfuzzer/FuzzerInternal.h:119:41
#11 0xa7de8d in fuzzer::Fuzzer::ShuffleAndMinimize(std::__1::vector<std::__1::vector<unsigned char, std::__1::allocator >, std::__1::allocator<std::__1::vector<unsigned char, std::__1::allocator > > >) /src/libfuzzer/FuzzerLoop.cpp:467:30
#12 0xa1aaba in fuzzer::FuzzerDriver(int, char***, int ()(unsigned char const, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:534:6
#13 0xa0dbe8 in main /src/libfuzzer/FuzzerMain.cpp:20:10
#14 0x7f29a269c82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

SUMMARY: AddressSanitizer: 16 byte(s) leaked in 1 allocation(s).

INFO: a leak has been found in the initial corpus.

INFO: to ignore leaks on libFuzzer side use -detect_leaks=0.

MS: 0 ; base unit: 0000000000000000000000000000000000000000
artifact_prefix='./'; Test unit written to ./leak-20377d83e9b7aa6cc4b7f8a3fa2602e1fb22d947

diff --git a/infra/helper.py b/infra/helper.py
index 103797e..c08cb48 100755
--- a/infra/helper.py
+++ b/infra/helper.py
@@ -184,6 +184,7 @@ def run_fuzzer(run_args):
command = [
'docker', 'run', '--rm', '-i',
'-v', '%s:/out' % os.path.join(BUILD_DIR, 'out', args.project_name),

•  '-e', 'ASAN_OPTIONS=coverage=0,detect_leaks=1',
   '-t', 'ossfuzz/libfuzzer-runner',
   'run_fuzzer',
   '/out/%s' % args.fuzzer_name,
  

[inferno-chromium]

Here are some options we can set in helper.py, these are set in ClusterFuzz.

ASAN_OPTIONS=handle_sigill=1:strict_string_check=1:strict_memcmp=1:detect_container_overflow=1:coverage=0:allocator_may_return_null=1:use_sigaltstack=1:allocator_release_to_os=1:detect_stack_use_after_return=1:alloc_dealloc_mismatch=0:detect_leaks=1:print_scariness=1:strip_path_prefix=/workspace/:max_uar_stack_size_log=16:handle_abort=1:check_malloc_usable_size=0:quarantine_size_mb=10:detect_odr_violation=0:symbolize=1:handle_segv=1:fast_unwind_on_fatal=0
UBSAN_OPTIONS=print_stacktrace=1:symbolize=1:halt_on_error=1:print_summary=1
MSAN_OPTIONS= don't see anything interesting other than symbolize=1:print_stats=1