LeakSanitizer fatal errors

[ibc]

fuzzer generates this output:

	NEW_FUNC[1/90]: 0x76b050 in std::vector<RTC::RTCP::SenderReport*, std::allocator<RTC::RTCP::SenderReport*> >::begin() /usr/lib/gcc/x86_64-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_vector.h:548
	NEW_FUNC[2/90]: 0x76b1e0 in std::vector<RTC::RTCP::SenderReport*, std::allocator<RTC::RTCP::SenderReport*> >::end() /usr/lib/gcc/x86_64-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/stl_vector.h:566
==10==Could not attach to thread 7 (errno 1).
==10==Could not attach to thread 8 (errno 1).
==10==Failed suspending threads.
==7==LeakSanitizer has encountered a fatal error.
==7==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==7==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
MS: 1 ChangeBit-; base unit: 0761b60a503f3ef1eabd57414e819ee05af16580


artifact_prefix='/reports/'; Test unit written to /reports/crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64:

@fippo do you know how to interpret this error?

[ibc]

NOTE: fuzzer is being run with LSAN_OPTIONS="verbosity=1:log_threads=1" environment variable set.

[ibc]

Another similar:

==7==T0: stack [0x7ffe72b7b000,0x7ffe7337b000) size 0x800000; local=0x7ffe7337a1c4
==7==AddressSanitizer Init done
INFO: Seed: 2275877116
INFO: Loaded 1 modules   (27999 inline 8-bit counters): 27999 [0x104c438, 0x1053197),
INFO: Loaded 1 PC tables (27999 PCs): 27999 [0xc7b3a0,0xce8990),
==7==T1: stack [0x7f5cc2079000,0x7f5cc2878f40) size 0x7fff40; local=0x7f5cc2878e54
INFO:      432 files found in /new-corpus
INFO:       19 files found in fuzzer/corpora/stun-corpus
INFO: seed corpus: files: 451 min: 1b max: 660b total: 89806b rss: 34Mb
#452	INITED cov: 958 ft: 3264 corp: 186/23Kb lim: 4 exec/s: 452 rss: 45Mb
#663	NEW    cov: 958 ft: 3265 corp: 187/23Kb lim: 4 exec/s: 663 rss: 45Mb L: 116/592 MS: 1 ChangeASCIIInt-
#1024	pulse  cov: 958 ft: 3265 corp: 187/23Kb lim: 4 exec/s: 512 rss: 45Mb
#2048	pulse  cov: 958 ft: 3265 corp: 187/23Kb lim: 4 exec/s: 1024 rss: 46Mb
#2695	REDUCE cov: 958 ft: 3265 corp: 187/23Kb lim: 6 exec/s: 1347 rss: 47Mb L: 24/592 MS: 2 ShuffleBytes-EraseBytes-
#4096	pulse  cov: 958 ft: 3265 corp: 187/23Kb lim: 6 exec/s: 2048 rss: 48Mb
#8192	pulse  cov: 958 ft: 3265 corp: 187/23Kb lim: 11 exec/s: 4096 rss: 51Mb
	NEW_FUNC[1/1]: 0xa28fc0 in RTC::RTCP::Packet::SetNext(RTC::RTCP::Packet*) /mediasoup/worker/out/../include/RTC/RTCP/Packet.hpp:116
==10==Could not attach to thread 7 (errno 1).
==10==Could not attach to thread 8 (errno 1).
==10==Failed suspending threads.
==7==LeakSanitizer has encountered a fatal error.
==7==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==7==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
MS: 4 ChangeBit-ChangeASCIIInt-EraseBytes-ChangeByte-; base unit: 371d4ecc043249b12a03bdc7568695911fdc6405


artifact_prefix='/reports/'; Test unit written to /reports/crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64:

[fippo]

lowtech approach: limiting it to a single thread and printing the hex version of the input at the beginning of the fuzz target might give you a repro?

[ibc]

Note, mediasoup-worker is single thread. I'll test that tomorrow anyway.

[ibc]

Na, same happens with -workers=1, so there may be a bug somewhere.

[ibc]

NOTE: The crash file generated when the error in the issue description happens is completely empty (0 bytes). Also, the Base64: output is empty too.

Does it mean that fuzzer is testing with len = 0 in this case?

[fippo]

zero length input can happen from libfuzzer. Not in our typical usecases because of the demuxing

[ibc]

Yep, it must be anything else.

[ibc]

I know that, by forcing a memory leak in the code (non deallocating objects), a similar failure happens. So IMHO it should be related.

[ibc]

Right now this strange error just happens when testing RTCP::Packet::Parse(data, len). Sometimes the error says absolutely nothing:

==16487==AddressSanitizer: libc interceptors initialized
|| `[0x10007fff8000, 0x7fffffffffff]` || HighMem    ||
|| `[0x02008fff7000, 0x10007fff7fff]` || HighShadow ||
|| `[0x00008fff7000, 0x02008fff6fff]` || ShadowGap  ||
|| `[0x00007fff8000, 0x00008fff6fff]` || LowShadow  ||
|| `[0x000000000000, 0x00007fff7fff]` || LowMem     ||
MemToShadow(shadow): 0x00008fff7000 0x000091ff6dff 0x004091ff6e00 0x02008fff6fff
redzone=16
max_redzone=2048
quarantine_size_mb=256M
thread_local_quarantine_size_kb=1024K
malloc_context_size=30
SHADOW_SCALE: 3
SHADOW_GRANULARITY: 8
SHADOW_OFFSET: 0x7fff8000
==16487==Installed the sigaction for signal 11
==16487==Installed the sigaction for signal 7
==16487==Installed the sigaction for signal 8
==16487==T0: stack [0x7ffda2a55000,0x7ffda3255000) size 0x800000; local=0x7ffda32531c4
==16487==AddressSanitizer Init done
INFO: Seed: 2816933944
INFO: Loaded 1 modules   (28029 inline 8-bit counters): 28029 [0x104d5b8, 0x1054335),
INFO: Loaded 1 PC tables (28029 PCs): 28029 [0xc7c900,0xcea0d0),
==16487==T1: stack [0x7f3acfc79000,0x7f3ad0478f40) size 0x7fff40; local=0x7f3ad0478e54
INFO:      553 files found in fuzzer/new-corpus
INFO:       66 files found in fuzzer/corpora/rtcp-corpus
INFO: seed corpus: files: 619 min: 1b max: 660b total: 106185b rss: 34Mb
==16489==Could not attach to thread 16487 (errno 1).
==16489==Could not attach to thread 16488 (errno 1).
==16489==Failed suspending threads.
==16487==LeakSanitizer has encountered a fatal error.
==16487==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==16487==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
MS: 0 ; base unit: 0000000000000000000000000000000000000000


artifact_prefix='fuzzer/reports/'; Test unit written to fuzzer/reports/crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64:

[ibc]

@fippo theoretically if I call the fuzzer binary passing as argument a crash report file, it should reproduce the issue(if not fixed), right? However when I do it I see no output other than the annoying LeakSanitizer has encountered a fatal error shown in the comment before this one.

[ibc]

@jmillan I strongly think there is a leaking issue in RTCP::Packet::Parse() at the end:

				data += packetLen;
				len -= packetLen;

				if (first == nullptr)
					first = current;
				else
					last->SetNext(current);

				last = current->GetNext() != nullptr ? current->GetNext() : current;

I've tested by disabling some specific RTCP parsers in there and the issue also happens. So IHMO there is a leak when parsing compound packets.

[ibc]

It may even happen that last->SetNext(current); is being called with last being nullptr. Not sure, but sometimes I get this:

	NEW_FUNC[1/1]: 0xa29e90 in RTC::RTCP::Packet::SetNext(RTC::RTCP::Packet*) /mediasoup/worker/out/../include/RTC/RTCP/Packet.hpp:116
==19813==Could not attach to thread 19810 (errno 1).
==19813==Could not attach to thread 19811 (errno 1).
==19813==Failed suspending threads.
==19810==LeakSanitizer has encountered a fatal error.
==19810==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==19810==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
MS: 1 CopyPart-; base unit: c1f338585ebdd9996859988013de1dc4a6f7c6c2


artifact_prefix='fuzzer/reports/'; Test unit written to fuzzer/reports/crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64:

[ibc]

I've added some logs in fuzzer.cpp to log whether it looks like a RTCP packet and whether it was indeed (or not) a RTCP packet. It's unclear to me what the issue is, but logs say this:

[...]
>>> 1) it looks like RTCP
>>> 3) no, it was no a real RTCP packet!!!
>>> 1) it looks like RTCP
>>> 2) yes, it was valid RTCP :)
>>> 1) it looks like RTCP
>>> 2) yes, it was valid RTCP :)
	NEW_FUNC[1/1]: 0xa29e90 in RTC::RTCP::Packet::SetNext(RTC::RTCP::Packet*) /mediasoup/worker/out/../include/RTC/RTCP/Packet.hpp:116
>>> 1) it looks like RTCP
>>> 2) yes, it was valid RTCP :)
==19966==Could not attach to thread 19963 (errno 1).
==19966==Could not attach to thread 19964 (errno 1).
==19966==Failed suspending threads.
==19963==LeakSanitizer has encountered a fatal error.
==19963==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==19963==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
MS: 2 ChangeBinInt-CopyPart-; base unit: 7c3cd04c4ff1e8737aa2877a221054131e7b8521


artifact_prefix='fuzzer/reports/'; Test unit written to fuzzer/reports/crash-da39a3ee5e6b4b0d3255bfef95601890afd80709

The code in fuzzer.cpp is just this one:

if (RTCP::Packet::IsRtcp(data, len))
{
	std::cout << ">>> 1) it looks like RTCP" << std::endl;

	RTCP::Packet* packet = RTCP::Packet::Parse(data, len);

	if (packet)
	{
		std::cout << ">>> 2) yes, it was valid RTCP :)" << std::endl;

		delete packet;
	}
	else
	{
		std::cout << ">>> 3) no, it was no a real RTCP packet!!!" << std::endl;
	}
}

[ibc]

OK, it seems that in docker run the following is required to enable ptrace (needed by LSAN): --cap-add SYS_PTRACE.

More info here.

[ibc]

OK, so it was all! By enabling ptrace support and running the binary in the Docker container as follows:

LSAN_OPTIONS=verbosity=1:log_threads=1 ./out/Release/mediasoup-worker-fuzzer -artifact_prefix=fuzzer/reports/ -max_len=1800 -workers=1 fuzzer/new-corpus fuzzer/corpora/rtcp-corpus

we get the following: #247.

So let's close this issue.