NSS startup crash

[inferno-chromium]

Weird that this is only reproducing on ClusterFuzz and not on Docker container locally with fuzzer hash in nss. @ttaubert - any clues ?

hash: ../../fuzz/shared.h:21: NSSDatabase::NSSDatabase(): Assertion `NSS_NoDB_Init(nullptr) == SECSuccess failed.
ASAN:DEADLYSIGNAL

==1==ERROR: AddressSanitizer: ABRT on unknown address 0x000000000001 (pc 0x7f8fb75f9418 bp 0x000000c06760 sp 0x7ffcca718ea8 T0)
SCARINESS: 10 (signal)
#0 0x7f8fb75f9417 in gsignal
#1 0x7f8fb75fb019 in abort
#2 0x7f8fb75f1bd6 in libc.so.6
#3 0x7f8fb75f1c81 in __assert_fail
#4 0x51502a in NSSDatabase::NSSDatabase() /src/nss/fuzz/shared.h:21:19
#5 0x51502a in LLVMFuzzerTestOneInput /src/nss/fuzz/hash_target.cc:19
#6 0x969ee8 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:550:13
#7 0x96906d in fuzzer::Fuzzer::ShuffleAndMinimize(std::__1::vector<std::__1::vector<unsigned char, std::__1::allocator >, std::__1::allocator<std::__1::vector<unsigned char, std::__1::allocator > > >) /src/libfuzzer/FuzzerLoop.cpp:477:3
#8 0xa4790a in fuzzer::FuzzerDriver(int, char***, int ()(unsigned char const, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:565:6
#9 0x9c69b8 in main /src/libfuzzer/FuzzerMain.cpp:20:10
#10 0x7f8fb75e482f in __libc_start_main
#11 0x41dc68 in _start
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: ABRT (/lib/x86_64-linux-gnu/libc.so.6+0x35417)
==1==ABORTING

[ttaubert]

Weird that this is only reproducing on ClusterFuzz and not on Docker container locally with fuzzer hash in nss.

Yeah, I was thinking that too, I tested it before filing the PR. So we had to work around a few issues here, trying to create a static NSS build. NSS usually isn't built that way, and maybe there's something wrong still and it's trying to load some DSO?

[ttaubert]

Does ClusterFuzz use the same base-runner image that's here in the repository?

[ttaubert]

What would be a good way to debug this? Is it possible to get an strace of the fuzzer run? Or maybe we'll get it to somehow reproduce locally.

[inferno-chromium]

So, this is a minijail sandboxing issue, Oliver can take a look. The binary runs fine on the bot without minijail.

[inferno-chromium]

@ttaubert - does hash fuzzer writes any temp database anywhere like user home, etc (as part of NSS_NoDB_Init) ? I think that could be the reason for this initialization failure.

[inferno-chromium]

@ttaubert - see https://github.com/google/oss-fuzz/blob/master/docs/fuzzer_environment.md#file-system, can you modify code to write database in /tmp. everything else will be read-only.

[ttaubert]

We currently hard-fail when seeding from /dev/urandom fails, and we assume that this might be the cause. We seed from urandom but never use it because we have a deterministic PRNG for fuzzing. We now simply skip seeding in "fuzzing mode".

The latest NSS revision has the fix, can you please try again on ClusterFuzz? Thanks!

[Dor1s]

I've just kicked off the NSS build. In 1-2 hours we'll see how it's going on CF.

[inferno-chromium]

Ok now fixed.

Command: /mnt/scratch0/clusterfuzz/slave-bot/builds/clusterfuzz-builds_nss_c7f62f1d50ead7b467de2ea6701001cb50f4098f/revisions/hash -max_len=6048 -rss_limit_mb=2048 -timeout=25 -artifact_prefix=/ -max_total_time=2950 -print_final_stats=1 /new /nss_hash
Bot: ossfuzz-linux-4027
Time ran: 2951.060843

INFO: Seed: 1888960566
INFO: Loaded 1 modules (56073 guards): [0xffa3f0, 0x1031014),
Loading corpus dir: /new
Loading corpus dir: /nss_hash
Loaded 1024/4111 files from /nss_hash
Loaded 2048/4111 files from /nss_hash
Loaded 4096/4111 files from /nss_hash
#0 READ units: 4107
#4096 pulse cov: 1217 ft: 350 corp: 28/17Kb exec/s: 2048 rss: 60Mb
#4107 INITED cov: 1217 ft: 350 corp: 28/17Kb exec/s: 2053 rss: 60Mb
#8192 pulse cov: 1217 ft: 350 corp: 28/17Kb exec/s: 2048 rss: 60Mb
#16384 pulse cov: 1217 ft: 350 corp: 28/17Kb exec/s: 1820 rss: 60Mb
#32768 pulse cov: 1217 ft: 350 corp: 28/17Kb exec/s: 1820 rss: 60Mb
#65536 pulse cov: 1217 ft: 350 corp: 28/17Kb exec/s: 1820 rss: 60Mb
#131072 pulse cov: 1217 ft: 350 corp: 28/17Kb exec/s: 1872 rss: 60Mb
#262144 pulse cov: 1217 ft: 350 corp: 28/17Kb exec/s: 1941 rss: 60Mb
#524288 pulse cov: 1217 ft: 350 corp: 28/17Kb exec/s: 1963 rss: 60Mb
#1048576 pulse cov: 1217 ft: 350 corp: 28/17Kb exec/s: 1952 rss: 60Mb
#1677721 NEW cov: 1219 ft: 352 corp: 29/17Kb exec/s: 1983 rss: 60Mb L: 64 MS: 4 ChangeASCIIInt-ChangeASCIIInt-ChangeASCIIInt-ChangeBinInt-
#2097152 pulse cov: 1219 ft: 352 corp: 29/17Kb exec/s: 1989 rss: 60Mb
#4194304 pulse cov: 1219 ft: 352 corp: 29/17Kb exec/s: 1985 rss: 60Mb
#5865347 DONE cov: 1219 ft: 352 corp: 29/17Kb exec/s: 1987 rss: 60Mb
Done 5865347 runs in 2951 second(s)
stat::number_of_executed_units: 5865347
stat::average_exec_per_sec: 1987
stat::new_units_added: 1
stat::slowest_unit_time_sec: 0
stat::peak_rss_mb: 60