Add fuzzer for libchewing #5
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Ah, I found a problem. This fuzzer needs data files. With helper script's "run_ruzzer", it cannot find the data files and the coverage (cov) stop at 160. May I install data files into /out ? |
|
Yes, you can install data files into /out. What's in it? |
|
Done. CL revised. libchewing is a library for Chinese character input. It needs tables for "keystrokes to characters" conversion. Now my fuzzer assumes the data files put at the same directory as the fuzzer executable. |
mikea
reviewed
Oct 6, 2016
| test/stress.o test/libtesthelper.la src/libchewing.la $LDFLAGS /work/libfuzzer/*.o | ||
|
|
||
| # install data files | ||
| make -C data pkgdatadir=/out install |
There was a problem hiding this comment.
This also creates /out/.libs file
Can these be removed?
mikea
added a commit
that referenced
this pull request
Nov 14, 2016
fuzzers fail with:
=================================================================
�[1m�[31m==18057==ERROR: AddressSanitizer: global-buffer-overflow on address 0x000000b8a144 at pc 0x0000007ae0ca bp 0x7fff2b91a4d0 sp 0x7fff2b91a4c8
�[1m�[0m�[1m�[34mWRITE of size 4 at 0x000000b8a144 thread T0�[1m�[0m
#0 0x7ae0c9 in fuzzer::TracePC::HandleInit(unsigned int*, unsigned int*) /src/libfuzzer/FuzzerTracePC.cpp:49:8
#1 0x7bcab9 in __sanitizer_cov_trace_pc_guard_init /src/libfuzzer/FuzzerTracePC.cpp:286:15
#2 0x5156bf in sancov.module_ctor (/out/curl_fuzzer+0x5156bf)
#3 0x88c1cc in __libc_csu_init (/out/curl_fuzzer+0x88c1cc)
#4 0x7f4ab7aed7be in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x207be)
#5 0x41fd78 in _start (/out/curl_fuzzer+0x41fd78)
This was referenced Dec 8, 2016
This was referenced Jan 27, 2017
Closed
Merged
Closed
This was referenced Oct 22, 2018
Merged
Merged
mhlakhani
added a commit
to mhlakhani/oss-fuzz
that referenced
this pull request
Sep 25, 2019
The corresponding github issue has more detail on the problem.
This code is not in any meaningful state to be committed -- it contains a bunch of hacks I had to make to get the binary to build statically -- libunwind is linked in (via folly), and I had to make changes to include liblzma as well (which libunwind depends on) as well as fix some other unrelated cmake issues.
Reproduction instructions below.
First, build it with `python infra/helper.py build_fuzzers --sanitizer address proxygen`
Then, check the build (it will pass): `python infra/helper.py check_build proxygen`
Then, run the fuzzer: `python infra/helper.py run_fuzzer proxygen ProxygenHTTP1xFuzzer`
It will fail in about ~30 seconds with the below trace. If the fuzzer goes down another path, I've had it reproduce easily by pulling up the shell, running `base64 -d`, pushing that input to a file, and then running the fuzzer directly on that file.
```
terminating with uncaught exception of type folly::ConversionError: Non-digit character found: "OPY"
AddressSanitizer:DEADLYSIGNAL
=================================================================
==11==ERROR: AddressSanitizer: ABRT on unknown address 0x00000000000b (pc 0x7fc166eb7428 bp 0x000000000001 sp 0x7ffcff0a6c78 T0)
SCARINESS: 10 (signal)
#0 0x7fc166eb7427 in gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x35427)
google#1 0x7fc166eb9029 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x37029)
google#2 0x9ab7fa in abort_message (/out/ProxygenHTTP1xFuzzer+0x9ab7fa)
google#3 0x9b009d in demangling_terminate_handler() (/out/ProxygenHTTP1xFuzzer+0x9b009d)
google#4 0x9ab282 in std::__terminate(void (*)()) (/out/ProxygenHTTP1xFuzzer+0x9ab282)
google#5 0x9adebd in __cxxabiv1::call_terminate(bool, _Unwind_Exception*) (/out/ProxygenHTTP1xFuzzer+0x9adebd)
google#6 0x9ade60 in __cxxabiv1::scan_eh_tab(__cxxabiv1::(anonymous namespace)::scan_results&, _Unwind_Action, bool, _Unwind_Exception*, _Unwind_Context*) (/out/ProxygenHTTP1xFuzzer+0x9ade60)
google#7 0x9ad5c6 in __gxx_personality_v0 (/out/ProxygenHTTP1xFuzzer+0x9ad5c6)
google#8 0x7fc16725c262 in _Unwind_RaiseException (/lib/x86_64-linux-gnu/libgcc_s.so.1+0x10262)
google#9 0x9acfd6 in __cxa_throw (/out/ProxygenHTTP1xFuzzer+0x9acfd6)
google#10 0x5cedab in void folly::throw_exception<folly::ConversionError>(folly::ConversionError&&) /src/proxygen/proxygen/_build/deps/include/folly/lang/Exception.h:36:3
google#11 0x5eea03 in _ZZN5folly2toItEENSt3__19enable_ifIXntsr3std7is_sameINS_5RangeIPKcEET_EE5valueES7_E4typeES6_ENKUlNS_14ConversionCodeEE_clESA_ /src/proxygen/proxygen/_build/deps/include/folly/Conv.h:1581:26
google#12 0x5eea03 in _ZN5folly15expected_detail30expected_detail_ExpectedHelper14ExpectedHelper12thenOrThrow_IRNS0_15ExpectedStorageINS_5RangeIPKcEENS_14ConversionCodeELNS0_11StorageTypeE1EEENS_6detail18CheckTrailingSpaceEZNS_2toItEENSt3__19enable_ifIXntsr3std7is_sameIS8_T_EE5valueESI_E4typeES8_EUlS9_E_NS_8ExpectedINS_4UnitES9_EEvLb0ELi0EEET2_OSI_OT0_OT1_ /src/proxygen/proxygen/_build/deps/include/folly/Expected.h:610:5
google#13 0x5ed846 in _ZNR5folly8ExpectedINS_5RangeIPKcEENS_14ConversionCodeEE11thenOrThrowINS_6detail18CheckTrailingSpaceEZNS_2toItEENSt3__19enable_ifIXntsr3std7is_sameIS4_T_EE5valueESD_E4typeES4_EUlS5_E_EEDTclclsr3stdE7declvalISD_EEclL_ZNSB_7declvalIRS4_EEDTclsr3std3__1E9__declvalISD_ELi0EEEvEEEEOSD_OT0_ /src/proxygen/proxygen/_build/deps/include/folly/Expected.h:1226:16
google#14 0x5ed846 in _ZN5folly2toItEENSt3__19enable_ifIXntsr3std7is_sameINS_5RangeIPKcEET_EE5valueES7_E4typeES6_ /src/proxygen/proxygen/_build/deps/include/folly/Conv.h:1579:8
google#15 0x5ed445 in proxygen::ParseURL::parseAuthority() /src/proxygen/proxygen/lib/utils/ParseURL.cpp:155:15
google#16 0x5ec5e9 in proxygen::ParseURL::parseNonFully() /src/proxygen/proxygen/lib/utils/ParseURL.cpp:140:8
google#17 0x5ea87a in proxygen::ParseURL::parse() /src/proxygen/proxygen/lib/utils/ParseURL.cpp:96:5
google#18 0x59a99c in proxygen::ParseURL::init(folly::Range<char const*>) /src/proxygen/proxygen/lib/utils/ParseURL.h:34:5
google#19 0x58add1 in proxygen::ParseURL::ParseURL(folly::Range<char const*>) /src/proxygen/proxygen/lib/utils/ParseURL.h:28:5
google#20 0x58add1 in proxygen::ParseURL proxygen::HTTPMessage::setURL<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) /src/proxygen/proxygen/lib/http/HTTPMessage.h:202:14
google#21 0x5795f0 in proxygen::HTTP1xCodec::onHeadersComplete(unsigned long) /src/proxygen/proxygen/lib/http/codec/HTTP1xCodec.cpp:976:31
google#22 0x58e2a3 in proxygen::HTTP1xCodec::onHeadersCompleteCB(proxygen::http_parser*, char const*, unsigned long) /src/proxygen/proxygen/lib/http/codec/HTTP1xCodec.cpp:1315:19
google#23 0x5f2222 in proxygen::http_parser_execute(proxygen::http_parser*, proxygen::http_parser_settings const*, char const*, unsigned long) /src/proxygen/proxygen/external/http_parser/http_parser_cpp.cpp:1868:17
google#24 0x577783 in proxygen::HTTP1xCodec::onIngress(folly::IOBuf const&) /src/proxygen/proxygen/lib/http/codec/HTTP1xCodec.cpp:200:26
google#25 0x55b728 in unsigned long proxygen::parse<proxygen::HTTP1xCodec>(proxygen::HTTP1xCodec*, unsigned char const*, unsigned int, int, std::__1::function<bool ()>) /src/proxygen/proxygen/lib/http/codec/test/TestUtils.h:57:23
google#26 0x55a6ef in LLVMFuzzerTestOneInput /src/proxygen/proxygen/fuzzers/ProxygenHTTP1xFuzzer.cpp:23:3
google#27 0x460771 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:554:15
google#28 0x45fe95 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:470:3
google#29 0x462247 in fuzzer::Fuzzer::MutateAndTestOne() /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:696:19
google#30 0x462fe5 in fuzzer::Fuzzer::Loop(std::Fuzzer::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:832:5
google#31 0x450dd8 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:825:6
google#32 0x47b442 in main /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMain.cpp:19:10
google#33 0x7fc166ea282f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
google#34 0x424468 in _start (/out/ProxygenHTTP1xFuzzer+0x424468)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: ABRT (/lib/x86_64-linux-gnu/libc.so.6+0x35427) in gsignal
==11==ABORTING
MS: 3 CrossOver-ChangeBit-EraseBytes-; base unit: 467e3fbf38770044dea566169b84c60cb269c89a
0x43,0x4f,0x50,0x59,0x20,0x43,0x3a,0x4f,0x50,0x59,0x2f,0x4f,0xff,0xa,0x43,0x4f,0x50,0x59,0x20,0x2f,0x4f,0xff,0xa,0x43,0x4f,0x50,0x59,0x20,0x2f,0x6f,0x20,0xff,0x50,0x59,0x0,0x43,0xcf,0x50,0x59,0xff,0x4f,0x50,0x33,
COPY C:OPY/O\xff\x0aCOPY /O\xff\x0aCOPY /o \xffPY\x00C\xcfPY\xffOP3
artifact_prefix='./'; Test unit written to ./crash-2cd12847d55ebf08ec2eaee4e814c52e545e7f92
Base64: Q09QWSBDOk9QWS9P/wpDT1BZIC9P/wpDT1BZIC9vIP9QWQBDz1BZ/09QMw==
```
This seems wrong to me since the code is inside a try/catch block: https://github.com/facebook/proxygen/blob/master/proxygen/lib/utils/ParseURL.cpp#L159-L164
I can confirm that the same input does not crash the fuzzer in the version being built on oss-fuzz trunk (and indeed it would have also reported this as a test-case by now given it's been running over a day)
jonathanmetzman
added a commit
that referenced
this pull request
Dec 6, 2022
cc @oliverchang @alan32liu after #9100 and #8448 After compiling locally, I can see that `./SystemSan ./target_dns -dict=vuln.dict` crashes in a few seconds with ``` ===BUG DETECTED: Arbitrary domain name resolution=== ===Domain resolved: .f.z=== ===DNS request type: 0, class: 256=== ==315== ERROR: libFuzzer: deadly signal #0 0x539131 in __sanitizer_print_stack_trace /src/llvm-project/compiler-rt/lib/asan/asan_stack.cpp:87:3 #1 0x457c48 in fuzzer::PrintStackTrace() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtil.cpp:210:5 #2 0x43c923 in fuzzer::Fuzzer::CrashCallback() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:233:3 #3 0x7fa57940041f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1441f) (BuildId: 7b4536f41cdaa5888408e82d0836e33dcf436466) #4 0x7fa5793ff7db in send (/lib/x86_64-linux-gnu/libpthread.so.0+0x137db) (BuildId: 7b4536f41cdaa5888408e82d0836e33dcf436466) #5 0x503ba4 in __interceptor_send /src/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:6802:17 #6 0x7fa578abf462 (/lib/x86_64-linux-gnu/libresolv.so.2+0xb462) (BuildId: 4519041bde5b859c55798ac0745b0b6199cb7d94) #7 0x7fa578abbc43 in __res_context_query (/lib/x86_64-linux-gnu/libresolv.so.2+0x7c43) (BuildId: 4519041bde5b859c55798ac0745b0b6199cb7d94) #8 0x7fa578abc8ed in __res_context_search (/lib/x86_64-linux-gnu/libresolv.so.2+0x88ed) (BuildId: 4519041bde5b859c55798ac0745b0b6199cb7d94) #9 0x7fa578ad2cc1 (/lib/x86_64-linux-gnu/libnss_dns.so.2+0x2cc1) (BuildId: 3fac4ec397ba8e8938fe298f103113f315465130) #10 0x7fa578ad2e8b in _nss_dns_gethostbyname3_r (/lib/x86_64-linux-gnu/libnss_dns.so.2+0x2e8b) (BuildId: 3fac4ec397ba8e8938fe298f103113f315465130) #11 0x7fa578ad2f41 in _nss_dns_gethostbyname2_r (/lib/x86_64-linux-gnu/libnss_dns.so.2+0x2f41) (BuildId: 3fac4ec397ba8e8938fe298f103113f315465130) #12 0x7fa5792fdc9d in gethostbyname2_r (/lib/x86_64-linux-gnu/libc.so.6+0x130c9d) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) #13 0x7fa5792d179e (/lib/x86_64-linux-gnu/libc.so.6+0x10479e) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) #14 0x7fa5792d2f58 in getaddrinfo (/lib/x86_64-linux-gnu/libc.so.6+0x105f58) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) #15 0x4d93ac in getaddrinfo /src/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2667:13 #16 0x56c8d9 in LLVMFuzzerTestOneInput /out/SystemSan/target_dns.cpp:35:11 #17 0x43dec3 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:611:15 #18 0x43d6aa in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:514:3 #19 0x43ed79 in fuzzer::Fuzzer::MutateAndTestOne() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:757:19 #20 0x43fa45 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile> >&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:895:5 #21 0x42edaf in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:912:6 #22 0x458402 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10 #23 0x7fa5791f1082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) #24 0x41f7ed in _start (/out/SystemSan/target_dns+0x41f7ed) NOTE: libFuzzer has rudimentary signal handlers. Combine libFuzzer with AddressSanitizer or similar for better crash reports. SUMMARY: libFuzzer: deadly signal MS: 2 CrossOver-ManualDict- DE: "f.z"-; base unit: ac3478d69a3c81fa62e60f5c3696165a4e5e6ac4 0x66,0x2e,0x7a, f.z artifact_prefix='./'; Test unit written to ./crash-926813b2d6adde373f96a10594a5314951588384 Base64: Zi56 ``` You can also try ``` echo -n f.z > toto ./SystemSan ./target_dns toto ``` Co-authored-by: Oliver Chang <oliverchang@users.noreply.github.com> Co-authored-by: jonathanmetzman <31354670+jonathanmetzman@users.noreply.github.com>
eamonnmcmanus
pushed a commit
to eamonnmcmanus/oss-fuzz
that referenced
this pull request
Mar 15, 2023
cc @oliverchang @alan32liu after google#9100 and google#8448 After compiling locally, I can see that `./SystemSan ./target_dns -dict=vuln.dict` crashes in a few seconds with ``` ===BUG DETECTED: Arbitrary domain name resolution=== ===Domain resolved: .f.z=== ===DNS request type: 0, class: 256=== ==315== ERROR: libFuzzer: deadly signal #0 0x539131 in __sanitizer_print_stack_trace /src/llvm-project/compiler-rt/lib/asan/asan_stack.cpp:87:3 google#1 0x457c48 in fuzzer::PrintStackTrace() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtil.cpp:210:5 google#2 0x43c923 in fuzzer::Fuzzer::CrashCallback() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:233:3 google#3 0x7fa57940041f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1441f) (BuildId: 7b4536f41cdaa5888408e82d0836e33dcf436466) google#4 0x7fa5793ff7db in send (/lib/x86_64-linux-gnu/libpthread.so.0+0x137db) (BuildId: 7b4536f41cdaa5888408e82d0836e33dcf436466) google#5 0x503ba4 in __interceptor_send /src/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:6802:17 google#6 0x7fa578abf462 (/lib/x86_64-linux-gnu/libresolv.so.2+0xb462) (BuildId: 4519041bde5b859c55798ac0745b0b6199cb7d94) google#7 0x7fa578abbc43 in __res_context_query (/lib/x86_64-linux-gnu/libresolv.so.2+0x7c43) (BuildId: 4519041bde5b859c55798ac0745b0b6199cb7d94) google#8 0x7fa578abc8ed in __res_context_search (/lib/x86_64-linux-gnu/libresolv.so.2+0x88ed) (BuildId: 4519041bde5b859c55798ac0745b0b6199cb7d94) google#9 0x7fa578ad2cc1 (/lib/x86_64-linux-gnu/libnss_dns.so.2+0x2cc1) (BuildId: 3fac4ec397ba8e8938fe298f103113f315465130) google#10 0x7fa578ad2e8b in _nss_dns_gethostbyname3_r (/lib/x86_64-linux-gnu/libnss_dns.so.2+0x2e8b) (BuildId: 3fac4ec397ba8e8938fe298f103113f315465130) google#11 0x7fa578ad2f41 in _nss_dns_gethostbyname2_r (/lib/x86_64-linux-gnu/libnss_dns.so.2+0x2f41) (BuildId: 3fac4ec397ba8e8938fe298f103113f315465130) google#12 0x7fa5792fdc9d in gethostbyname2_r (/lib/x86_64-linux-gnu/libc.so.6+0x130c9d) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) google#13 0x7fa5792d179e (/lib/x86_64-linux-gnu/libc.so.6+0x10479e) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) google#14 0x7fa5792d2f58 in getaddrinfo (/lib/x86_64-linux-gnu/libc.so.6+0x105f58) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) google#15 0x4d93ac in getaddrinfo /src/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2667:13 google#16 0x56c8d9 in LLVMFuzzerTestOneInput /out/SystemSan/target_dns.cpp:35:11 google#17 0x43dec3 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:611:15 google#18 0x43d6aa in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:514:3 google#19 0x43ed79 in fuzzer::Fuzzer::MutateAndTestOne() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:757:19 google#20 0x43fa45 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile> >&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:895:5 google#21 0x42edaf in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:912:6 google#22 0x458402 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10 google#23 0x7fa5791f1082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee) google#24 0x41f7ed in _start (/out/SystemSan/target_dns+0x41f7ed) NOTE: libFuzzer has rudimentary signal handlers. Combine libFuzzer with AddressSanitizer or similar for better crash reports. SUMMARY: libFuzzer: deadly signal MS: 2 CrossOver-ManualDict- DE: "f.z"-; base unit: ac3478d69a3c81fa62e60f5c3696165a4e5e6ac4 0x66,0x2e,0x7a, f.z artifact_prefix='./'; Test unit written to ./crash-926813b2d6adde373f96a10594a5314951588384 Base64: Zi56 ``` You can also try ``` echo -n f.z > toto ./SystemSan ./target_dns toto ``` Co-authored-by: Oliver Chang <oliverchang@users.noreply.github.com> Co-authored-by: jonathanmetzman <31354670+jonathanmetzman@users.noreply.github.com>
DavidKorczynski
pushed a commit
that referenced
this pull request
Apr 5, 2023
We've still got an issue with crashes on the urllib3 requests test that uses the mock HTTP server. Fix #9958 to handle port mapping errors didn't resolve it. I got a feeling there's an ordering issue. Looking at the error logs [https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=56500#c2](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=56500#c2) there appears to be an issue where we're throwing exceptions before the coverage completes. ``` === Uncaught Python exception: === -- | MaxRetryError: HTTPConnectionPool(host='localhost', port=8011): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4cdf33d1f0>: Failed to establish a new connection: [Errno 101] Network is unreachable')) | Traceback (most recent call last): | File "fuzz_requests.py", line 109, in TestOneInput | File "urllib3/_request_methods.py", line 118, in request | File "urllib3/_request_methods.py", line 217, in request_encode_body | File "urllib3/poolmanager.py", line 433, in urlopen | File "urllib3/connectionpool.py", line 874, in urlopen | File "urllib3/connectionpool.py", line 874, in urlopen | File "urllib3/connectionpool.py", line 874, in urlopen | File "urllib3/connectionpool.py", line 844, in urlopen | File "urllib3/util/retry.py", line 505, in increment | MaxRetryError: HTTPConnectionPool(host='localhost', port=8011): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4cdf33d1f0>: Failed to establish a new connection: [Errno 101] Network is unreachable')) | | INFO: Instrumenting 3854 functions... | INFO: Instrumentation complete. | ==10674== ERROR: libFuzzer: fuzz target exited | #0 0x7f4ce0bac694 in __sanitizer_print_stack_trace /src/llvm-project/compiler-rt/lib/ubsan/ubsan_diag_standalone.cpp:31:3 | #1 0x7f4ce0b2df48 in fuzzer::PrintStackTrace() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtil.cpp:210:5 | #2 0x7f4ce0b12cdc in fuzzer::Fuzzer::ExitCallback() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:250:3 | #3 0x7f4ce09068a6 in __run_exit_handlers /build/glibc-SzIz7B/glibc-2.31/stdlib/exit.c:108:8 | #4 0x7f4ce0906a5f in exit /build/glibc-SzIz7B/glibc-2.31/stdlib/exit.c:139:3 | #5 0x7f4ce03b2c78 in libpython3.8.so.1.0 | #6 0x7f4ce03b76cf in libpython3.8.so.1.0 | #7 0x403ad2 in fuzz_requests.pkg | #8 0x403e67 in fuzz_requests.pkg | #9 0x7f4ce08e4082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16 | #10 0x40249d in fuzz_requests.pkg | | SUMMARY: libFuzzer: fuzz target exited ``` This is an attempted fix inspired by the requests [fuzz_server.py](https://github.com/google/oss-fuzz/blob/master/projects/requests/fuzz_server.py) where the lifecycle of the test thread is managed within the server. Since the web server is created at the start of `TestOneInput` I don't expect there to be any timing issues or thread initialisation issues.
DavidKorczynski
added a commit
that referenced
this pull request
May 8, 2023
The current number of parallel fuzzers running is set to the number of available CPUs. This is causing issues in Tensorflow: ``` Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! Step #5: /usr/local/bin/coverage: line 75: 4501 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file ... Step #5: error: decode_compressed_fuzz: Failed to load coverage: No such file or directory Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! Step #5: /usr/local/bin/coverage: line 75: 4873 Killed lvm-cov show -instr-profile=$profdata_file -object=$target -line-coverage-gt=0 $shared_libraries $BRANCH_COV_ARGS $LL VM_COV_COMMON_ARGS > ${TEXTCOV_REPORT_DIR}/$target.covreport Step #5: /usr/local/bin/coverage: line 75: 4897 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file ... Step #5: error: saved_model_fuzz: Failed to load coverage: No such file or directory Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! Step #5: /usr/local/bin/coverage: line 75: 4638 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file Step #5: [2023-05-08 11:57:05,246 INFO] Finding shared libraries for targets (if any). ... Step #5: [2023-05-08 11:57:09,911 INFO] Finished finding shared libraries for targets. Step #5: /usr/local/bin/coverage: line 75: 4276 Killed llvm-cov expor -summary-only -instr-profile=$profdata_file -object=$target $shared_libraries $LLVM_COV_COMMON_ARGS > $FUZZER_STATS_DIR/$target.json Step #5: /usr/local/bin/coverage: line 75: 5450 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file Step #5: [2023-05-08 11:57:40,282 INFO] Finding shared libraries for targets (if any). Step #5: [2023-05-08 11:57:40,323 INFO] Finished finding shared libraries for targets. Step #5: error: end_to_end_fuzz: Failed to load coverage: No such file or directory Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! ``` I assume this is because the fuzzers take up lots of the memory. A Tensorflow fuzzer can be ~3GB and there are ~50 fuzzers in Tensorflow. I could image it happens for more projects with many fuzzers of large size?
jonathanmetzman
pushed a commit
that referenced
this pull request
Jun 13, 2023
When running fuzz targets we check for validity by checking if `LLVMFuzzerTestOneInput` exists in the target file: https://github.com/google/oss-fuzz/blob/8d1f1306fda3464fb3a7ec8b4227308d315ed495/infra/base-images/base-runner/coverage#L307-L314 However, this is not done in the post processing step of the coverage utility: https://github.com/google/oss-fuzz/blob/8d1f1306fda3464fb3a7ec8b4227308d315ed495/infra/base-images/base-runner/coverage#L415-L418 This causes coverage build issues e.g. https://oss-fuzz-build-logs.storage.googleapis.com/log-b8d4899d-ecc3-498c-8485-2e88d162dc57.txt ``` Step #5: [INFO] Loading execution data file /workspace/out/libfuzzer-coverage-x86_64/dumps/OpenSSHConfigFuzzer.exec. Step #5: [INFO] Analyzing 3 classes. Step #5: [INFO] Loading execution data file /workspace/out/libfuzzer-coverage-x86_64/dumps/OpenSSHConfigFuzzer.exec. Step #5: [INFO] Writing execution data to /workspace/out/libfuzzer-coverage-x86_64/dumps/jacoco.merged.exec. Step #5: cp: cannot stat '/workspace/out/libfuzzer-coverage-x86_64/dumps/jsch-fuzzer-0.2.10-SNAPSHOT.jar_classes/*': No such file or directory Step #5: ******************************************************************************** ``` Signed-off-by: David Korczynski <david@adalogics.com>
DavidKorczynski
added a commit
that referenced
this pull request
Sep 28, 2023
The current number of parallel fuzzers running is set to the number of available CPUs. This is causing issues in Tensorflow: ``` Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! Step #5: /usr/local/bin/coverage: line 75: 4501 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file ... Step #5: error: decode_compressed_fuzz: Failed to load coverage: No such file or directory Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! Step #5: /usr/local/bin/coverage: line 75: 4873 Killed lvm-cov show -instr-profile=$profdata_file -object=$target -line-coverage-gt=0 $shared_libraries $BRANCH_COV_ARGS $LL VM_COV_COMMON_ARGS > ${TEXTCOV_REPORT_DIR}/$target.covreport Step #5: /usr/local/bin/coverage: line 75: 4897 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file ... Step #5: error: saved_model_fuzz: Failed to load coverage: No such file or directory Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! Step #5: /usr/local/bin/coverage: line 75: 4638 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file Step #5: [2023-05-08 11:57:05,246 INFO] Finding shared libraries for targets (if any). ... Step #5: [2023-05-08 11:57:09,911 INFO] Finished finding shared libraries for targets. Step #5: /usr/local/bin/coverage: line 75: 4276 Killed llvm-cov expor -summary-only -instr-profile=$profdata_file -object=$target $shared_libraries $LLVM_COV_COMMON_ARGS > $FUZZER_STATS_DIR/$target.json Step #5: /usr/local/bin/coverage: line 75: 5450 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file Step #5: [2023-05-08 11:57:40,282 INFO] Finding shared libraries for targets (if any). Step #5: [2023-05-08 11:57:40,323 INFO] Finished finding shared libraries for targets. Step #5: error: end_to_end_fuzz: Failed to load coverage: No such file or directory Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! ``` I assume this is because the fuzzers take up lots of the memory. A Tensorflow fuzzer can be ~3GB and there are ~50 fuzzers in Tensorflow. caps max processes at 10 Signed-off-by: David Korczynski <david@adalogics.com>
afq984
added a commit
to afq984/oss-fuzz
that referenced
this pull request
Oct 15, 2023
The coverage builder complains that rust stdlib sources is missing. ``` Step google#5: �[0;31merror: /workspace/out/libfuzzer-coverage-x86_64/rustc/187b8131d4f760f856b214fce34534903276f2ef/library/core/src/panic.rs: No such file or directory Step google#5: �[0m�[0;31mwarning: The file '/rustc/187b8131d4f760f856b214fce34534903276f2ef/library/core/src/panic.rs' isn't covered. Step google#5: �[0m[2023-10-04 06:50:42,764 DEBUG] Finished generating per-file code coverage summary. Step google#5: [2023-10-04 06:50:42,764 DEBUG] Generating file view html index file as: "/workspace/out/libfuzzer-coverage-x86_64/report/linux/file_view_index.html". Step google#5: Traceback (most recent call last): Step google#5: File "/opt/code_coverage/coverage_utils.py", line 829, in <module> Step google#5: sys.exit(Main()) Step google#5: File "/opt/code_coverage/coverage_utils.py", line 823, in Main Step google#5: return _CmdPostProcess(args) Step google#5: File "/opt/code_coverage/coverage_utils.py", line 780, in _CmdPostProcess Step google#5: processor.PrepareHtmlReport() Step google#5: File "/opt/code_coverage/coverage_utils.py", line 577, in PrepareHtmlReport Step google#5: self.GenerateFileViewHtmlIndexFile(per_file_coverage_summary, Step google#5: File "/opt/code_coverage/coverage_utils.py", line 450, in GenerateFileViewHtmlIndexFile Step google#5: self.GetCoverageHtmlReportPathForFile(file_path), Step google#5: File "/opt/code_coverage/coverage_utils.py", line 422, in GetCoverageHtmlReportPathForFile Step google#5: assert os.path.isfile( Step google#5: AssertionError: "/rustc/187b8131d4f760f856b214fce34534903276f2ef/library/core/src/panic.rs" is not a file. ``` I think the sources would be copied in base-builder if we declare cras as a rust project: https://github.com/google/oss-fuzz/blob/d514fac92686c656633aa8549fd6f239c964b2bc/infra/base-images/base-builder/compile#L226-L231 Fixes: https://crbug.com/oss-fuzz/62974
DavidKorczynski
pushed a commit
that referenced
this pull request
Oct 15, 2023
The coverage builder complains that rust stdlib sources is missing. ``` Step #5: �[0;31merror: /workspace/out/libfuzzer-coverage-x86_64/rustc/187b8131d4f760f856b214fce34534903276f2ef/library/core/src/panic.rs: No such file or directory Step #5: �[0m�[0;31mwarning: The file '/rustc/187b8131d4f760f856b214fce34534903276f2ef/library/core/src/panic.rs' isn't covered. Step #5: �[0m[2023-10-04 06:50:42,764 DEBUG] Finished generating per-file code coverage summary. Step #5: [2023-10-04 06:50:42,764 DEBUG] Generating file view html index file as: "/workspace/out/libfuzzer-coverage-x86_64/report/linux/file_view_index.html". Step #5: Traceback (most recent call last): Step #5: File "/opt/code_coverage/coverage_utils.py", line 829, in <module> Step #5: sys.exit(Main()) Step #5: File "/opt/code_coverage/coverage_utils.py", line 823, in Main Step #5: return _CmdPostProcess(args) Step #5: File "/opt/code_coverage/coverage_utils.py", line 780, in _CmdPostProcess Step #5: processor.PrepareHtmlReport() Step #5: File "/opt/code_coverage/coverage_utils.py", line 577, in PrepareHtmlReport Step #5: self.GenerateFileViewHtmlIndexFile(per_file_coverage_summary, Step #5: File "/opt/code_coverage/coverage_utils.py", line 450, in GenerateFileViewHtmlIndexFile Step #5: self.GetCoverageHtmlReportPathForFile(file_path), Step #5: File "/opt/code_coverage/coverage_utils.py", line 422, in GetCoverageHtmlReportPathForFile Step #5: assert os.path.isfile( Step #5: AssertionError: "/rustc/187b8131d4f760f856b214fce34534903276f2ef/library/core/src/panic.rs" is not a file. ``` I think the sources would be copied in base-builder if we declare cras as a rust project: https://github.com/google/oss-fuzz/blob/d514fac92686c656633aa8549fd6f239c964b2bc/infra/base-images/base-builder/compile#L226-L231 Fixes: https://crbug.com/oss-fuzz/62974
oliverchang
added a commit
that referenced
this pull request
Nov 23, 2023
… count (#10277) The current number of parallel fuzzers running is set to the number of available CPUs. This is causing issues in Tensorflow: ``` Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! Step #5: /usr/local/bin/coverage: line 75: 4501 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file ... Step #5: error: decode_compressed_fuzz: Failed to load coverage: No such file or directory Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! Step #5: /usr/local/bin/coverage: line 75: 4873 Killed lvm-cov show -instr-profile=$profdata_file -object=$target -line-coverage-gt=0 $shared_libraries $BRANCH_COV_ARGS $LL VM_COV_COMMON_ARGS > ${TEXTCOV_REPORT_DIR}/$target.covreport Step #5: /usr/local/bin/coverage: line 75: 4897 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file ... Step #5: error: saved_model_fuzz: Failed to load coverage: No such file or directory Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! Step #5: /usr/local/bin/coverage: line 75: 4638 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file Step #5: [2023-05-08 11:57:05,246 INFO] Finding shared libraries for targets (if any). ... Step #5: [2023-05-08 11:57:09,911 INFO] Finished finding shared libraries for targets. Step #5: /usr/local/bin/coverage: line 75: 4276 Killed llvm-cov expor -summary-only -instr-profile=$profdata_file -object=$target $shared_libraries $LLVM_COV_COMMON_ARGS > $FUZZER_STATS_DIR/$target.json Step #5: /usr/local/bin/coverage: line 75: 5450 Killed llvm-profdata merge -j=1 -sparse $profraw_file_mask -o $profdata_file Step #5: [2023-05-08 11:57:40,282 INFO] Finding shared libraries for targets (if any). Step #5: [2023-05-08 11:57:40,323 INFO] Finished finding shared libraries for targets. Step #5: error: end_to_end_fuzz: Failed to load coverage: No such file or directory Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! ``` [log](https://oss-fuzz-build-logs.storage.googleapis.com/log-050f4040-5009-4a23-81c4-9093922b4ffb.txt) (don't open in a browser but `wget`/`curl` it, as it's quite a large file and will probably annoy the browser). I assume this is because the fuzzers take up lots of the memory. A Tensorflow fuzzer can be ~3GB and there are ~50 fuzzers in Tensorflow, so I think the artifacts read by `llvm-profdata merge` will eat up memory, which consequently starts to crash processes on the system. I could imagine this happens for more projects with many fuzzers of large size? Signed-off-by: David Korczynski <david@adalogics.com> Co-authored-by: Oliver Chang <oliverchang@users.noreply.github.com>
DavidKorczynski
pushed a commit
that referenced
this pull request
Jan 24, 2024
Base PR apache/brpc#2420 ; NOTE: I can't enable memory sanitizer due to ```log BAD BUILD: /tmp/not-out/tmpmptlk01q/fuzz_esp seems to have either startup crash or exit: /tmp/not-out/tmpmptlk01q/fuzz_esp -rss_limit_mb=2560 -timeout=25 -seed=1337 -runs=4 < /dev/null Uninitialized bytes in MemcmpInterceptorCommon at offset 15 inside [0x7030000000f0, 19) ==428==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x682b90 in __interceptor_memcmp /src/llvm-project/compiler-rt/lib/msan/../sanitizer_common/sanitizer_common_interceptors.inc:892:10 #1 0x7fa8ef4cf62a in google::protobuf::SimpleDescriptorDatabase::DescriptorIndex<std::pair<void const*, int> >::FindLastLessOrEqual(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x15062a) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) #2 0x7fa8ef4d259f in google::protobuf::SimpleDescriptorDatabase::DescriptorIndex<std::pair<void const*, int> >::AddSymbol(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::pair<void const*, int>) (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x15359f) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) #3 0x7fa8ef4d2a15 in google::protobuf::SimpleDescriptorDatabase::DescriptorIndex<std::pair<void const*, int> >::AddFile(google::protobuf::FileDescriptorProto const&, std::pair<void const*, int>) (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x153a15) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) #4 0x7fa8ef4cebef in google::protobuf::EncodedDescriptorDatabase::Add(void const*, int) (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x14fbef) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) #5 0x7fa8ef499f43 in google::protobuf::DescriptorPool::InternalAddGeneratedFile(void const*, int) (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x11af43) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) #6 0x7fa8ef49281d in protobuf_google_2fprotobuf_2fapi_2eproto::AddDescriptorsImpl() (/tmp/not-out/tmpmptlk01q/lib/libprotobuf.so.17+0x11381d) (BuildId: 64affeb0f489ae4bcea211ed99e1eca15ff97d68) ``` Signed-off-by: Arjun Singh <ajsinghyadav00@gmail.com>
tomtau
added a commit
to tomtau/oss-fuzz
that referenced
this pull request
Mar 1, 2024
this should hopefully resolve the recent build issue: ``` Step google#5: warning: /workspace/out/libfuzzer-coverage-x86_64/dumps/parser.14845295624977050394_0.profraw: unsupported instrumentation profile format version Step google#5: error: no profile can be merged Step google#5: [2024-02-28 06:09:14,766 INFO] Finding shared libraries for targets (if any). Step google#5: [2024-02-28 06:09:14,775 INFO] Finished finding shared libraries for targets. Step google#5: error: parser: Failed to load coverage: No such file or directory Step google#5: error: Could not load coverage information Step google#5: error: No such file or directory: Could not read profile data! Step google#5: Traceback (most recent call last): Step google#5: File "/usr/local/bin/profraw_update.py", line 129, in <module> Step google#5: sys.exit(main()) Step google#5: File "/usr/local/bin/profraw_update.py", line 120, in main Step google#5: profraw_latest = upgrade(profraw_base, sect_prf_cnts, sect_prf_data) Step google#5: File "/usr/local/bin/profraw_update.py", line 87, in upgrade Step google#5: relativize_address(data, offset + 16, dataref, sect_prf_cnts, sect_prf_data) Step google#5: File "/usr/local/bin/profraw_update.py", line 35, in relativize_address Step google#5: value = struct.unpack('Q', data[offset:offset + 8])[0] Step google#5: struct.error: unpack requires a buffer of 8 bytes ```
DavidKorczynski
pushed a commit
that referenced
this pull request
Mar 1, 2024
this should hopefully resolve the recent build issue: ``` Step #5: warning: /workspace/out/libfuzzer-coverage-x86_64/dumps/parser.14845295624977050394_0.profraw: unsupported instrumentation profile format version Step #5: error: no profile can be merged Step #5: [2024-02-28 06:09:14,766 INFO] Finding shared libraries for targets (if any). Step #5: [2024-02-28 06:09:14,775 INFO] Finished finding shared libraries for targets. Step #5: error: parser: Failed to load coverage: No such file or directory Step #5: error: Could not load coverage information Step #5: error: No such file or directory: Could not read profile data! Step #5: Traceback (most recent call last): Step #5: File "/usr/local/bin/profraw_update.py", line 129, in <module> Step #5: sys.exit(main()) Step #5: File "/usr/local/bin/profraw_update.py", line 120, in main Step #5: profraw_latest = upgrade(profraw_base, sect_prf_cnts, sect_prf_data) Step #5: File "/usr/local/bin/profraw_update.py", line 87, in upgrade Step #5: relativize_address(data, offset + 16, dataref, sect_prf_cnts, sect_prf_data) Step #5: File "/usr/local/bin/profraw_update.py", line 35, in relativize_address Step #5: value = struct.unpack('Q', data[offset:offset + 8])[0] Step #5: struct.error: unpack requires a buffer of 8 bytes ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.