Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite PDB parser #1517

Merged
merged 9 commits into from
Oct 9, 2021
Merged

Rewrite PDB parser #1517

merged 9 commits into from
Oct 9, 2021

Conversation

Basstorm
Copy link
Member

@Basstorm Basstorm commented Aug 24, 2021

Your checklist for this pull request

  • I've read the guidelines for contributing to this repository
  • I made sure to follow the project's coding style
  • I've added tests that prove my fix is effective or that my feature works (if possible)
  • I've updated the documentation and the rizin book with the relevant information (if needed)

Detailed description

Test plan

CI is green.

Closing issues

Closes #279, partially addresses #297

TODO

  • Remove rz_bin_pdb_gvars_as_cmd_string and just use the proper API instead of creating commands + execute them
  • Remove/Move rz_bin_pdb_types_as_string
  • Remove/Move `rz_bin_pdb_gvars_as_string

test/db/cmd/cmd_idp Outdated Show resolved Hide resolved
test/db/cmd/cmd_idp Outdated Show resolved Hide resolved
test/db/formats/pdb Show resolved Hide resolved
test/db/formats/pdb Outdated Show resolved Hide resolved
librz/bin/pdb/dbi.c Show resolved Hide resolved
librz/bin/pdb/dbi.c Outdated Show resolved Hide resolved
librz/bin/pdb/gdata.c Outdated Show resolved Hide resolved
librz/bin/pdb/gdata.c Outdated Show resolved Hide resolved
librz/core/cpdb.c Outdated Show resolved Hide resolved
librz/core/cpdb.c Outdated Show resolved Hide resolved
@ret2libc ret2libc marked this pull request as draft August 25, 2021 07:01
@XVilka XVilka modified the milestones: 0.3.0, 0.4.0 Aug 25, 2021
@Basstorm Basstorm force-pushed the dist-asan-fuzz-pdb-rewrite branch 3 times, most recently from 3a13542 to 39cf7dd Compare August 25, 2021 08:35
test/db/formats/pdb Outdated Show resolved Hide resolved
librz/core/cpdb.c Outdated Show resolved Hide resolved
@Basstorm Basstorm changed the title [WIP]Rewrite PDB parser Rewrite PDB parser Aug 25, 2021
@XVilka XVilka requested a review from ret2libc September 6, 2021 04:48
@XVilka XVilka marked this pull request as ready for review September 7, 2021 05:46
@Basstorm Basstorm force-pushed the dist-asan-fuzz-pdb-rewrite branch 2 times, most recently from 9d3975a to bcac6ed Compare September 14, 2021 04:27
@XVilka
Copy link
Member

XVilka commented Sep 14, 2021

It makes SystemZ unit tests pass: https://app.travis-ci.com/github/rizinorg/rizin/jobs/537216040#L3041

We have a long road to fix all integration tests though...:

[**]                                         db/io/srec    13391 OK       963 BR      890 XX       26 FX

Copy link
Member

@XVilka XVilka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some small regressions related to the missing classes support in RzType and missing bitfields support in RzType but it's already a huge improvement over the old code. I think we should merge it as is.

@XVilka

This comment has been minimized.

@Basstorm Basstorm force-pushed the dist-asan-fuzz-pdb-rewrite branch 3 times, most recently from 3925afb to 7033456 Compare October 7, 2021 13:29
Copy link
Member

@XVilka XVilka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ret2libc your last comment was addressed, please take a look again, and lets not wait anymore and merge if green.

librz/core/cpdb.c Outdated Show resolved Hide resolved
librz/bin/pdb/pdb.c Outdated Show resolved Hide resolved
librz/bin/pdb/pdb.c Outdated Show resolved Hide resolved
librz/bin/pdb/pdb.c Outdated Show resolved Hide resolved
librz/bin/pdb/pdb.c Outdated Show resolved Hide resolved
librz/bin/pdb/pdb.c Outdated Show resolved Hide resolved
@XVilka
Copy link
Member

XVilka commented Oct 8, 2021

Please fix the WIndows linking error:

   Creating library binrz/rz-hash/rz-hash.lib and object binrz/rz-hash/rz-hash.exp
librz_core.a(windows_heap.c.obj) : error LNK2019: unresolved external symbol rz_bin_pdb_gvars_as_string referenced in function GetHeapGlobalsOffset
binrz/rz-hash/rz-hash.exe : fatal error LNK1120: 1 unresolved externals
[1591/1690] 

@XVilka XVilka requested a review from ret2libc October 8, 2021 18:23
@XVilka
Copy link
Member

XVilka commented Oct 9, 2021

I broke the Travis PPC and ARM tests:

[XX] db/cmd/cmd_idp idpij
RZ_NOPLUGINS=1 rizin -escr.utf8=0 -escr.color=0 -escr.interactive=0 -N -Qc 'idpij bins/pdb/minimal.pdb~{}' =
-- stdout
--- expected
+++ actual
@@ -1,126 +1,0 @@
-[
-  {
...
-        "gdata_name": "_main"
-      }
-    ]
-  }
-]
-- stderr
Warning: Cannot find base address, flags will probably be misplaced
Cannot find base type "struct_typedef"
Cannot find base type "<unnamed-tag>"
Cannot find base type "unnamed_member_types_typedef"
Cannot find base type "bitfield_typedef"
Cannot find base type "union_typedef"
Warning: Cannot find base address, flags will probably be misplaced
double free or corruption (top)
-- exit status: -1

[**]                                  db/cmd/cmd_search    13199 OK       919 BR       24 XX       27 FX

[XX] db/cmd/cmd_idp idp bin state bug (#9441)

RZ_NOPLUGINS=1 rizin -escr.utf8=0 -escr.color=0 -escr.interactive=0 -N -Qc 'e asm.bits
idpi bins/pdb/SimplePDB.pdb~SomeCoolFunction
e asm.bits
' bins/pdb/SimplePDB.exe
-- stdout
--- expected
+++ actual
@@ --1,5 +-1,3 @@
 32
-0x00401000  2  .text  void __cdecl SomeCoolFunction(void)
-32
-- stderr
Cannot find base type "localeinfo_struct"
Cannot find base type "threadlocaleinfostruct"
Cannot find base type "__vc_attributes::threadingAttribute::threading_e"
Cannot find base type "__vc_attributes::threadingAttribute"
Cannot find base type "__vc_attributes::event_receiverAttribute::type_e"
Cannot find base type "__vc_attributes::event_receiverAttribute"
Cannot find base type "__vc_attributes::aggregatableAttribute::type_e"
Cannot find base type "__vc_attributes::aggregatableAttribute"
Cannot find base type "__vc_attributes::event_sourceAttribute::type_e"
Cannot find base type "__vc_attributes::event_sourceAttribute::optimize_e"
Cannot find base type "__vc_attributes::event_sourceAttribute"
Cannot find base type "__vc_attributes::moduleAttribute::type_e"
Cannot find base type "__vc_attributes::moduleAttribute"
Cannot find base type "__vc_attributes::helper_attributes::usageAttribute::usage_e"
Cannot find base type "__vc_attributes::helper_attributes::usageAttribute"
Cannot find base type "__vc_attributes::helper_attributes::v1_alttypeAttribute::type_e"
Cannot find base type "__vc_attributes::helper_attributes::v1_alttypeAttribute"
Cannot find base type "_iobuf"
Cannot find base type "localerefcount"
Cannot find base type "_USER_ACTIVITY_PRESENCE"
Cannot find base type "ReplacesCorHdrNumericDefines"
Cannot find base type "TypeDescriptor"
Cannot find base type "_s_CatchableType"
Cannot find base type "PMD"
Cannot find base type "_s_CatchableTypeArray"
Cannot find base type "_FLOATING_SAVE_AREA"
Cannot find base type "_TP_CALLBACK_ENVIRON_V1"
Cannot find base type "_TP_CALLBACK_ENVIRON_V1::<unnamed-type-u>"
Cannot find base type "_TP_CALLBACK_ENVIRON_V1::<unnamed-type-u>"
Cannot find base type "_TP_CALLBACK_ENVIRON_V1::<unnamed-type-u>::<unnamed-type-s>"
Cannot find base type "_TP_CALLBACK_ENVIRON_V1::<unnamed-type-u>::<unnamed-type-s>"
Cannot find base type "_CONTEXT"
Cannot find base type "_s_ThrowInfo"
Cannot find base type "_EXCEPTION_POINTERS"
Cannot find base type "_lldiv_t"
Cannot find base type "EHExceptionRecord"
Cannot find base type "EHExceptionRecord::EHParameters"
Cannot find base type "EHExceptionRecord::EHParameters"
Cannot find base type "_ldiv_t"
Cannot find base type "_TP_CALLBACK_PRIORITY"
Cannot find base type "wrapEncodedKERNEL32Functions"
Cannot find base type "<unnamed-tag>"
Cannot find base type "__enative_startup_state"
Cannot find base type "_IMAGE_DOS_HEADER"
Cannot find base type "_startupinfo"
Cannot find base type "_EXCEPTION_DISPOSITION"
Cannot find base type "_IMAGE_OPTIONAL_HEADER"
Cannot find base type "_IMAGE_FILE_HEADER"
Cannot find base type "_exception"
Cannot find base type "_IMAGE_NT_HEADERS"
Cannot find base type "FT"
Cannot find base type "__m128"
Cannot find base type "_LARGE_INTEGER"
Cannot find base type "__m64"
Cannot find base type "_IMAGE_SECTION_HEADER"
Cannot find base type "IMAGE_LOAD_CONFIG_DIRECTORY32_2"
Cannot find base type "_RTC_ErrorNumber"
double free or corruption (!prev)
-- exit status: -1

@XVilka
Copy link
Member

XVilka commented Oct 9, 2021

Two ASAN failed tests are unrelated to this PR and happen on the latest dev as well:

[XX] TIMEOUT db/formats/elf/sections more than 65535 segments
RZ_NOPLUGINS=1 rizin -escr.utf8=0 -escr.color=0 -escr.interactive=0 -N -Qc iSS~? gzip://bins/elf/core/more-than-65535-segments.gz
-- stdout
--- expected
+++ actual
@@ -1,1 +1,0 @@
-131073

-- stderr
Cannot retrieve regstate on: AMD x86-64 architecture (not yet supported)

-- exit status: -1

[XX] TIMEOUT db/formats/mdmp mdmp 64bit - strings
RZ_NOPLUGINS=1 rizin -escr.utf8=0 -escr.color=0 -escr.interactive=0 -N -Qc 'iz~0x002e2fe1[1,2,3,4,5,6,7,8]
iz~0x002e2feb[1,2,3,4,5,6,7,8]
iz~0x002e2ff5[1,2,3,4,5,6,7,8]
iz~0x00330d85[1,2,3,4,5,6,7,8]
iz~0x003310c1[1,2,3,4,5,6,7,8]
iz~0x00161a37[1,2,3,4,5,6,7,8]
iz~0x00161c7f[1,2,3,4,5,6,7,8]
iz~0x00161dbf[1,2,3,4,5,6,7,8]
iz~0x001d9d45[1,2,3,4,5,6,7,8]
iz~0x001da0a5[1,2,3,4,5,6,7,8]
iz~0x003fad45[1,2,3,4,5,6,7,8]
iz~0x003fb0ad[1,2,3,4,5,6,7,8]
iz~0x00480857[1,2,3,4,5,6,7,8]
iz~0x004808c7[1,2,3,4,5,6,7,8]
iz~0x00499cf5[1,2,3,4,5,6,7,8]
iz~0x0049a045[1,2,3,4,5,6,7,8]
' bins/mdmp/hello64.dmp
-- stdout
--- expected
+++ actual
@@ -8,9 +8,3 @@
 0x00161dbf 0x7759e130 45 92 C:_Windows_System32_kernel32.dll utf16le \REGISTRY\USER\*\SOFTWARE\Classes\Wow6432Node
 0x001d9d45 0x776160b6 15 32 C:_Windows_System32_kernel32.dll utf16le VS_VERSION_INFO
 0x001da0a5 0x77616416 11 24 C:_Windows_System32_kernel32.dll utf16le VarFileInfo
-0x003fad45 0x7fefd5080b6 15 32 C:_Windows_System32_KERNELBASE.dll utf16le VS_VERSION_INFO
-0x003fb0ad 0x7fefd50841e 11 24 C:_Windows_System32_KERNELBASE.dll utf16le VarFileInfo
-0x00480857 0x7fefdb33bc8 4 10 C:_Windows_System32_msvcrt.dll utf16le PATH
-0x004808c7 0x7fefdb33c38 10 22 C:_Windows_System32_msvcrt.dll utf16le SystemRoot
-0x00499cf5 0x7fefdb4d066 15 32 C:_Windows_System32_msvcrt.dll utf16le VS_VERSION_INFO
-0x0049a045 0x7fefdb4d3b6 11 24 C:_Windows_System32_msvcrt.dll utf16le VarFileInfo

-- stderr
[WARN] Invalid or unsupported enumeration encountered 21
[WARN] Invalid or unsupported enumeration encountered 22
[INFO] Parsing data sections for large dumps can take time, please be patient (but if strings ain't your thing try with -z)!

-- exit status: -1

@XVilka XVilka merged commit 19a6a92 into dev Oct 9, 2021
@XVilka XVilka deleted the dist-asan-fuzz-pdb-rewrite branch October 9, 2021 08:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Failing unit tests on System Z (S390) platform
6 participants