Assertion `idx < elf->header.section.count' in elf_get_section_header

[yonghong-song]

In Meta, we used Google's PGO patch
https://patchwork.kernel.org/project/linux-kbuild/patch/20210407211704.367039-1-morbo@google.com/
to build the kernel for performance reason.

Recently when we enabled amdgpu, we found that build failed with the following error message:

  INSTALL /root/rpmbuild/BUILDROOT/kernel-6.13.1-0_fbk0_lto_rc5_166_gb4a9a3553c165.x86_64/lib/modules/6.13.1-0_fbk0_lto_rc5_166_gb4a9a3553c165/kernel/net/9p/9pnet_virtio.ko
  DEPMOD  /root/rpmbuild/BUILDROOT/kernel-6.13.1-0_fbk0_lto_rc5_166_gb4a9a3553c165.x86_64/lib/modules/6.13.1-0_fbk0_lto_rc5_166_gb4a9a3553c165
depmod: libkmod/libkmod-elf.c:214: elf_get_section_header: Assertion `idx < elf->header.section.count' failed.
make[2]: *** [scripts/Makefile.modinst:132: depmod] Aborted (core dumped)
make[1]: *** [/rw/kernel/Makefile:1904: modules_install] Error 2
make: *** [Makefile:251: __sub-make] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.3Q0MIr (%install)

The depmod version,

$ depmod --version
kmod version 28
+ZSTD +XZ +ZLIB +LIBCRYPTO -EXPERIMENTAL

Look at amdgpu.ko number of sections:

$ llvm-readelf -S amdgpu.ko
...
  [79357] .group         GROUP           0000000000000000 2675e628 000010 04     79383 40006  4
  [79358] __llvm_prf_cnts PROGBITS       0000000000000000 c4bd68 0002c0 00 WAG  0   0  8
  [79359] __llvm_prf_data PROGBITS       0000000000000000 c4c028 000040 00 WAG  0   0  8
  [79360] .rela__llvm_prf_data RELA      0000000000000000 2675e638 000018 18  IG 79383 79359  8
  [79361] .group         GROUP           0000000000000000 2675e650 000010 04     79383 40007  4
  [79362] __llvm_prf_cnts PROGBITS       0000000000000000 c4c068 000010 00 WAG  0   0  8
  [79363] __llvm_prf_data PROGBITS       0000000000000000 c4c078 000040 00 WAG  0   0  8
  [79364] .rela__llvm_prf_data RELA      0000000000000000 2675e660 000018 18  IG 79383 79363  8
  [79365] .group         GROUP           0000000000000000 2675e678 000010 04     79383 40008  4
  [79366] __llvm_prf_cnts PROGBITS       0000000000000000 c4c0b8 000310 00 WAG  0   0  8
  [79367] __llvm_prf_data PROGBITS       0000000000000000 c4c3c8 000040 00 WAG  0   0  8
  [79368] .rela__llvm_prf_data RELA      0000000000000000 2675e688 000018 18  IG 79383 79367  8
  [79369] .static_call_sites PROGBITS    0000000000000000 c4c408 000200 08  WA  0   0  1
  [79370] .rela.static_call_sites RELA   0000000000000000 2675e6a0 000c00 18   I 79383 79369  8
  [79371] __mcount_loc   PROGBITS        0000000000000000 c4c608 019948 08   A  0   0  8
  [79372] .rela__mcount_loc RELA         0000000000000000 2675f2a0 04cbd8 18   I 79383 79371  8
  [79373] .orc_unwind    PROGBITS        0000000000000000 c65f50 0c7710 06   A  0   0  1
  [79374] .orc_unwind_ip PROGBITS        0000000000000000 d2d660 084f60 04   A  0   0  1
  [79375] .rela.orc_unwind_ip RELA       0000000000000000 267abe78 31dc40 18   I 79383 79374  8
  [79376] .gnu.linkonce.this_module PROGBITS 0000000000000000 db25c0 000540 00  WA  0   0 64
  [79377] .rela.gnu.linkonce.this_module RELA 0000000000000000 26ac9ab8 000030 18   I 79383 79376  8
  [79378] __versions     PROGBITS        0000000000000000 db2b00 00f140 00   A  0   0 16
  [79379] .orc_header    PROGBITS        0000000000000000 dc1c40 000014 00   A  0   0  4
  [79380] .note.Linux    NOTE            0000000000000000 dc1c54 000030 00   A  0   0  4
  [79381] .note.gnu.build-id NOTE        0000000000000000 dc1c84 000024 00   A  0   0  4
  [79382] .note.GNU-stack PROGBITS       0000000000000000 26ac9ae8 000000 00      0   0  1
  [79383] .symtab        SYMTAB          0000000000000000 26ac9ae8 25c0c8 18     79386 93326  8
  [79384] .symtab_shndx  SYMTAB SECTION INDICES 0000000000000000 26d25bb0 064acc 04     79383   0  4
  [79385] .shstrtab      STRTAB          0000000000000000 26d8a67c 083afc 00      0   0  1
  [79386] .strtab        STRTAB          0000000000000000 26e0e178 153eae 00      0   0  1
  [79387] .BTF           PROGBITS        0000000000000000 26f62026 28a054 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  R (retain), l (large), p (processor specific)

It exceeded 65536 sections and I think this is the reason the build failed.

Is there anyway that this issue could be fixed in kmod?

[yonghong-song]

cc @liu-song-6

[stoeckmann]

Hi @yonghong-song,

Your assumption should be correct. I have prepared a pull request which should allow kmod to handle such modules. See here: #280

But looking at the module code in the kernel, I doubt that you will be able to load them.

Can you write down how exactly you managed to create such modules, i.e. which patches have been applied? I have manually prepared a module by using binutils to add enough sections to trigger the use of extended fields. But in order to adjust the kernel code, if necessary, it would be nice to have a system which actually produces such modules on itself and can load them.

[yonghong-song]

Thanks @stoeckmann I tried your patch. Indeed, now kernel build is successful. But load failed as you mentioned in the above. The following is the dmesg when loading,

[  253.364508] amdgpu: Unknown symbol drm_ioctl (err -2)                                                                                                   
[  253.374645] amdgpu: Unknown symbol drm_read (err -2)                                                                                                    
[  253.384604] amdgpu: Unknown symbol drm_poll (err -2)                                                                                                    
[  253.394590] amdgpu: Unknown symbol drm_gem_mmap (err -2)                                                                                                
[  253.405231] amdgpu: Unknown symbol drm_open (err -2)                                                                                                    
[  253.415181] amdgpu: Unknown symbol drm_release (err -2)                                                                                                 
[  253.425650] amdgpu: Unknown symbol drm_show_fdinfo (err -2)                                                                                             
[  253.436838] amdgpu: Unknown symbol __devm_drm_dev_alloc (err -2)                                                                                        
[  253.448883] amdgpu: Unknown symbol drm_dev_register (err -2)                                                                                            
[  253.460218] amdgpu: Unknown symbol drm_format_info (err -2)                                                                                             
[  253.471382] amdgpu: Unknown symbol __drm_err (err -2)                                                                                                   
[  253.481511] amdgpu: Unknown symbol drm_dev_unplug (err -2)
...

The command line

# insmod amdgpu.ko
insmod: ERROR: could not insert module amdgpu.ko: Unknown symbol in module

I guess kernel change is also necessary in order to fix above 'insmod amdgpu.ko' issue.

Our current PGO is based on https://patchwork.kernel.org/project/linux-kbuild/patch/20210407211704.367039-1-morbo@google.com/ but since it is originally based on 5.12 and we have adapted it to newer kernels (e..g 6.13) with some change. I will try to make a reproducible test (based on upstream) later. But I think your synthetic case should also work to fix the kernel issue.

[yonghong-song]

I added something like below in our internal linux kernel.

diff --git a/scripts/module.lds.S b/scripts/module.lds.S
index c2f80f9141d4..050c168e8018 100644
--- a/scripts/module.lds.S
+++ b/scripts/module.lds.S
@@ -63,6 +63,28 @@ SECTIONS {
                MOD_CODETAG_SECTIONS()
        }
 #endif
+
+#ifdef CONFIG_PGO_CLANG
+       __llvm_prf_data : {
+               *(__llvm_prf_data)
+       }
+       .rela__llvm_prf_data : {
+               *(.rela__llvm_prf_data)
+       }
+       __llvm_prf_cnts : {
+               *(__llvm_prf_cnts)
+       }
+       __llvm_prf_names : {
+               *(__llvm_prf_names)
+       }
+       __llvm_prf_vals : {
+               *(__llvm_prf_vals)
+       }
+       __llvm_prf_vnds : {
+               *(__llvm_prf_vnds)
+       }
+#endif
+
        MOD_SEPARATE_CODETAG_SECTIONS()
 }

And duplicated section name like __llvm_prf_cnts is gone and only one entry is left. This resolved the too many sections issue. So this issue is resolved. Thanks for looking at this issue!