bash workload fails with Error relocating ls: statx: symbol not found for Alpine Container

[anjalirai-intel]

Description of the problem

This workload has started failing recently inside Alpine 3.18 Docker container with Error relocating ls: statx: symbol not found however it is still passing inside host system

Issue is consitently reproducible

Observation:
Same bash workloads passes in Alpine host system, The last binary opened inside Alpine container is "/lib/libskarnet.so.2.14" whereas for host it is older version of same library "/lib/libskarnet.so.2.13"

I have attached both host and docker logs
bash-gramine-direct-container.log
bash-gramine-direct-host.log

Steps to reproduce

1. Create a Alpine based docker image. Reference can be taken from https://github.com/jinengandhi-intel/graphene_local_ci/blob/master/ci/dockerfile/alpine3.18.dockerfile
2. Start the docker container with seccomp policy https://github.com/gramineproject/gramine/blob/master/scripts/docker_seccomp_aug_2022.json
3. Build Gramine meson setup build/ --buildtype=release -Ddirect=enabled -Dsgx=enabled --prefix=$ PWD/gramine_install/usr -Dlibc=musl
4. cd CI-Examples/bash
5. We need to modify below changes in manifest templates to support Alpine Distro
   Git Diff: manifest.template

    - loader.env.LD_LIBRARY_PATH = "/lib:{{ arch_libdir }}:/usr{{ arch_libdir }}"
    + loader.env.LD_LIBRARY_PATH = "/usr/local/lib:{{ arch_libdir }}:/usr{{ arch_libdir }}"
    
    - { path = "/lib", uri = "file:{{ gramine.runtimedir() }}" },
    + { path = "/usr/bin", uri="file:/usr/bin" },
    + { path = "/usr/local/lib", uri = "file:{{ gramine.runtimedir(libc='musl') }}" },
    
    - "file:{{ gramine.runtimedir() }}/",
    + "file:{{ gramine.runtimedir(libc='musl') }}/",

6. make SGX=1 ARCH_LIBDIR=/lib
7. gramine-direct ./bash -c "ls"

Expected results

List the contents of files in directory

Actual results

Error relocating ls: statx: symbol not found

Gramine commit hash

0ee48ee

[dimakuv]

Same bash workloads passes in Alpine host system

What is the Alpine version on the host?

---

Looks like the problem is in you using edge repositories: https://github.com/jinengandhi-intel/graphene_local_ci/blob/586d09b14ba40e51f74db86af480f69085d4dc39/ci/dockerfile/alpine3.18.dockerfile#L3-L5

I see that vanilla Alpine 3.18 uses libskarnet.so.2.13 and only this version: https://alpine.pkgs.org/3.18/alpine-main-x86_64/5/

I also see that Alpine 3.19 uses libskarnet.so.2.14: https://alpine.pkgs.org/3.19/alpine-main-x86_64/skalibs-2.14.0.1-r0.apk.html

Finally, the edge repos use the latest version of libskarnet: https://pkgs.alpinelinux.org/contents?file=&path=&name=skalibs&branch=edge&repo=main&arch=x86

Basically, you're not supposed to use the edge repositories in your Dockerfile, otherwise you may have such stability issues (where a newer version of a package/library requires some new symbols like statx). Why do you need these edge repos?

[anjalirai-intel]

Same bash workloads passes in Alpine host system

What is the Alpine version on the host?

Looks like the problem is in you using edge repositories: https://github.com/jinengandhi-intel/graphene_local_ci/blob/586d09b14ba40e51f74db86af480f69085d4dc39/ci/dockerfile/alpine3.18.dockerfile#L3-L5

I see that vanilla Alpine 3.18 uses libskarnet.so.2.13 and only this version: https://alpine.pkgs.org/3.18/alpine-main-x86_64/5/

I also see that Alpine 3.19 uses libskarnet.so.2.14: https://alpine.pkgs.org/3.19/alpine-main-x86_64/skalibs-2.14.0.1-r0.apk.html

Finally, the edge repos use the latest version of libskarnet: https://pkgs.alpinelinux.org/contents?file=&path=&name=skalibs&branch=edge&repo=main&arch=x86

Basically, you're not supposed to use the edge repositories in your Dockerfile, otherwise you may have such stability issues (where a newer version of a package/library requires some new symbols like statx). Why do you need these edge repos?

Host version is also same, Alpine 3.18

I can check whether it is feasible to remove edge repositories because some workloads might have dependencies. I will test and update same here

[anjalirai-intel]

It worked, bash workload is passed and no other workload got impacted