Error while loading shared libraries only when using config file

[crazynds]

Setup

I'm running nsjail inside a docker container, using the last version from github repository today (27/04/2024). The container has the necessary privileges.

The steps I'm using to build nsjail in the Dockerfile:

FROM ubuntu:22.04
# Install the necessary libs...
# Download and install NSJail
RUN git clone https://github.com/google/nsjail.git  /nsjail
RUN cd /nsjail && make && mv /nsjail/nsjail /bin && rm -rf -- /nsjail
# End of dockerfile

Problem

When I execute nsjail with the following args all works fine:
nsjail -Mo -R /bin/ -R /lib -R /lib64/ -R /usr/ -R /sbin/ -T /dev -R /dev/urandom -- /usr/bin/pypy --version
The output is as following:

[I][2024-04-27T23:12:02+0000] Mode: STANDALONE_ONCE
[I][2024-04-27T23:12:02+0000] Jail parameters: hostname:'NSJAIL', chroot:'', process:'/usr/bin/pypy', bind:[::]:0, max_conns:0, max_conns_per_ip:0, time_limit:0, personality:0, daemonize:false, clone_newnet:true, clone_newuser:true, clone_newns:true, clone_newpid:true, clone_newipc:true, clone_newuts:true, clone_newcgroup:true, clone_newtime:false, keep_caps:false, disable_no_new_privs:false, max_cpus:0
[I][2024-04-27T23:12:02+0000] Mount: '/' flags:MS_RDONLY type:'tmpfs' options:'' dir:true
[I][2024-04-27T23:12:02+0000] Mount: '/bin/' -> '/bin/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-04-27T23:12:02+0000] Mount: '/lib' -> '/lib' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-04-27T23:12:02+0000] Mount: '/lib64/' -> '/lib64/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-04-27T23:12:02+0000] Mount: '/usr/' -> '/usr/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-04-27T23:12:02+0000] Mount: '/sbin/' -> '/sbin/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-04-27T23:12:02+0000] Mount: '/dev' flags: type:'tmpfs' options:'size=4194304' dir:true
[I][2024-04-27T23:12:02+0000] Mount: '/dev/urandom' -> '/dev/urandom' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:false
[I][2024-04-27T23:12:02+0000] Mount: '/proc' flags:MS_RDONLY type:'proc' options:'' dir:true
[I][2024-04-27T23:12:02+0000] Uid map: inside_uid:0 outside_uid:0 count:1 newuidmap:false
[W][2024-04-27T23:12:02+0000][192] logParams():313 Process will be UID/EUID=0 in the global user namespace, and will have user root-level access to files
[I][2024-04-27T23:12:02+0000] Gid map: inside_gid:0 outside_gid:0 count:1 newgidmap:false
[W][2024-04-27T23:12:02+0000][192] logParams():323 Process will be GID/EGID=0 in the global user namespace, and will have group root-level access to files
[I][2024-04-27T23:12:02+0000] Executing '/usr/bin/pypy' for '[STANDALONE MODE]'
Python 3.10.14 (75b3de9d9035, Apr 21 2024, 10:54:48)
[PyPy 7.3.16 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]
[I][2024-04-27T23:12:02+0000] pid=193 ([STANDALONE MODE]) exited with status: 0, (PIDs left: 0)

But when I create a config file with the same specs this dont work:
nsjail --config pypy-env.conf -- /usr/bin/pypy --version

pypy-env.conf

# Example config for nsjail

name: "pypy3.10-env"

mode: ONCE

mount {
	src: "/bin"
	dst: "/bin"
	is_bind: true
}
mount {
	src: "/lib"
	dst: "/lib"
	is_bind: true
}
mount {
	src: "/lib64"
	dst: "/lib64"
	is_bind: true
}
mount {
	src: "/usr"
	dst: "/usr"
	is_bind: true
}
mount {
	src: "/sbin"
	dst: "/sbin"
	is_bind: true
}
mount {
	dst: "/dev"
	fstype: "tmpfs"
	rw: true
	is_bind: false
}

The output is as following:

[I][2024-04-27T23:13:55+0000] Mode: STANDALONE_ONCE
[I][2024-04-27T23:13:55+0000] Jail parameters: hostname:'NSJAIL', chroot:'', process:'/usr/bin/pypy', bind:[::]:0, max_conns:0, max_conns_per_ip:0, time_limit:600, personality:0, daemonize:false, clone_newnet:true, clone_newuser:true, clone_newns:true, clone_newpid:true, clone_newipc:true, clone_newuts:true, clone_newcgroup:true, clone_newtime:false, keep_caps:false, disable_no_new_privs:false, max_cpus:0
[I][2024-04-27T23:13:55+0000] Mount: '/' flags:MS_RDONLY type:'tmpfs' options:'' dir:true
[I][2024-04-27T23:13:55+0000] Mount: '/bin' -> '/bin' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-04-27T23:13:55+0000] Mount: '/lib' -> '/lib' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-04-27T23:13:55+0000] Mount: '/lib64' -> '/lib64' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-04-27T23:13:55+0000] Mount: '/usr' -> '/usr' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-04-27T23:13:55+0000] Mount: '/sbin' -> '/sbin' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-04-27T23:13:55+0000] Mount: '/dev' flags: type:'tmpfs' options:'' dir:true
[I][2024-04-27T23:13:55+0000] Mount: '/var/nsjail/b.py' -> '/var/nsjail/b.py' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:false
[I][2024-04-27T23:13:55+0000] Uid map: inside_uid:0 outside_uid:0 count:1 newuidmap:false
[W][2024-04-27T23:13:55+0000][196] logParams():313 Process will be UID/EUID=0 in the global user namespace, and will have user root-level access to files
[I][2024-04-27T23:13:55+0000] Gid map: inside_gid:0 outside_gid:0 count:1 newgidmap:false
[W][2024-04-27T23:13:55+0000][196] logParams():323 Process will be GID/EGID=0 in the global user namespace, and will have group root-level access to files
[I][2024-04-27T23:13:55+0000] Executing '/usr/bin/pypy' for '[STANDALONE MODE]'
/usr/bin/pypy: error while loading shared libraries: libpypy3.10-c.so: cannot open shared object file: No such file or directory
[I][2024-04-27T23:13:55+0000] pid=197 ([STANDALONE MODE]) exited with status: 127, (PIDs left: 0)

I notice if I use the same first command but pass in the first arg an empty config file, the error is the same:
nsjail --config empty.conf -Mo -R /bin/ -R /lib -R /lib64/ -R /usr/ -R /sbin/ -T /dev -R /dev/urandom -- /usr/bin/pypy --version

[okunz]

Have you ever managed to resolve this or is this still something you need help with?

[crazynds]

This is still not resolved.
I manage to work arround using the args in the command, but I would prefer to use inside the config file.

[okunz]

Okay, I'll see if I can replicate that in a Docker later this week.

[okunz]

I've build it from source (not via Docker though) and could not verify the issue. The only other difference is that I'm running pypy3 --version.

Have you built from HEAD as well?

[crazynds]

I created a minimal example to reproduce this problem.
I have in the root folder these files:

• Dockerfile
• empty.conf
• start-container

Dockerfile

FROM ubuntu:22.04 as php_base

RUN apt-get update

RUN apt-get install -y \
   nano time wget dos2unix \
   autoconf \
   bison \
   flex \
   gcc \
   g++ \
   libprotobuf-dev \
   libnl-route-3-dev \
   libtool \
   make \
   pkg-config \
   protobuf-compiler \
   python3.11 \
   git \
   && apt-get -y autoremove \
   && apt-get clean 

RUN mkdir /python

# Download and install pypy
RUN wget https://downloads.python.org/pypy/pypy3.10-v7.3.16-linux64.tar.bz2 -P /python
RUN tar -xf /python/pypy3.10-v7.3.16-linux64.tar.bz2 -C /python
RUN mv /python/pypy3.10-v7.3.16-linux64 /python/pypy3.10
RUN rm /python/pypy3.10-v7.3.16-linux64.tar.bz2
RUN cp /python/pypy3.10/bin/* /usr/bin -r
RUN cp /python/pypy3.10/lib/* /usr/lib -r
RUN cp /python/pypy3.10/include/* /usr/include -r


# Download and install NSJail
RUN git clone https://github.com/google/nsjail.git  /nsjail
RUN cd /nsjail && make && mv /nsjail/nsjail /bin && rm -rf -- /nsjail

RUN rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

# Copy nsjail config
RUN mkdir /var/nsjail
RUN chmod 0777 /var/nsjail

COPY empty.conf /var/nsjail/

#Copy Starter bash
COPY start-container /usr/local/bin/start-container

# Set permissions in folders

# This line is needed only for windows, and has no effect in linux.
RUN dos2unix /usr/local/bin/start-container

RUN chmod +x /usr/local/bin/start-container

ENTRYPOINT [ "start-container" ]

empty.conf

start-container

#!/bin/bash

echo -e "!!Starting!!"
pypy3.10 --version

echo -e "\n\n\n!!NSJail no config!!"
nsjail -Mo -R /bin/ -R /lib -R /lib64/ -R /usr/ -R /sbin/ -T /dev -R /dev/urandom -- /usr/bin/pypy --version

echo -e "\n\n\n!!NSJail empty config!!"
nsjail --config /var/nsjail/empty.conf -Mo -R /bin/ -R /lib -R /lib64/ -R /usr/ -R /sbin/ -T /dev -R /dev/urandom -- /usr/bin/pypy --version

Results

The output after building and running this image is as follow:

!!Starting!!
Python 3.10.14 (75b3de9d9035, Apr 21 2024, 10:54:48)
[PyPy 7.3.16 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]



!!NSJail no config!!
[I][2024-06-18T02:40:52+0000] Mode: STANDALONE_ONCE
[I][2024-06-18T02:40:52+0000] Jail parameters: hostname:'NSJAIL', chroot:'', process:'/usr/bin/pypy', bind:[::]:0, max_conns:0, max_conns_per_ip:0, time_limit:0, personality:0, daemonize:false, clone_newnet:true, clone_newuser:true, clone_newns:true, clone_newpid:true, clone_newipc:true, clone_newuts:true, clone_newcgroup:true, clone_newtime:false, keep_caps:false, disable_no_new_privs:false, max_cpus:0
[I][2024-06-18T02:40:52+0000] Mount: '/' flags:MS_RDONLY type:'tmpfs' options:'' dir:true
[I][2024-06-18T02:40:52+0000] Mount: '/bin/' -> '/bin/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-06-18T02:40:52+0000] Mount: '/lib' -> '/lib' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-06-18T02:40:52+0000] Mount: '/lib64/' -> '/lib64/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-06-18T02:40:52+0000] Mount: '/usr/' -> '/usr/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-06-18T02:40:52+0000] Mount: '/sbin/' -> '/sbin/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-06-18T02:40:52+0000] Mount: '/dev' flags: type:'tmpfs' options:'size=4194304' dir:true
[I][2024-06-18T02:40:52+0000] Mount: '/dev/urandom' -> '/dev/urandom' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:false
[I][2024-06-18T02:40:52+0000] Mount: '/proc' flags:MS_RDONLY type:'proc' options:'' dir:true
[I][2024-06-18T02:40:52+0000] Uid map: inside_uid:0 outside_uid:0 count:1 newuidmap:false
[W][2024-06-18T02:40:52+0000][8] logParams():313 Process will be UID/EUID=0 in the global user namespace, and will have user root-level access to files
[I][2024-06-18T02:40:52+0000] Gid map: inside_gid:0 outside_gid:0 count:1 newgidmap:false
[W][2024-06-18T02:40:52+0000][8] logParams():323 Process will be GID/EGID=0 in the global user namespace, and will have group root-level access to files
[I][2024-06-18T02:40:52+0000] Executing '/usr/bin/pypy' for '[STANDALONE MODE]'
Python 3.10.14 (75b3de9d9035, Apr 21 2024, 10:54:48)
[PyPy 7.3.16 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]
[I][2024-06-18T02:40:52+0000] pid=9 ([STANDALONE MODE]) exited with status: 0, (PIDs left: 0)



!!NSJail empty config!!
[I][2024-06-18T02:40:52+0000] Mode: STANDALONE_ONCE
[I][2024-06-18T02:40:52+0000] Jail parameters: hostname:'NSJAIL', chroot:'', process:'/usr/bin/pypy', bind:[::]:0, max_conns:0, max_conns_per_ip:0, time_limit:600, personality:0, daemonize:false, clone_newnet:true, clone_newuser:true, clone_newns:true, clone_newpid:true, clone_newipc:true, clone_newuts:true, clone_newcgroup:true, clone_newtime:false, keep_caps:false, disable_no_new_privs:false, max_cpus:0
[I][2024-06-18T02:40:52+0000] Mount: '/' flags:MS_RDONLY type:'tmpfs' options:'' dir:true
[I][2024-06-18T02:40:52+0000] Mount: '/bin/' -> '/bin/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-06-18T02:40:52+0000] Mount: '/lib' -> '/lib' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-06-18T02:40:52+0000] Mount: '/lib64/' -> '/lib64/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-06-18T02:40:52+0000] Mount: '/usr/' -> '/usr/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-06-18T02:40:52+0000] Mount: '/sbin/' -> '/sbin/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
[I][2024-06-18T02:40:52+0000] Mount: '/dev' flags: type:'tmpfs' options:'size=4194304' dir:true
[I][2024-06-18T02:40:52+0000] Mount: '/dev/urandom' -> '/dev/urandom' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:false
[I][2024-06-18T02:40:52+0000] Uid map: inside_uid:0 outside_uid:0 count:1 newuidmap:false
[W][2024-06-18T02:40:52+0000][10] logParams():313 Process will be UID/EUID=0 in the global user namespace, and will have user root-level access to files
[I][2024-06-18T02:40:52+0000] Gid map: inside_gid:0 outside_gid:0 count:1 newgidmap:false
[W][2024-06-18T02:40:52+0000][10] logParams():323 Process will be GID/EGID=0 in the global user namespace, and will have group root-level access to files
[I][2024-06-18T02:40:52+0000] Executing '/usr/bin/pypy' for '[STANDALONE MODE]'
/usr/bin/pypy: error while loading shared libraries: libpypy3.10-c.so: cannot open shared object file: No such file or directory
[I][2024-06-18T02:40:52+0000] pid=11 ([STANDALONE MODE]) exited with status: 127, (PIDs left: 0)

I'm using the binaries from pypy website and not the ubuntu distribution.

[crazynds]

I dont know if this problem is docker related, but I think if you run with this setup outside the docker the problem should be the same.

[okunz]

Thanks for the very detailed information. I could spend some time today looking at this.

Your problem seems to be that your downloaded pypy has the libpypy3.10-c.so located in /usr/bin. Here the output of the ldd command:

linux-vdso.so.1 (0x00007ffc55ff1000)
libpypy3.10-c.so => /usr/bin/libpypy3.10-c.so (0x00007f73a8db8000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f73a8db3000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f73a8b8a000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f73a8b85000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f73a8b80000)
libffi.so.6 => /usr/bin/../lib/libffi.so.6 (0x00007f73a8800000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f73a8a97000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f73a8a7b000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f73a8a76000)
libncursesw.so.6 => /usr/bin/../lib/libncursesw.so.6 (0x00007f73a8a39000)
libtinfow.so.6 => /usr/bin/../lib/libtinfow.so.6 (0x00007f73a87c4000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f73a8a17000)
/lib64/ld-linux-x86-64.so.2 (0x00007f73ac72a000)

The error message that you get is a bit confusing because you don't get the full path of the library but just the which shared library can't be found. This is despite you having mounted the file path where the shared library is located. However when the linker tries to resolve the shared library it can't find it. That's likely because it's not located in a standard location and you don't have /proc mounted.

Nsjail seems to mount /proc by default if started with cmd line args but doesn't do so when a config file is provided. Two ways to resolve this for you are:

1. Set mount_proc in your config [This would be my recommendation]
2. Define an LD_LIBRARY_PATH envar in your config

The root cause of the problem seems to be that (at least certain) default values of the configuration when started using cmdline args are ignored when using the config file.

[crazynds]

I tried mount the proc in my config file and this realy solve the problem.

Is the behavior of overriding the default value for mounting /proc when passing a configuration file as intended?

[okunz]

Glad to see you're unlocked.

I had a brief chat with @robertswiecki about that and he mentioned there are some default values set. I believe those are the ones you can see in the config.proto. Although I haven't looked in depth into the code if that is really the case. Regardless it's probably something that should be better documented.