[BUG] onedir blocks installation of shared libraries for pygit2 gitfs in 3006.0

[DaAwesomeP]

Description
I did not change any configuration between 3005.1 and 3006.0 but suddently salt-master cannot find my public keys.

Setup

fileserver_backend:
  - gitfs

gitfs_provider: pygit2
gitfs_remotes:
  - ssh://git@github.com/myorg/mysalt.git:
    - base: main
    - root: salt
    - pubkey: /root/.ssh/id_salt.pub
    - privkey: /root/.ssh/id_salt
  - https://github.com/saltstack/salt-winrepo-ng.git:
    - all_saltenvs: master
    - mountpoint: salt://win/repo-ng/salt-winrepo-ng

ext_pillar:
  - git:
    - main ssh://git@github.com/myorg/mypillar.git:
      - env: base
      - root: pillar
      - pubkey: /root/.ssh/id_pillar.pub
      - privkey: /root/.ssh/id_pillar

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior
Upgrade from 3005.1 to 3006.0. Run salt-pip install pygit2.

Expected behavior
Should fetch gitfs normally.

Screenshots

[CRITICAL] SSH pubkey (/root/.ssh/id_salt.pub) for gitfs remote 'ssh://git@github.com/myorg/mysalt.git' could not be found, path may be incorrect. Note that it may be necessary to clear git_pillar locks to proceed once this is resolved and the master has been started back up. A warning will be logged if this is the case, with instructions.
[CRITICAL] SSH pubkey (/root/.ssh/id_pillar.pub) for git_pillar remote 'main ssh://git@github.com/myorg/mypillar.git' could not be found, path may be incorrect. Note that it may be necessary to clear git_pillar locks to proceed once this is resolved and the master has been started back up. A warning will be logged if this is the case, with instructions.
[CRITICAL] Failed to load gitfs
[CRITICAL] Failed to load git_pillar
[CRITICAL] Master failed pre flight checks, exiting

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.0
 
Python Version:
        Python: 3.10.11 (main, Apr 14 2023, 05:57:16) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.15.1
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: 1.6.3
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.12.0
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
        relenv: 0.11.2
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-21-cloud-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

Additional context
Running cat and ls on those files show they exist fine. The files have permission 0400/-r-------- and are owned by root:root.

[lkubb]

The Salt master does not not run as root anymore, see the release notes:

https://docs.saltproject.io/en/latest/topics/releases/3006.0.html#linux-packaging-salt-master-salt-user-and-group

[DaAwesomeP]

Ah, well that was half of it. I moved them to a directory owned by root and readable by the new salt user.

Now I am getting an error probably because the salt user doesn't have a home directory it also doesn't have a known_hosts file or a place to create one:

[ERROR   ] Error occurred fetching gitfs remote 'ssh://git@github.com/myorg/mysalt.git': error loading known_hosts:
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 1870, in _fetch
    fetch_results = origin.fetch(**fetch_kwargs)
  File "/opt/saltstack/salt/extras-3.10/pygit2/remote.py", line 146, in fetch
    payload.check_error(err)
  File "/opt/saltstack/salt/extras-3.10/pygit2/callbacks.py", line 98, in check_error
    check_error(error_code)
  File "/opt/saltstack/salt/extras-3.10/pygit2/errors.py", line 65, in check_error
    raise GitError(message)
_pygit2.GitError: error loading known_hosts:

The known_hosts functionality should probably be disabled or a specific path should be chosen for it.

[DaAwesomeP]

I stand corrected, there is a home directory assigned to it at the default /home/salt. I do find this odd since it is a system user. Often (at least on Debian) system users do not have home folders or they are put somewhere else like /var (at least this is what I have found).

EDIT: See #64141

[DaAwesomeP]

Running sudo -u salt ssh github.com and accepting the host does not appear to resolve the issue. I also tried sudo -u salt bash -c "ssh-keyscan github.com >> ~/.ssh/known_hosts" and salt [MASTER_ID] ssh.set_known_host user=salt hostname=github.com.

I verified that the /home/salt/.ssh/known_hosts file existed and contained some contents after running these commands.

[DaAwesomeP]

OK, I found a version inconsistency that may make dealing this this quite horrible:

• Debian 11 stable (Bullseye) includes libgit2 v1.1.0 in the libgit2-1.1 package
• Saltstack uses Python 3.10 in onedir in 3006.0
• pygit2 versions v1.4 through v1.6 support libgit2 v1.1
• pygit2 versions v1.4 through v1.6 only support Python 3.6 through 3.9
• pygit versions v1.7 through v1.8 are the first to support Python 3.10 and onward, but these only support libgit2 v1.3 and onward
• See https://www.pygit2.org/install.html#version-numbers

So it seems you have to pick libgit2 compatibility or Python 3.10 compatibility or build libgit2 from source?

Was gitfs on Debian stable tested as a part of the 3006.0 release? I'm sure this is a very common configuration. If not, how can we be sure this sort of thing is tested going forward? I imagine this is also a mess on RHEL 6 and other older stable platforms.

[DaAwesomeP]

I tried salt-pip install 'pygit2>=1.4,<1.7' but since there is no wheel available for pygit2 v1.6.1 with Python 3.10 it has to build the wheel on install. This effectively requires installing build-essential on the machine (not a solution for production machines) or building the wheel offline.

I also tried salt-pip install 'pygit2>=1.7,<1.9' but this resulted in pip warning of many linked library versions not available on Debian 11, and the known_hosts issue persisted.

I will try building the wheel separately and see if pygit2 v1.6.1 works with Python 3.10, but this is clearly not supported by the pygit2 developers and would be a temporary workaround.

I see why Salt moved to Python 3.10 but I'm sure this is not the only package version inconsistency that will be found on other stable/LTS distros.

[DaAwesomeP]

Another workaround: Since Salstack still support Python 3.9 just fine and Debian 11 packages are built around Python 3.9 as the latest available version, then the Salt packages could use Python 3.9 on Debian 11.

[OrangeDog]

you have to pick libgit2 compatibility or Python 3.10 compatibility or build libgit2 from source?

No, you have to install pygit2 via wheel: salt-pip install --only-binary=:all: pygit2

The system package of libgit2 does not usually work, as it doesn't have SSH support included.

[DaAwesomeP]

@OrangeDog To try that out I removed salt, wiped the Python 3.10 and package folders to start afresh, and reinstalled. Pip installs pygit2 but it complains of missing libssl, libcrypto, libpcre, and libssh2 presumably because the versions that the libgit2 bundled with pygit2 v1.12 relies on are not available in Debian 11.

I have verified that package libssh2-1 is installed which should provide libssh2.so.1.0.1 is installed on the system. It seems in Debian 11 libcrypto.so.3 should really link to libmbedcrypto.so.3 from the libmbedcrypto3 package or just libssl3 is not available on Debian 11. I stopped checking the rest

Given that it can't find libssh2.so.1.0.1, is the issue really that all of these are missing from /opt/saltstack/salt/lib/ and not from the system?

# uninstall 3006
$ rm -rf /opt/saltstack/salt/lib/python3.10 /opt/saltstack/salt/extras-3.10
# install 3006
$ salt-pip install --only-binary=:all: pygit2
Collecting pygit2
  Using cached pygit2-1.12.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.9 MB)
Collecting cffi>=1.9.1
  Using cached cffi-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Installing collected packages: pycparser, cffi, pygit2
  Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/_cffi_backend.cpython-310-x86_64-linux-gnu.so
  Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libcrypto-1b9ded14.so.3
  WARNING: Unable to find library libssl-d3387b0f.so.3 linked from /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libgit2-dc37b70b.so.1.6.3
  WARNING: Unable to find library libcrypto-1b9ded14.so.3 linked from /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libgit2-dc37b70b.so.1.6.3
  WARNING: Unable to find library libpcre-9513aab5.so.1.2.0 linked from /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libgit2-dc37b70b.so.1.6.3
  WARNING: Unable to find library libssh2-3ed487a7.so.1.0.1 linked from /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libgit2-dc37b70b.so.1.6.3
  Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libgit2-dc37b70b.so.1.6.3
  Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libpcre-9513aab5.so.1.2.0
  WARNING: Unable to find library libssl-d3387b0f.so.3 linked from /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libssh2-3ed487a7.so.1.0.1
  WARNING: Unable to find library libcrypto-1b9ded14.so.3 linked from /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libssh2-3ed487a7.so.1.0.1
  Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libssh2-3ed487a7.so.1.0.1
  WARNING: Unable to find library libcrypto-1b9ded14.so.3 linked from /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libssl-d3387b0f.so.3
  Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/pygit2.libs/libssl-d3387b0f.so.3
  WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libgit2-dc37b70b.so.1.6.3 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libssl-d3387b0f.so.3 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libcrypto-1b9ded14.so.3 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libpcre-9513aab5.so.1.2.0 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libssh2-3ed487a7.so.1.0.1 is not in /opt/saltstack/salt
  Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/pygit2/_libgit2.abi3.so
  WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libgit2-dc37b70b.so.1.6.3 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libssl-d3387b0f.so.3 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libcrypto-1b9ded14.so.3 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libpcre-9513aab5.so.1.2.0 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /tmp/pip-target-mvjvbnk5/lib/python/pygit2/../pygit2.libs/libssh2-3ed487a7.so.1.0.1 is not in /opt/saltstack/salt
  Do not adjust rpath of /tmp/pip-target-mvjvbnk5/lib/python/pygit2/_pygit2.cpython-310-x86_64-linux-gnu.so
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
salt 3006.0 requires cffi==1.14.6, but you have cffi 1.15.1 which is incompatible.
Successfully installed cffi-1.15.1 pycparser-2.21 pygit2-1.12.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
$ systemctl restart salt-master.service

Resulting log:

systemd[1]: Started The Salt Master Server.
salt-master[211397]: [ERROR   ] Error occurred fetching gitfs remote 'ssh://git@github.commyorg/mysalt.git': error loading known_hosts:
salt-master[211397]: Traceback (most recent call last):
salt-master[211397]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 1870, in _fetch
salt-master[211397]:     fetch_results = origin.fetch(**fetch_kwargs)
salt-master[211397]:   File "/opt/saltstack/salt/extras-3.10/pygit2/remote.py", line 146, in fetch
salt-master[211397]:     payload.check_error(err)
salt-master[211397]:   File "/opt/saltstack/salt/extras-3.10/pygit2/callbacks.py", line 98, in check_error
salt-master[211397]:     check_error(error_code)
salt-master[211397]:   File "/opt/saltstack/salt/extras-3.10/pygit2/errors.py", line 65, in check_error
salt-master[211397]:     raise GitError(message)
salt-master[211397]: _pygit2.GitError: error loading known_hosts:

[OrangeDog]

It's not looking for libssh2.so.1.0.1, it's looking for libssh2-3ed487a7.so.1.0.1.
It has unpacked that from the .whl into /tmp, but the onedir environment appears to be blocking it from loading during installation.