Bump GDAL from 3.9 to 3.10, QGIS from 3.38 to 3.40

[weiji14]

Attempt to include fix for memory leak documented at rasterio/rasterio#3250. Should also pull in GDAL 3.10.1 or higher.

Need to update these dependencies:

Bumps rasterio from 1.4.1 to 1.4.3.

• Release notes
• Changelog
• Commits

Bumps rioxarray from 0.17.0 to 0.19.0.

• Release notes
• Changelog
• Commits

Bumps pygmt from 0.12.0 to 0.16.0.

• Release notes
• Changelog
• Commits

Bumps qgis from 3.38.0 to 3.40.3.

• Release notes
• Changelog
• Commits

Bumps pillow from 10.2.0 to 11.3.0.

• Release notes
• Changelog
• Commits

Bumps jupyter-book from 2.0.0a0 to 2.0.0b3.

• Release notes
• Changelog
• Commits

Also switching from jupyter-desktop-server to jupyter-remote-desktop-proxy

Part of #129

[github-actions]

👈 Test this PR on Binder

[weiji14]

Getting this error:

{'solver_problems': ['package rasterio-1.4.3-py311h5394301_0 requires libgdal-core >=3.10.0,<3.11.0a0, but none of the providers can be installed'], 'success': False}

Need to see what other dependency pins relying on libgdal need to be updated.

[weiji14]

Looks like conda-lock timed out after 30min at https://github.com/CryoInTheCloud/hub-image/actions/runs/14849669105/job/41690814160#step:5:12 😅 Will retry again in a couple of days.

[weiji14]

Now hitting into some memory issue with conda-lock:

Locking dependencies for ['linux-64']...
INFO:conda_lock.conda_solver:linux-64 using specs ['python ~=3.11.0', 'jupyterhub-singleuser ~=4.0.1', 'nbgitpuller ~=1.2.0', 'jupyterlab >=4.4,<5', 'jupyter-resource-usage ~=1.0.1', 'jupyter-syncthing-proxy 1.0.3', 'syncthing ~=1.22.1', 'gh-scoped-creds 4.1', 'gh ~=2.32.1', 'jupyter-vscode-proxy ~=0.5', 'code-server ~=4.23', 'ipython ~=8.17', 'ipywidgets ~=8.1.1', 'jupyter-ai ~=2.20', 'jupyter-offlinenotebook ~=0.3.1', 'jupyter-remote-desktop-proxy >=3.0.1', 'jupyter-sshd-proxy ~=0.2', 'jupyterlab-favorites ~=3.2.1', 'jupyterlab-geojson ~=3.4.0', 'jupyterlab-git ~=0.50.0', 'jupyterlab-h5web ~=11.1.0', 'jupyterlab-myst ~=2.4.2', 'jupyterlab_pygments ~=0.3.0', 'jupytergis >=0.5.0', 'jupytext ~=1.17.0', 'nbdime ~=4.0.1', 'sphinx ~=7.3', 'sphinxcontrib-bibtex', 'itables ~=2.1', 'voila ~=0.5', 'cython ~=3.0.10', 'flox ~=0.9.7', 'hypothesis 6.58.0', 'networkx 2.8.8', 'numba ~=0.60.0', 'numpy 1.26.4', 'pandas >=2.0.1,<3.0', 'scikit-image ~=0.24.0', 'scikit-learn >=1.2.2,<2.0.0', 'scipy >=1.10', 'statsmodels 0.13.5', 'sympy ~=1.12', 'xarray >=2025.03.0', 'bokeh ~=3.2.2', 'cartopy ~=0.23.0', 'geemap ~=0.32', 'geoviews ~=1.10.1', 'hvplot ~=0.10.0', 'ipyleaflet ~=0.19.2', 'ipympl ~=0.9.3', 'leafmap ~=0.36', 'lonboard ~=0.9', 'matplotlib ~=3.9.0', 'plotly ~=5.18.0', 'seaborn ~=0.13.2', 'py-xgboost ~=2.1.1 cpu*', 'geopandas >=0.14.4', 'pygmt >=0.15.0', 'rasterio >=1.4.3', 'rioxarray >=0.19.0', 'h5py ~=3.11', 'h5netcdf ~=1.3.0', 'nco ~=5.1.6', 'pooch ~=1.8.0', 'zarr ~=2.18', 'kerchunk ~=0.2', 'awscli ~=2.17', 'boto3', 's3fs >=2023.6.0', 'earthaccess ~=0.10.0', 'pystac-client ~=0.8.3', 'stackstac ~=0.5.1', 'intake 0.6.6', 'intake-esm >=2023.7.7', 'intake-stac 0.4.0', 'intake-xarray 0.6.1', 'gcsfs >=2023.5.0', 'dask >=2023.12.1', 'dask-labextension ~=7.0.0', 'dask-geopandas ~=0.4.1', 'pillow >=11.1.0', 'pytest ~=8.3', 'pytest-cov ~=5.0', 'pep8 ~=1.7', 'flake8 ~=7.1', 'tqdm ~=4.66', 'esmpy ~=8.4.2', 'xmip ~=0.7.1', 'h5coro ~=0.0.6', 'icepyx ~=1.2.0', 'itslive ~=0.3.2', 'is2view ~=0.0.8', 'sliderule ~=4.5', 'snowexsql ~=0.5', 'mystmd >=1.3.26', 'qgis ~=3.40.0', 'pyopencl', 'ocl-icd-system', 'websockify >=0.10', 'git >=2.39', 'pip *']
Failed to parse json, Expecting value: line 1 column 1 (char 0)
Could not lock the environment for platform linux-64
    Command: ['/home/runner/micromamba/envs/conda-lock-env/bin/mamba', 'create', '--prefix', '/tmp/tmpdxoi5tqo/prefix', '--dry-run', '--json', '--override-channels', '--channel', 'conda-forge', '--channel', 'file:///home/runner/work/_temp/tmpko_lg2qu', 'python ~=3.11.0', 'jupyterhub-singleuser ~=4.0.1', 'nbgitpuller ~=1.2.0', 'jupyterlab >=4.4,<5', 'jupyter-resource-usage ~=1.0.1', 'jupyter-syncthing-proxy 1.0.3', 'syncthing ~=1.22.1', 'gh-scoped-creds 4.1', 'gh ~=2.32.1', 'jupyter-vscode-proxy ~=0.5', 'code-server ~=4.23', 'ipython ~=8.17', 'ipywidgets ~=8.1.1', 'jupyter-ai ~=2.20', 'jupyter-offlinenotebook ~=0.3.1', 'jupyter-remote-desktop-proxy >=3.0.1', 'jupyter-sshd-proxy ~=0.2', 'jupyterlab-favorites ~=3.2.1', 'jupyterlab-geojson ~=3.4.0', 'jupyterlab-git ~=0.50.0', 'jupyterlab-h5web ~=11.1.0', 'jupyterlab-myst ~=2.4.2', 'jupyterlab_pygments ~=0.3.0', 'jupytergis >=0.5.0', 'jupytext ~=1.17.0', 'nbdime ~=4.0.1', 'sphinx ~=7.3', 'sphinxcontrib-bibtex', 'itables ~=2.1', 'voila ~=0.5', 'cython ~=3.0.10', 'flox ~=0.9.7', 'hypothesis 6.58.0', 'networkx 2.8.8', 'numba ~=0.60.0', 'numpy 1.26.4', 'pandas >=2.0.1,<3.0', 'scikit-image ~=0.24.0', 'scikit-learn >=1.2.2,<2.0.0', 'scipy >=1.10', 'statsmodels 0.13.5', 'sympy ~=1.12', 'xarray >=2025.03.0', 'bokeh ~=3.2.2', 'cartopy ~=0.23.0', 'geemap ~=0.32', 'geoviews ~=1.10.1', 'hvplot ~=0.10.0', 'ipyleaflet ~=0.19.2', 'ipympl ~=0.9.3', 'leafmap ~=0.36', 'lonboard ~=0.9', 'matplotlib ~=3.9.0', 'plotly ~=5.18.0', 'seaborn ~=0.13.2', 'py-xgboost ~=2.1.1 cpu*', 'geopandas >=0.14.4', 'pygmt >=0.15.0', 'rasterio >=1.4.3', 'rioxarray >=0.19.0', 'h5py ~=3.11', 'h5netcdf ~=1.3.0', 'nco ~=5.1.6', 'pooch ~=1.8.0', 'zarr ~=2.18', 'kerchunk ~=0.2', 'awscli ~=2.17', 'boto3', 's3fs >=2023.6.0', 'earthaccess ~=0.10.0', 'pystac-client ~=0.8.3', 'stackstac ~=0.5.1', 'intake 0.6.6', 'intake-esm >=2023.7.7', 'intake-stac 0.4.0', 'intake-xarray 0.6.1', 'gcsfs >=2023.5.0', 'dask >=2023.12.1', 'dask-labextension ~=7.0.0', 'dask-geopandas ~=0.4.1', 'pillow >=11.1.0', 'pytest ~=8.3', 'pytest-cov ~=5.0', 'pep8 ~=1.7', 'flake8 ~=7.1', 'tqdm ~=4.66', 'esmpy ~=8.4.2', 'xmip ~=0.7.1', 'h5coro ~=0.0.6', 'icepyx ~=1.2.0', 'itslive ~=0.3.2', 'is2view ~=0.0.8', 'sliderule ~=4.5', 'snowexsql ~=0.5', 'mystmd >=1.3.26', 'qgis ~=3.40.0', 'pyopencl', 'ocl-icd-system', 'websockify >=0.10', 'git >=2.39', 'pip *']
    STDERR:
Out of memory allocating 18446744071562067968*4 bytes!

Traceback (most recent call last):
  File "/home/runner/micromamba/envs/conda-lock-env/bin/conda-lock", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/runner/micromamba/envs/conda-lock-env/lib/python3.11/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/conda-lock-env/lib/python3.11/site-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/conda-lock-env/lib/python3.11/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/conda-lock-env/lib/python3.11/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/conda-lock-env/lib/python3.11/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/conda-lock-env/lib/python3.11/site-packages/click/decorators.py", line 34, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/conda-lock-env/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 1494, in lock
    lock_func(filename_template=filename_template)
  File "/home/runner/micromamba/envs/conda-lock-env/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 1202, in run_lock
    make_lock_files(
  File "/home/runner/micromamba/envs/conda-lock-env/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 434, in make_lock_files
    fresh_lock_content = create_lockfile_from_spec(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/conda-lock-env/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 892, in create_lockfile_from_spec
    deps = _solve_for_arch(
           ^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/conda-lock-env/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 796, in _solve_for_arch
    conda_deps = solve_conda(
                 ^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/conda-lock-env/lib/python3.11/site-packages/conda_lock/conda_solver.py", line 120, in solve_conda
    dry_run_install = solve_specs_for_arch(
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/micromamba/envs/conda-lock-env/lib/python3.11/site-packages/conda_lock/conda_solver.py", line 319, in solve_specs_for_arch
    proc.check_returncode()
  File "/home/runner/micromamba/envs/conda-lock-env/lib/python3.11/subprocess.py", line 502, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['/home/runner/micromamba/envs/conda-lock-env/bin/mamba', 'create', '--prefix', '/tmp/tmpdxoi5tqo/prefix', '--dry-run', '--json', '--override-channels', '--channel', 'conda-forge', '--channel', 'file:///home/runner/work/_temp/tmpko_lg2qu', 'python ~=3.11.0', 'jupyterhub-singleuser ~=4.0.1', 'nbgitpuller ~=1.2.0', 'jupyterlab >=4.4,<5', 'jupyter-resource-usage ~=1.0.1', 'jupyter-syncthing-proxy 1.0.3', 'syncthing ~=1.22.1', 'gh-scoped-creds 4.1', 'gh ~=2.32.1', 'jupyter-vscode-proxy ~=0.5', 'code-server ~=4.23', 'ipython ~=8.17', 'ipywidgets ~=8.1.1', 'jupyter-ai ~=2.20', 'jupyter-offlinenotebook ~=0.3.1', 'jupyter-remote-desktop-proxy >=3.0.1', 'jupyter-sshd-proxy ~=0.2', 'jupyterlab-favorites ~=3.2.1', 'jupyterlab-geojson ~=3.4.0', 'jupyterlab-git ~=0.50.0', 'jupyterlab-h5web ~=11.1.0', 'jupyterlab-myst ~=2.4.2', 'jupyterlab_pygments ~=0.3.0', 'jupytergis >=0.5.0', 'jupytext ~=1.17.0', 'nbdime ~=4.0.1', 'sphinx ~=7.3', 'sphinxcontrib-bibtex', 'itables ~=2.1', 'voila ~=0.5', 'cython ~=3.0.10', 'flox ~=0.9.7', 'hypothesis 6.58.0', 'networkx 2.8.8', 'numba ~=0.60.0', 'numpy 1.26.4', 'pandas >=2.0.1,<3.0', 'scikit-image ~=0.24.0', 'scikit-learn >=1.2.2,<2.0.0', 'scipy >=1.10', 'statsmodels 0.13.5', 'sympy ~=1.12', 'xarray >=2025.03.0', 'bokeh ~=3.2.2', 'cartopy ~=0.23.0', 'geemap ~=0.32', 'geoviews ~=1.10.1', 'hvplot ~=0.10.0', 'ipyleaflet ~=0.19.2', 'ipympl ~=0.9.3', 'leafmap ~=0.36', 'lonboard ~=0.9', 'matplotlib ~=3.9.0', 'plotly ~=5.18.0', 'seaborn ~=0.13.2', 'py-xgboost ~=2.1.1 cpu*', 'geopandas >=0.14.4', 'pygmt >=0.15.0', 'rasterio >=1.4.3', 'rioxarray >=0.19.0', 'h5py ~=3.11', 'h5netcdf ~=1.3.0', 'nco ~=5.1.6', 'pooch ~=1.8.0', 'zarr ~=2.18', 'kerchunk ~=0.2', 'awscli ~=2.17', 'boto3', 's3fs >=2023.6.0', 'earthaccess ~=0.10.0', 'pystac-client ~=0.8.3', 'stackstac ~=0.5.1', 'intake 0.6.6', 'intake-esm >=2023.7.7', 'intake-stac 0.4.0', 'intake-xarray 0.6.1', 'gcsfs >=2023.5.0', 'dask >=2023.12.1', 'dask-labextension ~=7.0.0', 'dask-geopandas ~=0.4.1', 'pillow >=11.1.0', 'pytest ~=8.3', 'pytest-cov ~=5.0', 'pep8 ~=1.7', 'flake8 ~=7.1', 'tqdm ~=4.66', 'esmpy ~=8.4.2', 'xmip ~=0.7.1', 'h5coro ~=0.0.6', 'icepyx ~=1.2.0', 'itslive ~=0.3.2', 'is2view ~=0.0.8', 'sliderule ~=4.5', 'snowexsql ~=0.5', 'mystmd >=1.3.26', 'qgis ~=3.40.0', 'pyopencl', 'ocl-icd-system', 'websockify >=0.10', 'git >=2.39', 'pip *']' died with <Signals.SIGABRT: 6>.
Rendering lockfile(s) for linux-64...
WARNING:conda_lock.conda_lock:WARNING: installation of pip dependencies from explicit lockfiles is only supported by the 'conda-lock install' and 'micromamba install' commands. Other tools may silently ignore them. For portability, we recommend using the newer unified lockfile format (i.e. removing the --kind=explicit argument.
 - Install lock using : conda create --name YOURENV --file conda-linux-64.lock

Not sure why it's trying to allocate so many bytes!

Edit: probably because this was run under micromamba v1 instead of v2. Have updated things at #144 and that seems to have fixed it (though other errors to handle now).

[weiji14]

/condalock

https://github.com/CryoInTheCloud/hub-image/actions/runs/17417004754/job/49447650103

[weiji14]

Finally got this to lock after 4 months 😭 Ready for review!

[mfisher87]

Finally got this to lock after 4 months

Oof. Hate to ask... but GDAL 3.11? It's got an awesome new CLI. Maybe for a future PR?

[weiji14]

Hate to ask... but GDAL 3.11? It's got an awesome new CLI. Maybe for a future PR?

Doesn't work with rasterio yet unfortunately - conda-forge/rasterio-feedstock#332

[mfisher87]

D'oh!

[mfisher87]

LGTM!

[weiji14]

CI build failed at https://github.com/CryoInTheCloud/hub-image/actions/runs/17417843669/job/49452284156?pr=137#step:5:4852, so need to figure this out again before merging:

   > [16/22] RUN TIMEFORMAT='time: %3R' bash -c 'time ${MAMBA_EXE} env update -p ${NB_PYTHON_PREFIX} --file "environment.yml" && time ${MAMBA_EXE} clean --all -f -y && ${MAMBA_EXE} list -p ${NB_PYTHON_PREFIX} ':
  9.985 
  8352.2 Out of memory allocating 18446744071562067968*4 bytes!
  8481.8 bash: line 1:     8 Aborted                 (core dumped) ${MAMBA_EXE} env update -p ${NB_PYTHON_PREFIX} --file "environment.yml"
  8481.8 time: 8481.680

probably something to fix upstream in repo2docker? Edit: yeah, only upgraded to mamba=2.1.0 at jupyterhub/repo2docker#1423, probably need to match what we're using here (mamba=2.3.2?)

[tsnow03]

@weiji14 I was thinking about adding some newer versions and packages to our env for the next onboarding on Thursday (Sept 11th). It sounds like you're close on this, but its not a rush if you aren't. Just wanted to find out if you think we will be able to merge this early this week? I don't want to disrupt anything here, so we can pip install the few things I would have updated.

[weiji14]

Hmm, I was gonna wait jupyterhub/repo2docker#1465, but it seems like the CI tests pass now without the error from #137 (comment), so I guess we could just merge this in and see if the docker image builds fine?