Internal server errors in freshly installed Invenio instance

[jobo-prod]

Package version (if known): ghcr.io/inveniosoftware/demo-inveniordm/demo-inveniordm:12.0.10
Helm chart version: 0.8.1

Describe the bug

Not really sure if this is the proper repository for raising this issue, but I used this Helm chart for installation and configuration.

I am trying to setup a test/PoC instance of Invenio (that's why I'm currently using the demo-inveniordm image). So far, I managed to get all the services up and running and connect to S3 storage and Keycloak IdM, by mounting a customized invenio.cfg file. I can login and look around but there are several actions that do not work, but instead just throw internal server errors, such as creating a new record or opening the configuration page of a community.

Here is an exemplary stack trace from the invenio-web service's logs:

[2025-08-01 14:55:46,942] ERROR in app: Exception on /uploads/new [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/local/lib/python3.9/site-packages/flask_login/utils.py", line 290, in decorated_view
    return current_app.ensure_sync(func)(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/invenio_app_rdm/records_ui/views/decorators.py", line 363, in view
    return f(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/invenio_app_rdm/records_ui/views/deposits.py", line 394, in deposit_create
    forms_config=get_form_config(
  File "/usr/local/lib/python3.9/site-packages/invenio_app_rdm/records_ui/views/deposits.py", line 317, in get_form_config
    custom_fields = load_custom_fields()
  File "/usr/local/lib/python3.9/site-packages/invenio_app_rdm/records_ui/views/deposits.py", line 297, in load_custom_fields
    field["props"]["options"] = field_instance.options(g.identity)
  File "/usr/local/lib/python3.9/site-packages/invenio_vocabularies/services/custom_fields/vocabulary.py", line 86, in options
    vocabs = current_service.read_all(
  File "/usr/local/lib/python3.9/site-packages/invenio_vocabularies/services/service.py", line 174, in read_all
    vocabulary_type = VocabularyType.query.filter_by(id=type).one()
  File "/usr/local/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one
    return self._iter().one()
  File "/usr/local/lib64/python3.9/site-packages/sqlalchemy/engine/result.py", line 1522, in one
    return self._only_one_row(
  File "/usr/local/lib64/python3.9/site-packages/sqlalchemy/engine/result.py", line 562, in _only_one_row
    raise exc.NoResultFound(
sqlalchemy.exc.NoResultFound: No row was found when one was required

Looks like an issue with the database, but it has been connected to and initialized without any problems.

Steps to Reproduce

Installed with the following Helm values (we use umbrella charts to deploy stuff, the helm-invenio chart values are located under key invenio):

invenio:
  ingress:
    enabled: true

  invenio:
    hostname: invenio.example.org
    existingSecret: invenio-secret-salts

  web:
    # inject custom CA certificates (required for S3)
    # this effectively mounts a file to /etc/ssl/certs/ca-certificates.crt from a ConfigMap
    podAnnotations:
      trust-manager.example.org/inject-ca-bundle: "true"

    # add S3 parameters and credentials (from OBC) as ENVs
    extraEnvFrom:
      - configMapRef:
          name: invenio-data
      - secretRef:
          name: invenio-data

    extraEnvVars:
      # used in invenio.cfg
      - name: OIDC_CLIENT_SECRET
        valueFrom:
          secretKeyRef:
            name: invenio-oidc
            key: clientSecret

    # mount custom invenio.cfg
    extraVolumes:
      - name: custom-config
        configMap:
          name: invenio-cfg
    extraVolumeMounts:
      - name: custom-config
        subPath: invenio.cfg
        mountPath: /opt/invenio/var/instance/invenio.cfg
        readOnly: true

  # We use S3 instead of shared PVC
  persistence:
    enabled: false

  redis:
    enabled: true
    auth:
      existingSecret: invenio-redis
      existingSecretPasswordKey: password

  rabbitmq:
    enabled: true
    auth:
      existingPasswordSecret: invenio-rabbitmq
      existingSecretPasswordKey: password
      existingErlangSecret: invenio-rabbitmq
      existingSecretErlangKey: erlang-cookie

  # I have no idea what this is for
  flower:
    enabled: true
    existing_secret: true
    secret_name: inveio-flower
    host: ""

  opensearch:
    sysctlImage:
      enabled: false # we have the proper kernel params set on node level by default

  postgresql:
    enabled: false
  postgresqlExternal:
    hostname: invenio-postgres-rw
    port: 5432
    username: app
    database: app
    existingSecret: invenio-postgres-app
    existingSecretPasswordKey: password

# just a cloudnative-pg cluster with pretty default settings
postgres:
  instances: 1
  imageName: ghcr.io/cloudnative-pg/postgresql:17.5
  storage:
    size: 1Gi

After that, initialized the instance by manually running the following commands within one of the invenio-web containers (took them from the chart's init job template and adapted as required):

invenio db init
invenio db create
invenio index init
invenio index queue init purge
invenio files location s3-default "s3://$BUCKET_NAME" --default
invenio roles create admin
invenio access allow superuser-access role admin
invenio rdm-records demo

Then, logged in as normal user via Keycloak

Expected behavior

Invenio should be working without problems

Additional context

The custom invenio.cfg file (most of it is just copied from the demo-invenio image, adaptations are mostly at the bottom with authentication and S3):

    import os

    # Invenio-App
    # ===========
    # See https://invenio-app.readthedocs.io/en/latest/configuration.html

    APP_DEFAULT_SECURE_HEADERS = {
        'content_security_policy': {
            'default-src': [
                "'self'",
                'data:', # for fonts
                "'unsafe-inline'",  # for inline scripts and styles
                "blob:",            # for pdf preview
                # Add your own policies here (e.g. analytics)
            ],
        },
        'content_security_policy_report_only': False,
        'content_security_policy_report_uri': None,
        'force_file_save': False,
        'force_https': True,
        'force_https_permanent': False,
        'frame_options': 'sameorigin',
        'frame_options_allow_from': None,
        'session_cookie_http_only': True,
        'session_cookie_secure': True,
        'strict_transport_security': True,
        'strict_transport_security_include_subdomains': True,
        'strict_transport_security_max_age': 31556926,  # One year in seconds
        'strict_transport_security_preload': False,
    }


    # Flask-Babel
    # ===========
    # See https://pythonhosted.org/Flask-Babel/#configuration

    # Default locale (language)
    BABEL_DEFAULT_LOCALE = 'en'
    # Default time zone
    BABEL_DEFAULT_TIMEZONE = 'Europe/Zurich'


    # Invenio-I18N
    # ============
    # See https://invenio-i18n.readthedocs.io/en/latest/configuration.html

    # Other supported languages (do not include BABEL_DEFAULT_LOCALE in list).
    # I18N_LANGUAGES = [
    #     ('fr', _('French'))
    # ]


    # Invenio-Theme
    # =============
    # See https://invenio-theme.readthedocs.io/en/latest/configuration.html

    # Frontpage title
    THEME_FRONTPAGE_TITLE = "InvenioRDM test instance"
    # Frontpage template
    THEME_FRONTPAGE_TEMPLATE = 'demo_inveniordm/demo_frontpage.html'
    # Header logo
    THEME_LOGO="images/invenio-rdm.svg"
    # Site tracking code template for matomo analytics
    # Enable the below on the dedicated environment
    # THEME_TRACKINGCODE_TEMPLATE = "demo_inveniordm/trackingcode-prod.html"


    # Invenio-App-RDM
    # ===============
    # See https://invenio-app-rdm.readthedocs.io/en/latest/configuration.html

    # Instance's theme entrypoint file. Path relative to the ``assets/`` folder.
    INSTANCE_THEME_FILE = './less/theme.less'

    # For invenio-vocabularies
    JSONSCHEMAS_HOST = "localhost"

    # admin user password
    RDM_RECORDS_USER_FIXTURE_PASSWORDS = {
        'admin@inveniosoftware.org': '123456'
    }

    # Invenio-Records-Resources
    # =========================
    # See https://github.com/inveniosoftware/invenio-records-resources/blob/master/invenio_records_resources/config.py

    SITE_UI_URL = "https://127.0.0.1"
    SITE_API_URL = "https://127.0.0.1/api"


    # Invenio-RDM-Records
    # ===================
    # See https://github.com/inveniosoftware/invenio-rdm-records/blob/master/invenio_rdm_records/config.py
    DATACITE_ENABLED = True
    DATACITE_USERNAME = ""
    DATACITE_PASSWORD = ""
    DATACITE_PREFIX = "10.1234"
    DATACITE_TEST_MODE = True


    # Custom Fields
    # =============

    from invenio_i18n import lazy_gettext as _
    from invenio_records_resources.services.custom_fields import (
        BaseCF,
        BooleanCF,
        DoubleCF,
        EDTFDateStringCF,
        IntegerCF,
        ISODateStringCF,
        KeywordCF,
        TextCF,
    )
    from invenio_records_resources.services.records.facets import CFTermsFacet
    from invenio_rdm_records.config import RDM_FACETS, RDM_SEARCH
    from invenio_rdm_records.services.schemas.metadata import _valid_url
    from invenio_vocabularies.services.custom_fields import VocabularyCF
    from invenio_vocabularies.services.facets import VocabularyLabels
    from marshmallow_utils.fields.edtfdatestring import EDTFValidator
    from marshmallow import ValidationError

    #
    # Namespaces
    #

    RDM_NAMESPACES = {
        "rdm": "https://inveniordm.docs.cern.ch"
    }

    #
    # Records custom fields
    #

    def _validate_release_year(value):
        """Accept only edtf year values i.e YYYY."""
        _validate = EDTFValidator()
        _value = _validate(value)
        if not _value.isdigit():
            raise ValidationError("Please provide a valid year.", field_name="release_year")

    RDM_CUSTOM_FIELDS = [
        TextCF(
            name="rdm:repository_url",
            field_args={"validate": _valid_url(_("Not a valid URL."))},
        ),
        VocabularyCF(
            name="rdm:development_status",
            vocabulary_id="developmentstatus",
            dump_options=True,
            multiple=False,
        ),
        VocabularyCF(
            name="rdm:programming_languages",
            vocabulary_id="programminglanguages",
            dump_options=True,
            multiple=True
        ),
        EDTFDateStringCF(
            name="rdm:release_year",
            field_args={"validate": _validate_release_year}
        ),
        ISODateStringCF(name="rdm:release_date"),
        TextCF(name="rdm:release_notes"),
        IntegerCF(name="rdm:release_number_of_lines"),
        DoubleCF(name="rdm:release_test_percentage"),
        BooleanCF(name="rdm:lts_release")
    ]

    RDM_CUSTOM_FIELDS_UI = [
        {
            "section": _("Software"),
            "fields": [
                dict(
                    field="rdm:repository_url",
                    ui_widget="Input",
                    props=dict(
                        label="Repository URL",
                        placeholder="https://your.repository.url",
                        icon="linkify",
                        description="URL of the repository where the software is hosted.",
                    )
                ),
                dict(
                    field="rdm:development_status",
                    ui_widget="Dropdown",  # predefined widget
                    props=dict(
                        label="Development Status",
                        placeholder="Concept, WIP, Active...",
                        icon="group",
                        description="Find more at repostatus.org",
                        search=False,
                        multiple=False,
                        clearable=True
                    )
                ),
                dict(
                    field="rdm:programming_languages",
                    ui_widget="AutocompleteDropdown",  # predefined widget
                    props=dict(
                        label="Programming Languages",
                        placeholder="Type a programming language...",
                        icon="language",
                        description="List of programming fields used in this project.",
                        autocompleteFrom="/api/vocabularies/programminglanguages",
                        clearable=True,
                        multiple=True,
                    )
                ),
                dict(
                    field="rdm:release_year",
                    ui_widget="Input",
                    props=dict(
                        label="Release Year",
                        placeholder="Release year (EDTF date of format YYYY).",
                        icon="calendar",
                        description="Format: DATE where DATE is YYYY.",
                    )
                ),
                dict(
                    field="rdm:release_date",
                    ui_widget="Input",
                    props=dict(
                        label="Release Date",
                        placeholder="Release date in ISO format (YYYY-MM-DD).",
                        icon="calendar",
                        description="Format: YYYY-MM-DD.",
                    )
                ),
                dict(
                    field="rdm:release_notes",
                    ui_widget="RichInput",
                    props=dict(
                        label="Release Notes",
                        placeholder="Added new feature X...",
                        icon="book",
                        description="Changes made in the release.",
                    )
                ),
                dict(
                    field="rdm:release_number_of_lines",
                    ui_widget="NumberInput",
                    props=dict(
                        label="Release number of lines",
                        placeholder="Number of lines in code.",
                        icon="calculator",
                        description="Insert the integer number of code lines changed in this release.",
                    )
                ),
                dict(
                    field="rdm:release_test_percentage",
                    ui_widget="NumberInput",
                    props=dict(
                        label="Release test coverage (%)",
                        placeholder="Release test coverage (e.g 95,75).",
                        icon="calculator",
                        description="Format: double number.",
                    )
                ),
                dict(
                    field="rdm:lts_release",
                    ui_widget="BooleanCheckbox",
                    props=dict(
                        label="LTS release",
                        trueLabel="Yes",
                        falseLabel="No",
                        icon="tag",
                        description="Mark if this is a Long Term Support release (LTS).",
                    )
                )
            ]
        }
    ]

    RDM_FACETS = {
        **RDM_FACETS,
        "development_status": {
            "facet": CFTermsFacet(
                field="rdm:development_status.id",
                label=_("Development Status"),
                value_labels=VocabularyLabels("developmentstatus"),
            ),
            "ui": {
                "field": CFTermsFacet.field("rdm:development_status"),
            },
        },
        "programming_language": {
            "facet": CFTermsFacet(
                field="rdm:programming_languages.id",
                label=_("Programming Languages"),
                value_labels=VocabularyLabels("programminglanguages"),
            ),
            "ui": {
                "field": CFTermsFacet.field("rdm:programming_languages"),
            },
        },
    }

    RDM_SEARCH = {
        **RDM_SEARCH,
        "facets": RDM_SEARCH["facets"] + ["programming_language", "development_status"]
    }


    #
    # Communities custom fields
    #

    COMMUNITIES_CUSTOM_FIELDS = [
        TextCF(
            name="rdm:external_url",
            field_args={"validate": _valid_url(_("Not a valid URL."))},
        ),
        TextCF(name="rdm:policy"),
    ]


    COMMUNITIES_CUSTOM_FIELDS_UI = [
        {
            "section": _("Other details"),
            "fields": [
                dict(
                    field="rdm:external_url",
                    ui_widget="Input",
                    props=dict(
                        label="External URL",
                        placeholder="https://your.community.url",
                        icon="linkify",
                        description="External URL of the community...",
                    )
                ),
                dict(
                    field="rdm:policy",
                    ui_widget="RichInput",
                    props=dict(
                        label="Policy",
                        placeholder="For a record to be accepted into this community it must...",
                        icon="pencil",
                        description="Conditions for a record to be accepted into the community",
                    )
                ),
            ]
        }
    ]

    # Authentication
    # ==============

    # Disable local authentication
    ACCOUNTS_LOCAL_LOGIN_ENABLED = False
    SECURITY_REGISTERABLE = False
    SECURITY_RECOVERABLE = False
    SECURITY_CHANGEABLE = False
    USERPROFILES_READ_ONLY = True

    # Redirect directly to auth server as there is only one auth method
    from invenio_oauthclient.views.client import auto_redirect_login
    ACCOUNTS_LOGIN_VIEW_FUNCTION = auto_redirect_login
    OAUTHCLIENT_AUTO_REDIRECT_TO_EXTERNAL_LOGIN = True

    from invenio_oauthclient.contrib.keycloak import KeycloakSettingsHelper
    _keycloak_helper = KeycloakSettingsHelper(
        title="SSO",
        description="OIDC authentication via Keycloak",
        base_url="https://auth.example.org/",
        realm="example"
    )
    OAUTHCLIENT_KEYCLOAK_REALM_URL = _keycloak_helper.realm_url
    OAUTHCLIENT_KEYCLOAK_USER_INFO_URL = _keycloak_helper.user_info_url
    OAUTHCLIENT_KEYCLOAK_VERIFY_EXP = True
    OAUTHCLIENT_KEYCLOAK_VERIFY_AUD = True
    OAUTHCLIENT_KEYCLOAK_AUD = "invenio"

    # without this variable, authentication fails
    OAUTHCLIENT_KEYCLOAK_USER_INFO_FROM_ENDPOINT = False

    OAUTHCLIENT_REMOTE_APPS = {
      "keycloak": _keycloak_helper.remote_app
    }

    KEYCLOAK_APP_CREDENTIALS = {
        "consumer_key": "invenio",
        "consumer_secret": os.environ['OIDC_CLIENT_SECRET'],
    }

    # S3 Storage
    # ==========
    # ENVs added from the Ceph bucket secret/configmap

    _scheme = "https" if os.environ['BUCKET_PORT'] == '443' else "http"
    S3_ENDPOINT_URL = f"{_scheme}://{os.environ['BUCKET_HOST']}:{os.environ['BUCKET_PORT']}/"
    print(f"Configured S3 endpoint {S3_ENDPOINT_URL}")

    S3_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
    S3_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']

    #
    # Invenio-Search
    #

    SEARCH_INDEX_PREFIX = "demo-inveniordm-"

[jobo-prod]

Any thoughts on this? I'm still stuck with the Invenio deployment

[jobo-prod]

Should I better post this in invenio-app-rdm repository or somewhere else?

[lindhe]

Sorry for being slow to respond. I was on vacation during August and I'm only spending a small part of my working time on this project.

I'm guessing you are suffering from #106 with this chart being somewhat hard to use out-of-the-box. Perhaps you have not initialized the database? That's not part of the default config in this Helm chart today, so that needs to be done manually after deploying before the application can work. I'm usually using this script to do that (but there are supposedly other ways): https://github.com/inveniosoftware/demo-inveniordm/blob/d5cea203784face733e39828433e2c251e9f138c/wipe_recreate.sh

I'm trying to be active here on GitHub, but sometimes it's hard to keep up. If you want to chat about it, this is the best place to do so: https://discord.com/channels/692989811736182844/819158359147675648

[jobo-prod]

Thanks for the reply! I'll be on vacation myself for the next three weeks and will get back to this afterwards.

[jobo-prod]

Okay, I used the script you mentioned for re-init, only changed the storage location to s3. It seems to work better now, thanks a lot! So the commands from the chart's init job do not seem to suffice.

We are now starting to explore and test things out. I will come back to this issue in case of a problem, or close it otherwise.

[jobo-prod]

@lindhe it seems like the file upload does not work yet. I have configured S3 storage via

files location s3-default s3://my-bucket --default

but when someone attempts to upload something, it fails with the following error messages in the invenio-web logs:

fs.opener.errors.UnsupportedProtocol: protocol 's3' is not supported

Again, if there is a better place for asking such questions, kindly let me know :)

[lindhe]

What version of Invenio are you on? We recently migrated from PVC to S3 and for that we had to be at v13 (I don't recall why/if v12 didn't work).

And looking through invenio.cfg, these are the only things with S3 that we set (I think):

S3_ENDPOINT_URL=os.environ.get("INVENIO_S3_ENDPOINT_URL", "http://localhost:9000")
S3_ACCESS_KEY_ID=os.environ.get("INVENIO_S3_ACCESS_KEY_ID", "CHANGE_ME")
S3_SECRET_ACCESS_KEY=os.environ.get("INVENIO_S3_SECRET_ACCESS_KEY", "CHANGE_ME")

APP_DEFAULT_SECURE_HEADERS['content_security_policy']['default-src'].append(
    S3_ENDPOINT_URL
)

During the migration, amongst other things, we ran this:

invenio files location create --default s3-default "s3://${OUR_S3_BUCKET_NAME:?}"

And migrated the old storage prefixes in the database (only required if you have existing files in PVC):

import os
from invenio_db import db
from invenio_files_rest.models import FileInstance
OLD_PREFIX = "/opt/invenio/var/instance/data/"
NEW_PREFIX = f"s3://{os.environ['OUR_S3_BUCKET_NAME']}/"
for f in FileInstance.query.filter(FileInstance.uri.startswith(OLD_PREFIX)):
  f.uri = f.uri.replace(OLD_PREFIX, NEW_PREFIX, 1)
db.session.commit()

Otherwise, we would have used the wipe_recreate.sh script which (at least in our case) includes:

# […]
invenio db create

OUR_S3_BUCKET_NAME=${OUR_S3_BUCKET_NAME:-"default"}
S3_LOCATION="s3://${OUR_S3_BUCKET_NAME}"
invenio files location create --default 'default-location' "$S3_LOCATION"

invenio roles create admin
# […]

Our S3-related Helm values (with Helm chart version v0.8.1) are:

invenio:
  extraConfig:
    OUR_S3_BUCKET_NAME: "my-bucket"
    INVENIO_S3_ENDPOINT_URL: "https://s3.example.com"
    INVENIO_S3_REGION_NAME: "us-east-1"
    INVENIO_S3_ACCESS_KEY_ID: foo  # via a secret
    INVENIO_S3_SECRET_ACCESS_KEY: bar  # via a secret

@Samk13 do you recall if the v13 image had some special settings to enable S3 to work?

[Samk13]

A few things to check because your error usually means the base image was built without the S3 extras, so the S3 opener never registers.

Are you running a local base image converted to s3, or the official cookiecutter setup as s3?
If the base image was built or modified manually from local setup, it’s easy to miss the parts that enable the S3 backend.

Here are the key places that must exist in the image you deploy:

You need to verify the following:

1. .invenio storage setting Storage must be S3, not local. Example: file_storage = S3
2. File storage factory Must point to S3
3. Dependencies must include the S3 extra Otherwise the S3 opener never loads. Example:

invenio-app-rdm[opensearch2,s3]~=14.0.0

If any of these are missing, S3 will not initialize and uploads will fail.
You can always double check in the cookicutter how the s3 setup is being done here
and double check with the docs

[jobo-prod]

Thank you and sorry for the late reply. For testing/PoC I intended to use just the basic demo-inveniordm image that is set by default in the chart, because I am not familiar with the dev/build process. Telling from the chart's appVersion this is v12.

If I am not mistaken here, the demo image is not built with S3 support, according to above requirements: [1] / 2: no FILES_REST_STORAGE_FACTORY in invenio.cfg / [3]

I assume this (esp. 3) can't be mitigated without building a custom image, right?

[lindhe]

I believe you are right. I'd be happy to see a version of the demo image that can toggle that via a flag or variable instead of having to rebuild, though. But perhaps there are good reasons to keep it out of the image, I don't know.