Authentication issue in solidcommunity.net

[VirginiaBalseiro]

The OIDC login flow itself seems to complete successfully, but once redirected back to the client app (we've tested with both SolidOS and Dokieli), any operation that requires authentication fails with a 401 Unauthorized.

And an error message pops up with "Ooops" in SolidOS (UI).

This is not not an issue with one particular account, I have tested with multiple accounts.

[bourgeoa]

Can you try to login with your browser in private mode ?

[VirginiaBalseiro]

Tried both on Chrome and Firefox, private mode, clearing cookies, storage, cache, etc. Same issue. Even tried creating a new account (which seems to succeed) and same issue.

[jeswr]

I suspect the root cause is that internal networking on the server is down - this can be confirmed by running curl solidcommunity.net on the server to see if there is a connection error. This is because the Community Solid Server will perform a fetch on the users WebID during AuthZ operations.

The fix in this case should be to run systemctl restart systemd-networkd. The following script should be able to keep things running.

#!/bin/bash

PING_TARGET="1.1.1.1"
INTERFACE="eth0"
MAX_ATTEMPTS=2
LOG_TAG="network-watchdog"

check_network() {
    if ! ping -c 2 -W 2 -I $INTERFACE $PING_TARGET > /dev/null 2>&1; then
        echo "Ping failed"
        return 1
    fi
    
    if ! ip link show $INTERFACE | grep -q "state UP"; then
        echo "Interface down"
        return 1
    fi
    
    return 0
}

attempt_recovery() {
    logger -t $LOG_TAG "Network outage detected. Attempting recovery..."
    systemctl restart systemd-networkd
    sleep 5
}

# Main execution
if ! check_network; then
    for ((i=1; i<=$MAX_ATTEMPTS; i++)); do
        attempt_recovery
        if check_network; then
            logger -t $LOG_TAG "Network recovered after $i attempts"
            exit 0
        fi
    done
    logger -t $LOG_TAG "Failed to recover network after $MAX_ATTEMPTS attempts"
    exit 1
else
    logger -t $LOG_TAG "Network operational"
    exit 0
fi

The element discussion from last time we fixed networking on the droplet can be found here.

---

@kkuffour the steps I used to reproduce the error reported by @VirginiaBalseiro were to:

• go to https://jeswr.solidcommunity.net/ (I.e. my storage URL)
• Click "Log In" on the SolidOS UI
• Follow the log in flow, and observe the reported error after completing the log in flow

[jeswr]

@kkuffour has found curl works within the droplet so my suspicion of a networking error is incorrect.

[kkuffour]

, these are what the logs show. Best guess is that there appears to be a time lock issue. So either a performance bottleneck or a storage bug.

[kkuffour]

@jeswr I propose restarting the application and monitoring to see if these errors reappear.

[kkuffour]

@bourgeoa @jeswr @VirginiaBalseiro, my theory is that the file system resource locker set to 6000 is timing out, option 1. restart the application and monitor or option 2. change the resource locker to 15000 and restart the application. What do you prefer/advise?

[jeswr]

It should be safe to pm2 restart the server @kkuffour - give that a go. @joachimvh do you have any insight as to what is going on here?

[joachimvh]

The log above seems to indicate that the cleanup of expired files is failing due to the lock timeout, but I can't say what is causing this. It seems to be failing on the folders itself and not individual entries, so perhaps because there are so many entries.

But there is no guarantee that this is related to the initial problem in this issue.

[bourgeoa]

I think your are not on the right track for this issue.
The timeout issue is from the origin. More frequent with the current issue.

Did you try to restart DNS ?

sudo systemctl restart systemd-resolved.service

PS : did you lôok at the wiki

If this fails :

There are other things to look at :

• disk size
• recent updates to Pivot or SolidOS not tested on a test server
  • conflict on npm versions mainly Rdflib
  • pivot build not run

[kkuffour]

This has already been tried "sudo systemctl restart systemd-resolved.service", update-wise, nothing has changed

Possibly it could be this?
{
"@id": "urn:solid-server:default:CookieStorage",
"@type": "WrappedExpiringStorage",
"timeout": 1
}

disk size
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 795M 1.1M 794M 1% /run
/dev/vda1 78G 17G 61G 22% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/vda15 105M 7.5M 97M 8% /boot/efi
/dev/sda 99G 67G 27G 72% /mnt/volume_lon1_01
/dev/sdc 100G 67G 28G 71% /mnt/volume_lon1_04
/dev/sdb 100G 56G 39G 60% /mnt/volume_lon1_03
/dev/loop8 92M 92M 0 100% /snap/lxd/29619
/dev/loop6 64M 64M 0 100% /snap/core20/2582
/dev/loop2 64M 64M 0 100% /snap/core20/2599
/dev/loop5 55M 55M 0 100% /snap/certbot/4737
/dev/loop7 50M 50M 0 100% /snap/snapd/24792
/dev/loop9 67M 67M 0 100% /snap/core24/1055
/dev/loop11 92M 92M 0 100% /snap/lxd/32662
/dev/loop0 51M 51M 0 100% /snap/snapd/25202
/dev/loop1 56M 56M 0 100% /snap/core18/2940
/dev/loop4 67M 67M 0 100% /snap/core24/1151
/dev/loop10 56M 56M 0 100% /snap/core18/2947
tmpfs 795M 0 795M 0% /run/user/1007

[bourgeoa]

Login with https://podpro.dev works without any issue.

We must consider that mashlib is the issue.
Can you display npm version with npm ls
But not only, also dependencies to look at version conflicts.

Did ODI update anything ?

[jeswr]

We must consider that mashlib is the issue.

There are reports about this being an issue in other apps on element
https://matrix.to/#/!VAJLTawStGAyYwzTTY:gitter.im/$5ii4QGHxJXvb5TjKd9Cz0eM1T7Zz9I3YA7H0xZcUfUs?via=gitter.im&via=matrix.org&via=azadehafzar.io

[ewingson]

it is right that podpro.dev logs in correctly.
but saving a file gives a 401, too.

[VirginiaBalseiro]

Just to clarify: there are no visible errors or issues when logging in, the problem is when trying to make any authenticated fetch, it returns a 401.