The model seems to favor optimizing the left arm first before moving on to the right arm. #28
Description
Hello, we encountered a peculiar issue while training our SEM model. Could you provide some guidance and suggestions?
We separately tracked the loss terms for the left and right arms and observed that the model initially only optimized the left arm's related loss, neglecting the right arm's loss. This does not appear to be a random occurrence, as repeated training runs revealed that checkpoints obtained early in training fit the left arm's movements well, while those obtained later in training fit the right arm's movements well. This does not appear to be a desirable outcome, as it seems to cause parameters initially optimized for the left arm to be gradually undermined by subsequent training. Could this be the cause of the right-hand bias mentioned in the previous issue?