Did you try training A2C agent on Swimmer environment? I was not able to train it! Tested many NN parameters, but was unsuccessful.