Stage 3 performance drops at higher resolution (720×1200) despite Stage 1 & 2 improvements

[isht7]

Hello! Thanks for the great work, and thanks for releasing the code.

I tried one experiment where I changed the input image dimensions of the image in the 3 config files (for all the 3 stages):

default values are:
img_h = 480
img_w = 800

I increased these values to a larger image size (1.5x):
img_h = 720
img_w = 1200

Observation:

• After Stage 1: big-image model > small-image model
• After Stage 2: big-image model > small-image model
• After Stage 3: small-image model > big-image model (big-image model also improved, but not as much as small-image model during Stage 3)

This observation seems strange to me. Do you have any idea why this might be happening?

[woodfrog]

Do you have the concrete numbers for the performance? For the first stage, the vector decoder has not been trained, so did you compare the segmentation performance?

A larger input resolution definitely helps in all recognition/detection-related tasks. But it's also natural that the gap between high and low resolutions shrinks as you train everything end-to-end toward convergence (stage 3).

[isht7]

Actually, the bigger image model eventually got better results - it was just taking way more time becasue I had to reduce the batch size significantly to fit in the GPU. I had to reduce the batch size to almost 1 or 2 per GPU (I used a total of 5 GPUs) to fit the larger image features in GPU for all the 3 stages. For the smaller image experiments, I used the default per GPU values that you used for NuScenes (3, 8, 4 for the 3 stages respectively).

From your experience, how important is batch size for the 3 stages for the final performance? One option to simulate larger batch size is to accumulate gradient over multiple steps and then apply. From your experience how much further can I improve the big image model performance if I am able to keep the default NuScenes values that you used?

Regarding your question:

Do you have the concrete numbers for the performance? For the first stage, the vector decoder has not been trained, so did you compare the segmentation performance?

After stage01:
mIoU of stage1 was about ~2% higher (53.1% mIoU for small image model vs 54.6% mIoU for big image model).

After stage2:
mAP for small image model: 41.5
mAP for big image model: 42.33

After stage3:
mAP for small image model: 49.31
mAP for big image model: 50.19

[woodfrog]

Cool. Consider you only use 5 GPUs, a total batch size of 5 or 10 seems too small. The performance should be better with a larger batch size (e.g., with grad accumulation like you mentioned), but the improvement won't be huge.