resnet50-res512-all predictions are wrong and 2 classes are missing

[PauleNajus]

Hello,

When I try classification with “densenet121-res224-all” “16747_3_1.jpg” and “00000001_000.png” (standard test images) I get identical results that you got. But when I try switching to resnet50-res512-all with those same test images, 2 classes are missing, and other classes results are very conservative and far from “densenet121-res224-all”.

I was wondering if you could share what results have you got with “resnet50-res512-all” “16747_3_1.jpg” and “00000001_000.png” test images for reference?

Results I got with “resnet50-res512-all” - “16747_3_1.jpg”:
Lung Opacity 0.785
Lung Lesion 0.500
Enlarged Cardiomediastinum 0.500
Infiltration 0.330
Pneumonia 0.191
Nodule 0.174
Fracture 0.052
Mass 0.044
Consolidation 0.039
Atelectasis 0.024
Cardiomegaly 0.015
Pleural_Thickening 0.014
Effusion 0.013
Pneumothorax 0.011
Emphysema 0.009
Edema 0.004
Fibrosis 0.002
Hernia 0.001

Results I got with “resnet50-res512-all” - “00000001_000.png”:
Lung Lesion 0.500
Enlarged Cardiomediastinum 0.500
Cardiomegaly 0.448
Effusion 0.148
Infiltration 0.125
Atelectasis 0.057
Fracture 0.036
Pleural_Thickening 0.032
Nodule 0.029
Fibrosis 0.017
Mass 0.012
Consolidation 0.010
Lung Opacity 0.008
Pneumonia 0.007
Edema 0.005
Emphysema 0.004
Hernia 0.003
Pneumothorax 0.003

[ieee8023]

predictions are wrong

What does wrong mean here?

I was wondering if you could share what results have you got with “resnet50-res512-all” “16747_3_1.jpg” and “00000001_000.png” test images for reference?

Sure, using the process_image.py in the scripts folder I get this:

$ python process_image.py ../tests/00000001_000.png 
Warning: Input size (512x512) is not the native resolution (224x224) for this model. A resize will be performed but this could impact performance.
{'preds': {'Atelectasis': 0.5064204,
           'Cardiomegaly': 0.6068723,
           'Consolidation': 0.32152757,
           'Edema': 0.20760165,
           'Effusion': 0.40498933,
           'Emphysema': 0.5032768,
           'Enlarged Cardiomediastinum': 0.46466783,
           'Fibrosis': 0.5410498,
           'Fracture': 0.29533696,
           'Hernia': 0.013291723,
           'Infiltration': 0.5222817,
           'Lung Lesion': 0.22291252,
           'Lung Opacity': 0.43178767,
           'Mass': 0.42115793,
           'Nodule': 0.50882816,
           'Pleural_Thickening': 0.510605,
           'Pneumonia': 0.13716412,
           'Pneumothorax': 0.33823627}}

$ python process_image.py ../tests/00000001_000.png -weights resnet50-res512-all
{'preds': {'Atelectasis': 0.056815326,
           'Cardiomegaly': 0.44829446,
           'Consolidation': 0.009982704,
           'Edema': 0.0050414177,
           'Effusion': 0.1477132,
           'Emphysema': 0.0041440246,
           'Enlarged Cardiomediastinum': 0.5,
           'Fibrosis': 0.017102875,
           'Fracture': 0.035871908,
           'Hernia': 0.002605671,
           'Infiltration': 0.12528789,
           'Lung Lesion': 0.5,
           'Lung Opacity': 0.008149157,
           'Mass': 0.0124504855,
           'Nodule': 0.028969117,
           'Pleural_Thickening': 0.032002825,
           'Pneumonia': 0.006612198,
           'Pneumothorax': 0.002571446}}


$ python process_image.py ../tests/16747_3_1.jpg 
Warning: Input size (885x885) is not the native resolution (224x224) for this model. A resize will be performed but this could impact performance.
{'preds': {'Atelectasis': 0.33015457,
           'Cardiomegaly': 0.375611,
           'Consolidation': 0.4268645,
           'Edema': 0.019578112,
           'Effusion': 0.28957728,
           'Emphysema': 0.5007649,
           'Enlarged Cardiomediastinum': 0.2713981,
           'Fibrosis': 0.5198858,
           'Fracture': 0.52066934,
           'Hernia': 0.0097252205,
           'Infiltration': 0.52939886,
           'Lung Lesion': 0.014771313,
           'Lung Opacity': 0.6033015,
           'Mass': 0.64704525,
           'Nodule': 0.69085634,
           'Pleural_Thickening': 0.25272927,
           'Pneumonia': 0.08875252,
           'Pneumothorax': 0.26505184}}

$ python process_image.py ../tests/16747_3_1.jpg -weights resnet50-res512-all
Warning: Input size (885x885) is not the native resolution (512x512) for this model. A resize will be performed but this could impact performance.
{'preds': {'Atelectasis': 0.02676412,
           'Cardiomegaly': 0.011656165,
           'Consolidation': 0.028988084,
           'Edema': 0.004503417,
           'Effusion': 0.025871767,
           'Emphysema': 0.011733518,
           'Enlarged Cardiomediastinum': 0.5,
           'Fibrosis': 0.0036442582,
           'Fracture': 0.05079817,
           'Hernia': 0.0010036419,
           'Infiltration': 0.20743765,
           'Lung Lesion': 0.5,
           'Lung Opacity': 0.821504,
           'Mass': 0.053269155,
           'Nodule': 0.15826094,
           'Pleural_Thickening': 0.023301573,
           'Pneumonia': 0.05978322,
           'Pneumothorax': 0.015683603}}

[PauleNajus]

Thank you for your prompt reply.

By “wrong,” I meant that my results deviated significantly from the radiologist-annotated labels. I’ve resolved this by mapping the ResNet predictions to xrv.datasets.default_pathologies instead of model.pathologies. I’m now getting better results - almost identical to yours.

Before closing this issue, I have one last question: Do you plan to include lung lesion and enlarged cardiomediastinum ResNet weights in future releases of TorchXRayVision?