Running ISCA in very warm climates

[AndrewILWilliams]

Hello there,

I'm currently using the SSM module ( PR #284 ) to run simulations of very warm climates, with surface temperatures >340K. I keep running into this error:

2025-09-30 12:06:55,656 - isca - DEBUG - Integration completed through    8 May  7   0: 0: 0
2025-09-30 12:07:10,370 - isca - DEBUG - Integration completed through    8 May  8   0: 0: 0
2025-09-30 12:07:15,122 - isca - DEBUG - double free or corruption (out)
2025-09-30 12:07:15,122 - isca - DEBUG - 
2025-09-30 12:07:15,122 - isca - DEBUG - Program received signal SIGABRT: Process abort signal.
2025-09-30 12:07:15,122 - isca - DEBUG - 
2025-09-30 12:07:15,122 - isca - DEBUG - Backtrace for this error:
2025-09-30 12:07:15,195 - isca - DEBUG - #0  0x1534f158f5af in ???
2025-09-30 12:07:15,195 - isca - DEBUG - #1  0x1534f158f52f in ???
2025-09-30 12:07:15,196 - isca - DEBUG - #2  0x1534f1562e64 in ???
2025-09-30 12:07:15,196 - isca - DEBUG - #3  0x1534f15d0806 in ???
2025-09-30 12:07:15,196 - isca - DEBUG - #4  0x1534f15d7b0b in ???
2025-09-30 12:07:15,196 - isca - DEBUG - #5  0x1534f15d9b8f in ???
2025-09-30 12:07:15,196 - isca - DEBUG - #6  0x563148d4b996 in ???
2025-09-30 12:07:15,196 - isca - DEBUG - #7  0x563148c7191f in ???
2025-09-30 12:07:15,196 - isca - DEBUG - #8  0x563148d4ed25 in ???
2025-09-30 12:07:15,196 - isca - DEBUG - #9  0x563148882e0e in ???
2025-09-30 12:07:15,196 - isca - DEBUG - #10  0x5631486ecb7e in ???
2025-09-30 12:07:15,196 - isca - DEBUG - #11  0x1534f157b7e4 in ???
2025-09-30 12:07:15,196 - isca - DEBUG - #12  0x5631486ecbae in ???
2025-09-30 12:07:15,196 - isca - DEBUG - #13  0xffffffffffffffff in ???
2025-09-30 12:07:15,234 - isca - DEBUG - --------------------------------------------------------------------------
2025-09-30 12:07:15,234 - isca - DEBUG - Primary job  terminated normally, but 1 process returned
2025-09-30 12:07:15,234 - isca - DEBUG - a non-zero exit code. Per user-direction, the job has been aborted.
2025-09-30 12:07:15,234 - isca - DEBUG - --------------------------------------------------------------------------
2025-09-30 12:07:15,271 - isca - DEBUG - --------------------------------------------------------------------------
2025-09-30 12:07:15,272 - isca - DEBUG - mpirun noticed that process rank 19 with PID 2701519 on node stellar-m06n23 exited on signal 6 (Aborted).
2025-09-30 12:07:15,272 - isca - DEBUG - --------------------------------------------------------------------------

The error persists if I use a smaller timestep, and always seems to happen when the surface temperatures reach ≈350K. I'm not getting any other errors in the run.

Does anyone recognize this kind of error? The error message doesn't mean much to me.

Note that I've already extended the min/max temperatures in the namelist:

    'qe_moist_convection_nml': {
        'rhbm':0.7,
        'Tmin':100.,
        'Tmax':550.
    },

...

    'sat_vapor_pres_nml': {
        'do_simple':True,
        'tcmin_simple': -273,
        'tcmax_simple': 1000,
    },

Cheers,
Andrew

[AndrewILWilliams]

I've also encountered this error:

2025-10-02 15:32:31,035 - isca - DEBUG - Integration completed through    8 Jun 11   0: 0: 0
2025-10-02 15:32:46,147 - isca - DEBUG - Integration completed through    8 Jun 12   0: 0: 0
2025-10-02 15:32:50,242 - isca - DEBUG - munmap_chunk(): invalid pointer
2025-10-02 15:32:50,242 - isca - DEBUG - 
2025-10-02 15:32:50,242 - isca - DEBUG - Program received signal SIGABRT: Process abort signal.
2025-10-02 15:32:50,242 - isca - DEBUG - 
2025-10-02 15:32:50,242 - isca - DEBUG - Backtrace for this error:
2025-10-02 15:32:50,315 - isca - DEBUG - #0  0x14e416e1f5af in ???
2025-10-02 15:32:50,315 - isca - DEBUG - #1  0x14e416e1f52f in ???
2025-10-02 15:32:50,315 - isca - DEBUG - #2  0x14e416df2e64 in ???
2025-10-02 15:32:50,315 - isca - DEBUG - #3  0x14e416e60806 in ???
2025-10-02 15:32:50,315 - isca - DEBUG - #4  0x14e416e67b0b in ???
2025-10-02 15:32:50,315 - isca - DEBUG - #5  0x14e416e67f2b in ???
2025-10-02 15:32:50,315 - isca - DEBUG - #6  0x557fa711997a in ???
2025-10-02 15:32:50,315 - isca - DEBUG - #7  0x557fa703f91f in ???
2025-10-02 15:32:50,315 - isca - DEBUG - #8  0x557fa711cd25 in ???
2025-10-02 15:32:50,315 - isca - DEBUG - #9  0x557fa6c50e0e in ???
2025-10-02 15:32:50,315 - isca - DEBUG - #10  0x557fa6abab7e in ???
2025-10-02 15:32:50,315 - isca - DEBUG - #11  0x14e416e0b7e4 in ???
2025-10-02 15:32:50,315 - isca - DEBUG - #12  0x557fa6ababae in ???
2025-10-02 15:32:50,315 - isca - DEBUG - #13  0xffffffffffffffff in ???
2025-10-02 15:32:50,322 - isca - DEBUG - --------------------------------------------------------------------------
2025-10-02 15:32:50,322 - isca - DEBUG - Primary job  terminated normally, but 1 process returned
2025-10-02 15:32:50,322 - isca - DEBUG - a non-zero exit code. Per user-direction, the job has been aborted.
2025-10-02 15:32:50,322 - isca - DEBUG - --------------------------------------------------------------------------
2025-10-02 15:32:50,322 - isca - DEBUG - --------------------------------------------------------------------------
2025-10-02 15:32:50,322 - isca - DEBUG - mpirun noticed that process rank 12 with PID 2739848 on node stellar-m01n14 exited on signal 6 (Aborted).
2025-10-02 15:32:50,322 - isca - DEBUG - --------------------------------------------------------------------------

It's different, but still seems like a memory issue (perhaps?)

[AndrewILWilliams]

@sit23 , for the sponge layer, do you mean this section of the namelist? Do you have ideas of reasonable values to use for sponge_pbottom?

    'damping_driver_nml': {
        'do_rayleigh': True,
        'trayfric': -0.25,              # neg. value: time in *days*
        'sponge_pbottom':  5000.,           #Bottom of the model's sponge down to 50hPa (units are Pa)
        'do_conserve_energy': True,             
    },

[daw538]

@AndrewILWilliams yes damping_driver_nml controls the sponge, so you might consider changing the value the value of sponge_pbottom

[ntlewis]

The sponge layer won’t do anything for the SCM. I am 99% confident this is associated with a bug in SBM that Ryan and I have come across (and Ryan has a fix for; I lost it when we stopped paying for Slack). Will get back to you once we’ve dug it out.

[sit23]

Sorry - I hadn't appreciated that this was the column model and not the full model.

[Ryan-Boukrouche]

Hello,

So my runs were using the full model and could reach temperatures up to 360 K including the presence of water vapor. It does crash when it goes any hotter or more humid, but not with these errors messages, rather it's temperatures exceeding the upper limit of the saturation vapor pressure table that makes it crash. With Neil we did have to tweak the model to make it run in these conditions, I just remember that we tried different values for 'surf_res', 'exponent', and 'scale_heights' to define the vertical levels, before settling on using custom bk and pk tables with 48 vertical levels. Because my atmosphere was warmer I also needed a much thinner sponge, down to 20 Pa. Below are my versions of the namelists you showed. I haven't used the 1D column version of Isca yet, so I'm not sure if it may bring about different error messages...

'qe_moist_convection_nml': {
    'rhbm':0.7,
    'Tmin':160., 
    'Tmax':400.
},
'sat_vapor_pres_nml': {
    'show_bad_value_count_by_slice':True,
    'show_all_bad_values':True, # Show all bad temperatures causing saturation vapor pressure table overflows
    'do_simple':True,
    'construct_table_wrt_liq_and_ice':False,
    'do_not_calculate': False, # Don't compute esat for non-Earth simulations
    'tcmin': -160, 
    'tcmax': 100, 
    'tcmin_simple': -273, 
    'tcmax_simple': 350 
   },
'damping_driver_nml': {
    'do_rayleigh': True,
    'trayfric': -0.5,              # neg. value: time in *days*
    'sponge_pbottom':  20.0*(p_surf/1e5), 
    'do_conserve_energy': True,      
},
'spectral_dynamics_nml': {
    'damping_order': 4,             
    'water_correction_limit': 200.e2,
    'reference_sea_level_press':p_surf,
    'num_levels':48,      #How many model pressure levels to use
    'valid_range_t':[60.,800.], #[20.,2000.]
    'initial_sphum':[2.e-6],
    'vert_coord_option':'input',
    'robert_coeff':0.03
},

[ntlewis]

@Ryan-Boukrouche apologies, I was referring to the change you made to the Fortran in SBM. Could you share that? To do with the array index at the top of atmosphere.

[Ryan-Boukrouche]

Ah, ok I think I found the fix we made to it: this is my qe_moist_convection.F90 starting from line 668:

                  ! State that you have CAPE
                  if (nocape) then
                     nocape = .false.
                     kLFC   = k
                  end if
                  if ((kLZB .eq. 0) .and. (k .eq. 1)) then
                    kLZB = 1 
                  endif 
               end if
            end if
         end if
      end if
   end do

20 end if

The difference is that we added the following:

                  if ((kLZB .eq. 0) .and. (k .eq. 1)) then
                    kLZB = 1 
                  endif 

Hope this helps!

[AndrewILWilliams]

Hi there, sorry for the confusion. I am actually using the full model! @Ryan-Boukrouche , would you be able to share the custom bk and pk tables you used for the high temperatures? I've also struggled to find a vertical grid which looks reasonable in these hot climates.

I'll also try implementing this fix to the SBM!

[Ryan-Boukrouche]

Here you are:

'bk': [ 0.00000,       0.00000,       0.00000,   
            0.00000,       0.00000,       0.00000,   
            0.00000,       0.00000,       0.00000,   
            0.00000,       0.00000,       0.00000,   
            0.00000,       0.00000,       0.00000,   
            0.00000,       0.00000,       0.00000,   
            0.00000,       0.00000,       0.00000,   
            0.00000,       0.00000,       0.00000,   
            0.00000,       0.00000,       0.00000,   
            0.00000,       0.00000,       0.01253,   
            0.04887,       0.10724,       0.18455,   
            0.27461,       0.36914,       0.46103,   
            0.54623,       0.62305,       0.69099,   
            0.75016,       0.80110,       0.84453,   
            0.88127,       0.91217,       0.93803,   
            0.95958,       0.97747,       0.99223,  
            1.00000],
    
    'pk': np.array([    1.00000,       2.69722,       5.17136,   
               8.89455,      14.24790,      22.07157,   
              33.61283,      50.48096,      74.79993,   
             109.40055,     158.00460,     225.44108,   
             317.89560,     443.19350,     611.11558,   
             833.74392,    1125.83405,    1505.20759,   
            1993.15829,    2614.86254,    3399.78420,   
            4382.06240,    5600.87014,    7100.73115,   
            8931.78242,   11149.97021,   13817.16841,   
           17001.20930,   20775.81856,   23967.33875,   
           25527.64563,   25671.22552,   24609.29622,   
           22640.51220,   20147.13482,   17477.63530,   
           14859.86462,   12414.92533,   10201.44191,   
            8241.50255,    6534.43202,    5066.17865,   
            3815.60705,    2758.60264,    1870.64631,   
            1128.33931,     510.47983,       0.00000,   
               0.00000]) 

[ntlewis]

@AndrewILWilliams (also @sit23) Apologies for the SCM/GCM mix up (I think I read SSM as SCM.... too many acronyms). Nonetheless, I think this might be resolved by the little bit of fortran code Ryan shared (it fixed 'hot planet' GCM runs of Ryan's that were crashing with some sort of memory issue). Let me know how you get on with this.

[ntlewis]

Also @sit23, we should incorporate the bug fix I suggested for Ryan (that he's shared above) in the master. Basically, if kLZB isn't found then it defaults to 0, but fortran array indexes start at 1, so this gives a memory error. I think that the intended behaviour in this situation is for kLZB to be set to 1 in this scenario (which is what the code Ryan shared above does).

[sit23]

That's great that the convection scheme fixed the memory issues. @ntlewis - let's try and get a pull request together to get this fix in.