Error after run for a few hours

[hptian]

Hi, my case crashed after running for a few hours and returned the following message:
"Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
mpirun noticed that process rank 0 with PID 0 on node ycy-3508 exited on signal 9". But without the libAcoustics, the case is able to run until the end. Can anyone tell me why? Thanks a lot.

[mkraposhin]

Perhaps, libAcoustics exhausted your memory.

Use top to analyze the memory consumption.

[hptian]

Thanks for your kind reply. I used top to observe what happened when it crashed. The memory usage seems normal, but the cpu usage is 200% for a single processor, while normally it is 100%. After about one minute of 200%, it crashed. I also noticed that the case would always crash at the 1285 step after the start time, but without the libAcoustics, it runs normally. Do you have any idea? Thanks again for providing this useful package for noise calculating.

[hptian]

By the way, I am using the FW-H method. What is the meaning of OAP printed during the calculation? Thanks

[mkraposhin]

Thanks for your kind reply. I used top to observe what happened when it crashed. The memory usage seems normal, but the cpu usage is 200% for a single processor, while normally it is 100%. After about one minute of 200%, it crashed. I also noticed that the case would always crash at the 1285 step after the start time, but without the libAcoustics, it runs normally. Do you have any idea? Thanks again for providing this useful package for noise calculating.

Well, most likely, this is a memory-related problem. How many CPUs do you use? Are they on the same node? How many copies of the solver do you run per one node?

[mkraposhin]

By the way, I am using the FW-H method. What is the meaning of OAP printed during the calculation? Thanks

OAP = Observed Acoustic Pressure

[hptian]

Thanks, 128 processors, yes they are on the same node. After I reduced that observer points, the error disappeared. However, after running for about 1 day, the program stopped at "Starting acoustics probe". I used TOP to see what happened but everything is OK. The cpu and the memory are all work well. The calculation does not crash but stopped to waiting for the acoustic probing process, and no errors are reported. Do you have any ideas, thanks a lot.

[mkraposhin]

As the first step, it's neccesary to gather call stack when your simulation fails.

Can you try also different partition approaches (scotch / simple / etc)?

[hptian]

Thanks a lot, I will try

[mkraposhin]

Thanks a lot, I will try

Because I need some information to localize the source of the error.

[mkraposhin]

One more question: is your mesh static or dynamic (moving)?

[hptian]

Thanks for your reply. My case is about numerical simulation of a naca airfoil, the airfoil is static and the incoming flow velocity is 30m/s. It is strange that the PSD shape at the observer position is similar to the experiments, but approximately 30 dB lower than that in experiments. I am using the FWH method and the flow field agrees well with experiments. Do you have any ideas? Thanks a lot. P.S. The case has not crashed since then.

[mkraposhin]

Is your case 2D or 3D?

If the case hasn't crashed, then it means there is a memory access error and I hope it will show itself again.

[DaraSamii]

Hi I have almost the same issue. my simulation works fine for 5000 iteration and I'm getting data normally, but my memory runs out and my simulation stops. I've tried to reduce cleanfreq or to minimize other application memory usages on the server. I wanted to know is there any setting in LibAcoustics to clean and flush the previous unneeded data accumulated by FWH to prevent this situation?

for reference these are my setting for FHW:

Farassat1A
{
        type            FfowcsWilliamsHawkings;
        formulationType Farassat1AFormulation;
        libs ("libAcoustics.so");
        
        log             true;
        enabled         true;
        
        probeFrequency  10;
        timeStart       $acousticStart;
        timeEnd         $endTime;
        
        writeFft       true;
        
        c0              343;
        pName           p;
        pInf            101325;      // For compressible solvers
        rhoName         rho;
        rho             rho;
        rhoInf          1.293;
        
        CofR (0 0 0);   // Center of rotation if needed
        
        region          region0;
        
        observers
        {
            On0_0 { position (0.0 0.0 0.0); pRef 2e-5; fftFreq 1024; }
        }


        patches (NozzleWall OuterNozzle);
        interpolationScheme point;

        surfaces
        (
                wall
                {
                    type            patch;
                    patches         (NozzleWall);
                    interpolate     false;
                }
        );
        
        cleanFreq       5;
        nonUniformSurfaceMotion true;
        Ufwh    (.0 .0 .0);
        U0      (.0 .0 .0);
        fixedResponseDelay true;
        responseDelay   0.00001;
}

Thank you very much for your library and assistance.

[mkraposhin]

hello,

1. how many faces do you have in your patch?
2. how far is the observer from your patch?
3. how many cells do you have in your mesh?

[DaraSamii]

Hi,

Currently, I have 2,363,676 mesh cells, but I am in the process of mesh analysis, so this number may change.
I have two walls of interest: OuterNozzle with 1,696 faces and NozzleWall with 42,016 faces. They are stationary and do not move.

I tried both Farassat1AFormulation and GTFormulation, but both ended up going out of memory.
I'm using a server to submit my simulations — currently running on 48 CPUs and 30 GB RAM.

With 30 GB RAM and the current mesh, I get exactly 512 rows of pressure fluctuation data before it crashes.
I increased my RAM to 60 GB, and it passed 512 rows. However, since I am running simulations with a very small timestep, I expect the simulation to run for a long time, and 60 GB may not be enough.

I have multiple observers, and I plan to increase their number. Initially, I placed an observer about 60 cm away from the center of my nozzle (for reference, the nozzle exit diameter is about 1 cm).

Consequently, I am looking for a way to dump the acoustic values to a file periodically to free up RAM and allow the simulation to continue longer.
I tried adjusting cleanFreq, but with multiple values, my simulation always ran out of memory after 512 iterations of the acoustics solver.
I know that if I disable acoustics, the simulation runs for hours and thousands of iterations with no problems.