Killed by Signal 9

When running in parallel, sometimes you may see an error like:

rank 1 in job 34  n7_3767
caused collective abort of all ranks
exit status of rank 1: killed by signal 9

In most situations this is a case of “out of memory”. Be careful not to run too many MPI processes on the same node. Sometimes threading can help to reduce memory usage and still parallelize on the available cores.