Why does BoxLib not use MPI_Init_thread()?

[bcfriesen]

Since BoxLib supports MPI+OpenMP, should it not use MPI_Init_thread() in some form? Currently it only uses MPI_Init(). The highest level that Cray MPICH supports is MPI_THREAD_SERIALIZED.

[WeiqunZhang]

I guess this is mostly due to historical reasons. But after some thinking, I don't think it's a good idea.

If we do want to use MPI_Init_thread, there are somethings we need to think about. The thread support level cannot be a ParmParse parameter because ParmParse is initialized after MPI is initialized. If we always initialize MPI with thread support, should we start to rely on that in BoxLib? What level of support should we require? A user is allowed to pass into their own communicator to BoxLib. What if their communicator does not support thread? So none of the codes in BoxLib should depend on thread support. If that's the case, why do we need to initialize MPI with thread support? If a user's own code uses MPI thread support, they can just pass in a thread safe communicator into BoxLib.

[WeiqunZhang]

I have changed my mind after some thinking. In some situations, we could potentially need to call MPI functions. We could use MPI_Init_thread(...,MPI_THREAD_SERIALIZED). However, we should avoid calling message passing functions inside OpenMP. But it should be OK to call things like MPI_Comm_rank. Also, if BoxLib::Abort() is called inside OpenMP, it actually needs to call MPI_Abort().