In OpenFOAM, the solvers and numerical models are separated. Most of the numerical models are located in the $FOAM_SRC path, and being compiled to a dynamic library “.so” binary file, which can be loaded either by linking it during the solver compiling process or loading it by specify the “.so” file in the case controlDict. For example, we develop a new combustion model, named PaSR, and compile it to a “libPaSR.so”. When we run a case, we just need to add a line libs ("libPaSR.so"); in controlDict. Then, we can see and use this model in the reactingFoam solver.

How does the “libPaSR.so” file is loaded, and why the solver knows that a new model “PaSR” is available? This blog will explain it.

dynamic library loading in C++

Before digging into OpenFOAM, let’s see how does the dynamic library is loaded in C++. There are three famous function named dlopen, dlsym and dlclose, which are used to open, find function address and close library, in your operating system.

library loading searching and closing

I wrote a short code to demonstrate how does it work, see https://github.com/ZmengXu/dynamicLibloading/tree/master/ch01

First, we define two models WSR and PaSR (which are two simple combustion model) in two .C files. Each file has only one simple function, named get_library_name. Compiling these two .C files to two libraries, libWSR.so and libPaSR.so.

1
2
3
4
5
// In WSR.C
char* get_library_name(void)
{
    return "This is WSR";
}
1
2
3
4
5
// In PaSR.C
char* get_library_name(void)
{
    return "This is PaSR";
}

Second, we write a solver named solverFoam to call get_library_name function in libWSR.so and libPaSR.so.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>

#include "word.H"
#include "IOstreams.H"
using namespace Foam;

// define a function pointer type
typedef char* (*funcType)(void);

int main(int argc, char *argv[])
{
    if(argc<2)
    {
        Info << "usage: <library>\n" << endl;
        return 0;
    }

    char* librarypath = argv[1];

    // RTLD_LAZY means "there may be symbols that can't be resolved;
    // don't try to resolve them until they're used." 
    auto libhandle = dlopen(librarypath, RTLD_LAZY);

    if(libhandle != NULL)
    {
        Info << librarypath << " is found." << endl;

        auto voidPtr = dlsym(libhandle, "_Z16get_library_namev");

        funcType funcPtr = (funcType)voidPtr;

        Info << funcPtr() << endl;

        dlclose(libhandle);
    }
    else
    {
        Info << librarypath << " is not found." << endl;
    }

    Info << "End." << endl;

    return 0;
}

In this code, we get the librarypath from the solver argument, and try to find the dynamic library, load the library to computer memory and get a handle libhandle using dlopen. We use dlsym to find the function _Z16get_library_namev in this library and call this function to print its information.

you can use Allrun in https://github.com/ZmengXu/dynamicLibloading/blob/master/Allrun to compile and run it.

1
solverFoam "libWSR.so"

For example, typing the above commands after compiling the library and solver. It will find the “libWSR.so” in $LD_LIBRARY_PATH and printing the following results.

1
2
3
libWSR.so is found.
This is WSR
End.

function name decoration

Here, we have two things need to be explained:

  • typedef char* (*funcType)(void); is an alias, define a function pointer type. This function has a return value of char* and a void argument. We call this kind of function as funcType. Now you can understand that Line 32: funcType funcPtr = (funcType)voidPtr; means that we are about to convert voidPtr to a function pointer and assign it to a new funcType type function pointer funcPtr. Thus, it can be used as funcPtr() in Line 34.

  • You must be curious that the function name we defined in PaSR.C is “get_library_name”, but why we are looking for the function name “_Z16get_library_namev”. This is due to the name mangling (also called name decoration) in C++ compiler. We can use objdump -tT $FOAM_USER_LIBBIN/libPaSR.so or nm -D $FOAM_USER_LIBBIN/libPaSR.so to see the function list in the libPaSR.so dynamic libray.

This is an example for nm -D $FOAM_USER_LIBBIN/libPaSR.so result, you can find _Z16get_library_namev in the optput.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
0000000000201040 B __bss_start
                 U __cxa_atexit
                 w __cxa_finalize
0000000000201040 D _edata
0000000000201048 B _end
0000000000000924 T _fini
                 w __gmon_start__
0000000000000748 T _init
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
                 w _Jv_RegisterClasses
0000000000000900 T _Z16get_library_namev
                 U _ZN4Foam13messageStreamcvRNS_8OSstreamEEv
                 U _ZN4Foam4InfoE
                 U _ZN4FoamlsERNS_7OstreamEPKc
                 U _ZNSt8ios_base4InitC1Ev
                 U _ZNSt8ios_base4InitD1Ev

The name mangling is a common issue when you want to use Hybrid Programming in OpenFOAM, like “C” and “C++”, or “fortune” and “C++”. You can use the keyword extern C to avoid the compiler converting function name. For example, we can change WSR.C to:

1
2
3
4
5
6
7
8
9
10
11
12
13
#ifdef __cplusplus
extern "C"
{
#endif

char* get_library_name(void)
{
    return "This is PaSR";
}

#ifdef __cplusplus
}
#endif

compile it and use nm -D $FOAM_USER_LIBBIN/libPaSR.so you can get this. The function name is not decorated anymore.

1
2
3
4
5
6
7
8
9
10
11
0000000000201020 B __bss_start
                 w __cxa_finalize
0000000000201020 D _edata
0000000000201028 B _end
0000000000000688 T _fini
0000000000000680 T get_library_name
                 w __gmon_start__
0000000000000540 T _init
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
                 w _Jv_RegisterClasses

dynamic library loading in OpenFOAM

In OpenFOAM, the dynamic library are loaded and unloaded implicitly in the dlLibraryTable class, see $FOAM_SRC/OpenFOAM/db/dynamicLibrary/dlLibraryTable. I created a tutorial code to demonstrate how does it work, see https://github.com/ZmengXu/dynamicLibloading/tree/master/ch02

dlLibraryTable class

In the dlLibraryTable class, it has a constructor using a specific dictionary to look through all the libraries, load the libraries to computer memory and store the library names and pointers to libNames_ and libPtrs_ list. When the dlLibraryTable object is deleted, it will call the deconstructor function to unload these libraries. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
// * * * * * * * * * * * * * * * * Constructors  * * * * * * * * * * * * * * //

Foam::dlLibraryTable::dlLibraryTable
(
    const dictionary& dict,
    const word& libsEntry
)
{
    open(dict, libsEntry);
}

// * * * * * * * * * * * * * * * * Destructor  * * * * * * * * * * * * * * * //

Foam::dlLibraryTable::~dlLibraryTable()
{
    forAllReverse(libPtrs_, i)
    {
        if (libPtrs_[i])
        {
            if (debug)
            {
                InfoInFunction
                    << "Closing " << libNames_[i]
                    << " with handle " << uintptr_t(libPtrs_[i]) << endl;
            }
            if (!dlClose(libPtrs_[i]))
            {
                WarningInFunction<< "Failed closing " << libNames_[i]
                    << " with handle " << uintptr_t(libPtrs_[i]) << endl;
            }
        }
    }
}

In the solverFoam, we create a dictionary named controlDict, which will read a dictionary file controlDict, and providing the “libs” as the libsEntry. Then the libraies in the controlDict.libs will be loaded and unloaded automatically. This work is done in the Time class in $FOAM_SRC/OpenFOAM/db/Time and be called from #include "createTime.H" by each CFD solver. This is the reason why the libraries we wrote in the “libs” of controlDict will be loaded automaticlly.

The implementation of a short runTime selection

In terms of the libraries, a base class combustionModelBase and a derived class WSR in the combustionModel folder are compiled to libcombustModel.so file.

In the base class, a static HashTable pointer member WordConstructorTablePtr_ is created. A template class addWordConstructorToTable is defined to operate this pointer. In the addWordConstructorToTable constructor, the static HashTable pointer member WordConstructorTablePtr_ is created and insert a “debug value - typeName” pair into the table. Once the constructor is called, it will print “Try to insert typeName” in the constructor, if the typeName exist in the table, it will print “Duplicate entry typeName in runtime selection table”.

1
2
3
//In combustionModel.H
typedef HashTable< label, word > WordConstructorTable;
static WordConstructorTable* WordConstructorTablePtr_;
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
template< class Type >
class addWordConstructorToTable
{
    public:
    addWordConstructorToTable( const word& lookup = Type::typeName )
    {
        constructWordConstructorTables();
        std::cout << "Try to insert " << lookup 
        << " into the WordConstructorTablePtr_"
        << std::endl;
        if (!WordConstructorTablePtr_->insert(lookup, Type::debug))
        {
            std::cerr<< "Duplicate entry "
                << lookup << " in runtime selection table "
                << "combustionModelBase" << std::endl;
            error::safePrintStack(std::cerr);
        }
    }
    
    ~addWordConstructorToTable()
    {
        destroyWordConstructorTables();
    }
};

WSR is inherited from the base class, so it also has the template class addWordConstructorToTable, in the WSR.C, an object addWSRWordConstructorTocombustionModelBaseTable_ is defined, which will call the constructor, and get the above output. We will see when the table is created. This is very important.

1
2
//In WSR.C
WSR::addWordConstructorToTable< WSR > addWSRWordConstructorTocombustionModelBaseTable_;

Apart from that, a static function “New” is defined in combustionModelNew.C to search the typeName from this table.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
// static Factory Method (selector)
void combustionModelBase::New (const word& modelName)
{

    Info<< "\nWe are now in combustionModelBase::New fucntion "
        << "\nlooking for the "
        << modelName
        << " in the WordConstructorTable\n"
        << WordConstructorTablePtr_->sortedToc()
        << endl;

    // Find the class typeName pointer in the RTS Table 
    // (HashTable<word, autoPtr<combustionModelBase>(*)(word))
    WordConstructorTable::iterator cstrIter =
        WordConstructorTablePtr_->find(modelName);

    // If the Factory Method was not found. 
    if (cstrIter == WordConstructorTablePtr_->end())
    {
        FatalErrorIn
        (
            "combustionModelBase::New(const word&)"
        )   << "Unknown combustionModelBase type "
            << modelName << nl << nl
            << "Valid combustionModelBase types are :" << endl
            << WordConstructorTablePtr_->sortedToc()
            << exit(FatalError);
    }

    Info<< "\nWe found the modelName "
        << modelName
        << "\nThe debug value for it is "
        << (*WordConstructorTablePtr_)[modelName] << endl;

}

The order of the dynamic libraries loading

In the same way, PaSR is defined, the difference is that it is compiled separately to libPaSR.so file. The libcombustModel.so is written in the solverFoam’s Make/options and linked to the solverFoam during the compiling stage. while the libPaSR.so file is loaded when we run the solverFoam. You can use Allrun in https://github.com/ZmengXu/dynamicLibloading/blob/master/Allrun to compile and run it. Before the libPaSR.so is created, we can not find it in the $LD_LIBRARY_PATH, thus only one model WSR is available in the WordConstructorTable. After libPaSR.so is created, you will see the following outputs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Try to insert WSR into the WordConstructorTablePtr_

Entering main()


Loading the libraries

Try to insert PaSR into the WordConstructorTablePtr_

After the libraries are loading


We are now in combustionModelBase::New fucntion
looking for the PaSR in the WordConstructorTable

2
(
PaSR
WSR
)


We found the modelName PaSR
The debug value for it is 2

End

It is interesting that WSR is inserted into the WordConstructorTablePtr_ before entring main() function, PaSR is inserted into the WordConstructorTablePtr_ after the libPaSR.so is loaded. This is because that libcombust.so is loaded before the main() function, while libPaSR.so is loaded during the run time. Once the dynamic library is loaded, the objects addWSRWordConstructorTocombustionModelBaseTable_ and addPaSRWordConstructorTocombustionModelBaseTable_ are constructed automatically in the computer memory as global variables. If we load one library twice, the insert function will reject the inserting. This is why we see the Duplicate entry before the OpenFOAM output header.