C/C++ How Does Dynamic Linking Work On Different Platforms?
How does dynamic linking work generally? On Windows (LoadLibrary), you need a .dll to call at runtime, but at link time, you need to provide a corresponding .lib file or the program won’t link. What does the .lib file contain? A description of the .dll methods? Isn’t that what the headers contain? Relatedly, on *nix, you don’t need a lib file. How how does the compiler know that the methods described in the header will be available at runtime? As a newbie, when you think about either one of the two schemes, then the other, neither of them make sense.
No, if you load a DLL with LoadLibrary you should not need to link to any specific library. That’s the point of dynamically loading libraries. It’s the same on other platforms.
@JoachimPileborg true, but the OP is (understandably) puzzled about why dynamic libs on Windows have a .lib file associated with the .dll, when it isn’t necessary on other platforms.
I think but do not know that a Windows .LIB file contains stub definitions which the dynamic linker then replaces. But this is not necessary. If a system uses stub functions, they can be emitted into the executable at compile time. Alternatively, the executable can simply have missing symbols (just like an object file) and the dynamic linker really goes and links them at startup time just like a static linker would do. I think both of those approaches have been used on unix.
Not sure where most people are getting their windows information from on here, but you certainly DO NOT need a .lib file to use LoadLibrary or link its .lib file, nor to call its functions this is completely wrong.
From what I can tell dlopen and dlsym are the same as LoadLibrary/GetProcAddress so the boiler plate code is actually the same?
9 Answers 9
To answer your questions one by one:
- Dynamic linking defers part of the linking process to runtime. It can be used in two ways: implicitly and explicitly. Implicitly, the static linker will insert information into the executable which will cause the library to load and resolve the necessary symbols. Explicitly, you must call LoadLibrary or dlopen manually, and then GetProcAddress / dlsym for each symbol you need to use. Implicit loading is used for things like the system library, where the implementation will depend on the version of the system, but the interface is guaranteed. Explicit loading is used for things like plug-ins, where the library to be loaded will be determined at runtime.
- The .lib file is only necessary for implicit loading. It contains the information that the library actually provides this symbol, so the linker won’t complain that the symbol is undefined, and it tells the linker in what library the symbols are located, so it can insert the necessary information to cause this library to automatically be loaded. All the header files tell the compiler is that the symbols will exist, somewhere; the linker needs the .lib to know where.
- Under Unix, all of the information is extracted from the .so . Why Windows requires two separate files, rather than putting all of the information in one file, I don’t know; it’s actually duplicating most of the information, since the information needed in the .lib is also needed in the .dll . (Perhaps licensing issues. You can distribute your program with the .dll , but no one can link against the libraries unless they have a .lib .)
The main thing to retain is that if you want implicit loading, you have to provide the linker with the appropriate information, either with a .lib or a .so file, so that it can insert that information into the executable. And that if you want explicit loading, you can’t refer to any of the symbols in the library directly; you have to call GetProcAddress / dlsym to get their addresses yourself (and do some funny casting to use them).
With two separate files, you can link against a DLL without actually having the DLL. That might not be the actual reason they use two files, but it’s a plausible reason.
The question is about dynamic linking but it may be good to know that .lib files can also be used for static linking, essentially linking the libraries content into the executable, making DLLs unnecessary.
Re point 2: You could point out that this problem is MSVC-specific, ant it’s possible to link to the DLL itself on MS Windows platform. See edit to my answer.
I guess it is worth to say, that the .lib file is not required to use the DLL. It just simplifies the process.
@Excelcius It’s not the same .lib . Microsoft uses the same extension for two different types of files.
The .lib file on Windows is not required for loading a dynamic library, it merely offers a convenient way of doing so.
In principle, you can use LoadLibrary for loading the dll and then use GetProcAddress for accessing functions provided by that dll. The compilation of the enclosing program does not need to access the dll in that case, it is only needed at runtime (ie. when LoadLibrary actually executes). MSDN has a code example.
The disadvantage here is that you need to manually write code for loading the functions from the dll. In case you compiled the dll yourself in the first place, this code simply duplicates knowledge that the compiler could have extracted from the dll source code automatically (like the names and signatures of exported functions).
This is what the .lib file does: It contains the GetProcAddress calls for the Dlls exported functions, generated by the compiler so you don’t have to worry about it. In Windows terms, this is called Load-Time Dynamic Linking, since the Dll is loaded automatically by the code from the .lib file when your enclosing program is loaded (as opposed to the manual approach, referred to as run-time dynamic linking).
The .lib file probably doesn’t have any information that isn’t present (perhaps in another format) in the .dll . I’m not sure why Microsoft decided to use two files (since the existing practice when Microsoft implemented DLLs was to use a single file); I can only guess that it was to provide a means by which you could distribute the libraries necessary to run your executable without distributing the means of creating a program which would link against those libraries.
@JamesKanze I’m not sure that’s entirely true. Dll allows limited reflection, for instance it is possible to enumerate the names of the exported functions. However I’m not sure if it’s possible to reconstruct the full function signature from just the dll.
@JamesKanze You are of course right, the signatures come from the headers and not the .lib files. So it seems that the convenience gain provided by the .lib files for loading could have been just as easily achieved with a smarter runtime loader.
I’ve seen DLLs with the function names stripped, where the functions were only accessible via ordinal. An import library would be able to give names to those functions without exposing them in the DLL.
@Charlie: It’s probably at least partly an IP thing. But keep in mind, too, the Windows ecosystem is quite different from *nix; there’s no One True Compiler and no One True Calling Convention in Windows. (There are at least two: cdecl, and stdcall. And any compiler could come up with its own as well.) Having the import library distinct from the DLL allows the differences to be hidden away behind a wrapper function. So the code can use the compiler’s native calling convention, and the import library would do the translation.
How does dynamic linking work generally?
The dynamic link library (aka shared object) file contains machine code instructions and data, along with a table of metadata saying which offsets in that code/data relate to which «symbols», the type of the symbol (e.g. function vs data), the number of bytes or words in the data, and a few other things. Different OS will tend to have different shared object file formats, and indeed the same OS may support several, but that’s the gist of it.
So, imagine the shared library’s a big chunk of bytes with an index like this:
SYMBOL ADDRESS TYPE SIZE my_function 1000 function 2893 my_number 4800 variable 4 In general, the exact type of the symbols need not be captured in the metadata table — it’s expected that declarations in the library’s header files contain all the missing information. C++ is a bit special — compared to say C — because overloading can mean there are several functions with the same name, and namespaces allow for further symbols that would otherwise be ambiguously named — for that reason name mangling is typically used to concatenate some representation of the namespace and function arguments to the function name, forming something that can be unique in the library object file.
A program wanting to use the shared object can generally do one of two things:
- have the OS load both itself and the shared object around the same time (before executing main() ), with the OS Loader responsible for finding the symbols and examining metadata in the program file image about the use of those symbols, then patching in symbol addresses in the memory the program uses, such that the program can then just run and work functionally as if it’d known about the symbol addresses when it was first compiled (but perhaps a little slower)
- or, explicitly in its own source code call dlopen sometime after main runs, then use dlsym or similar to get the symbol addresses, save them into (function/data) pointers based on the programmer’s knowledge of the expected data types, then call them explicitly using the pointers.
On Windows (LoadLibrary), you need a .dll to call at runtime, but at link time, you need to provide a corresponding .lib file or the program won’t link.
That doesn’t sound right. Should be one or the other I’d think.
Wtf does the .lib file contain? A description of the .dll methods? Isn’t that what the headers contain?
A lib file is — at this level of description — pretty much the same as a shared object file. the main difference is that the compiler’s finding the symbol addresses before the program’s shipped and run.
Where do executables look for shared objects at runtime?
I understand how to define include shared objects at linking/compile time. However, I still wonder how do executables look for the shared object ( *.so libraries) at execution time. For instance, my app a.out calls functions defined in the lib.so library. After compiling, I move lib.so to a new directory in my $HOME . How can I tell a.out to go look for it there?
A quick solution is to use g++ -Wl,-R. this will force loader to look into the current folder for so libraries. Or -Wl,-R$HOME/path to specidy some other fixed folder.
4 Answers 4
The shared library HOWTO explains most of the mechanisms involved, and the dynamic loader manual goes into more detail. Each unix variant has its own way, but most use the same executable format (ELF) and have similar dynamic linkers¹ (derived from Solaris). Below I’ll summarize the common behavior with a focus on Linux; check your system’s manuals for the complete story.
(Terminology note: the part of the system that loads shared libraries is often called “dynamic linker”, but sometimes “dynamic loader” to be more precise. “Dynamic linker” can also mean the tool that generates instructions for the dynamic loader when compiling a program, or the combination of the compile-time tool and the run-time loader. In this answer, “linker” refers to the run-time part.)
In a nutshell, when it’s looking for a dynamic library ( .so file) the linker tries:
- directories listed in the LD_LIBRARY_PATH environment variable ( DYLD_LIBRARY_PATH on OSX);
- directories listed in the executable’s rpath;
- directories on the system search path, which (on Linux at least) consists of the entries in /etc/ld.so.conf plus /lib and /usr/lib .
The rpath is stored in the executable (it’s the DT_RPATH or DT_RUNPATH dynamic attribute). It can contain absolute paths or paths starting with $ORIGIN to indicate a path relative to the location of the executable (e.g. if the executable is in /opt/myapp/bin and its rpath is $ORIGIN/../lib:$ORIGIN/../plugins then the dynamic linker will look in /opt/myapp/lib and /opt/myapp/plugins ). The rpath is normally determined when the executable is compiled, with the -rpath option to ld , but you can change it afterwards with chrpath .
In the scenario you describe, if you’re the developer or packager of the application and intend for it to be installed in a …/bin , …/lib structure, then link with -rpath=’$ORIGIN/../lib’ . If you’re installing a pre-built binary on your system, either put the library in a directory on the search path ( /usr/local/lib if you’re the system administrator, otherwise a directory that you add to $LD_LIBRARY_PATH ), or try chrpath .