Crascit

Caffeine-powered software

Enhanced source file handling with target_sources()

In all but trivial CMake projects, it is common to find targets built from a large number of source files. These files may be distributed across various subdirectories, which may themselves be nested multiple levels deep. In such projects, traditional approaches usually either list all source files at the top-most level or build up the list of source files in a variable and pass that to add_library(), add_executable(), etc. With CMake 3.1, a new command target_sources() was introduced which provides the missing piece among the various target_... commands. While the CMake documentation succintly describes what target_sources() does, it fails to highlight just how useful the new command is and why it promotes better CMake projects:

  • It can lead to cleaner and more concise CMakeLists.txt project files.
  • Dependency information can be specified closer to where the actual dependencies exist in the directory heirarchy.
  • Source files gain the ability to become part of a target’s interface.
  • Source files can be added to third party project targets without having to modify the third party project files.

Life before target_sources()

Typically, developers first learn CMake in a very simple manner, defining a target by listing the source files directly in the add_executable() or add_library() command itself. Eg

add_executable(myApp src1.cpp src2.cpp)

When the number of source files grows large and they get distributed over a number of subdirectories, possibly nested to multiple levels, this quickly becomes unwieldly. It also results in having to repeat the directory structure, which reduces the benefit of structuring source files into directories in the first place.

The logical improvement many developers then make is to build up the list of source files in a variable as each subdirectory is pulled in via include(). Then after all the subdirectories have been included, add_executable() or add_library() is called, but this time passing just the variable instead of an explicit list of files. The top level CMakeLists.txt file then looks something like this:

# The name of the included file could be anything,
# it doesn't have to be called CMakeLists.txt
include(foo/CMakeLists.txt)
include(bar/CMakeLists.txt)

add_executable(myApp ${myApp_SOURCES})

with the subdirectory files structured something like this:

list(APPEND myApp_SOURCES
    "${CMAKE_CURRENT_LIST_DIR}/foo.cpp"
    "${CMAKE_CURRENT_LIST_DIR}/foo_p.cpp"
)

This allows each subdirectory to define just the sources it provides and to delegate any further nested subdirectories with another include(). It also keeps the top level CMakeLists.txt file quite small and the CMakeLists.txt in each subdirectory also tends to be reasonably uncomplicated and focussed just on the things in that directory.

As an alternative to explicitly building up a list of source files in a variable, some developers instead choose to let CMake find the source files and generate the contents of that variable automatically with the file(GLOB_RECURSE ...) command. While at first this may seem very attractive for its simplicity, this technique has a number of drawbacks and is actively discouraged by the CMake documentation. Nevertheless, it is often used by developers new to CMake until they experience first hand the problems the technique introduces.

target_sources(): All the advantages without the drawbacks

It may not be immediately obvious what the down sides are to the approach above where the source files are built up in a variable and then that variable is passed to an add_library() or add_executable() call at the top level CMakeLists.txt file. One drawback is that the variables themselves are not a particularly robust way to record the source file list. For example, if many targets are being built up throughout a directory heirarchy, then the number and naming of variables can get out of hand. This can be somewhat addressed by sticking to some kind of naming convention associated with the target a variable is used with, but this relies on all developers knowing and adhering to that convention. Furthermore, if a developer inadvertently tries to re-use a variable name in a deeper directory level, sources could end up being added to unintended targets and CMake won’t typically issue any sort of diagnostic message, since it won’t know you didn’t intend to do that.

But perhaps the bigger drawback of using variables is that it precludes having the CMake target defined when descending into the subdirectories. This in turn means that subdirectories cannot directly call target_compile_definitions(), target_compile_options(), target_include_directories() or target_link_libraries() either. In order to associate compiler flags, options, header search paths and other libraries to be linked, more variables have to be defined to pass this information back up to the top level. Extra care has to be taken to properly handle quoting when doing this too. If you want to take full advantage of the PUBLIC, PRIVATE and INTERFACE capabilities of these various target_... commands too, the number of variables required just for one target alone already starts getting a bit silly. You can imagine the explosion of variables if many targets are defined throughout your project’s directory structure!

An example should help to highlight why target_sources() leads to much more robust and concise CMakeLists.txt files. Let’s say we have a project with two subdirectories foo and bar. The top level CMakeLists.txt file can be as simple as this:

cmake_minimum_required(VERSION 3.1)
project(MyProj)

add_library(myLib "")

include(foo/CMakeLists.txt) # See below for why we don't
include(bar/CMakeLists.txt) # use add_subdirectory() here

The empty quotes in the add_library() call are necessary because that command requires a list of source files, even if that list is empty. If there were sources to be added from this top level directory, they could be listed there.

Let’s now assume the source files in the foo subdirectory use features from some external third party library called barry. This requires myLib to link against the barry library. For the sake of discussion, let’s also assume we need to define a compiler symbol called USE_BARRY both when building myLib and also for any code that includes headers from myLib. The CMakeLists.txt file within the foo subdirectory might then look something like this:

target_sources(myLib
    PRIVATE
        "${CMAKE_CURRENT_LIST_DIR}/foo.cpp"
        "${CMAKE_CURRENT_LIST_DIR}/foo_p.cpp"
        "${CMAKE_CURRENT_LIST_DIR}/foo_p.h"
    PUBLIC
        "${CMAKE_CURRENT_LIST_DIR}/foo.h"
)

find_library(BARRY_LIB barry)
target_link_libraries(myLib PUBLIC ${BARRY_LIB})
target_compile_definitions(myLib PUBLIC USE_BARRY)
target_include_directories(myLib PUBLIC "${CMAKE_CURRENT_LIST_DIR}")

In the target_sources() command above, each of the sources is prefixed by the directory of the CMakeLists.txt file. This is necessary because the list of source files that are added to a target are interpreted by CMake as being relative to the directory in which the target is defined (i.e. where the add_library() call is made). If we listed the sources in subdirectories like foo without any path, those sources would not be found. We could list the sources with a path relative to where add_library() is called, but then we would end up reproducing the directory hierarchy throughout our CMakeLists.txt files again. By using the CMAKE_CURRENT_LIST_DIR variable, we are free to rename, move, etc. our subdirectories and the CMakeLists.txt file doesn’t have to be updated.

Those readers who are fairly familiar with CMake may be wondering why we didn’t use the more typical CMAKE_CURRENT_SOURCE_DIR instead of CMAKE_CURRENT_LIST_DIR and also why we chose to include() the subdirectories’ CMakeLists.txt files instead of using the more typical add_subdirectory(). The reasons are the same for both. Unfortunately, target_link_libraries() will only allow you to add libraries to a target if that target is defined in the same directory as the target_link_libraries() command. Calling add_subdirectory() changes CMake’s notion of the current directory, whereas include() doesn’t. Thus, from CMake’s perspective, using include() makes it look like all the various subdirectories’ CMakeLists.txt files were all inlined in the top level, but add_subdirectory() doesn’t. All the other target_... commands do not have this restriction and will work with either arrangement. Using include() instead of add_subdirectory() is slightly less convenient since you have to use full paths to sources anywhere you reference them, even when adding sources to a target defined in the same directory, but that’s the simplest workaround if you need to use target_link_libraries() to add a library to a target defined in a different directory.

In the above example for foo, note that .h header files were specified as sources, not just the .cpp implementation files. Headers listed as sources don’t get compiled directly on their own, but the effect of adding them is for the benefit of IDE generators like Visual Studio, Xcode or Qt Creator. This causes those headers to be listed in the project’s file list within the IDE, even if no source file refers to it via #include. This can make those headers easier to find during development and potentially aid things like refactoring functionality, etc.

The PRIVATE and PUBLIC keywords specify where those corresponding sources should be used. PRIVATE simply means those sources should only be added to myLib, whereas PUBLIC means those sources should be added to myLib and to any target that links to myLib. An INTERFACE keyword can be used for sources that should not be added to myLib but should be added to anything that links to myLib. Headers that are part of a public interface for a library will often be listed in a PUBLIC or INTERFACE section to improve the generated IDE projects, as mentioned above. Private headers not part of the API can be listed under the PRIVATE entries of target_sources() so that they don’t appear under IDE targets. There are also some less common cases where some files (eg resources, images, data files) may need to be compiled directly into targets linking against a library for them to be found at runtime. Listing such sources as PUBLIC or INTERFACE can help address such situations.

The same meaning for PRIVATE, PUBLIC and INTERFACE apply to the other target_... commands too. The above example shows how easy it is to specify that myLib and any target that links to it also needs to link to the barry library. Similarly, with just that one target_compile_definitions() call, both myLib and anything linking against it will have the USE_BARRY symbol defined. Any other source file in another directory that needs to #include the foo.h header will also be able to find it thanks to the target_include_directories() command adding the foo subdirectory to the header search path for both myLib and anything linking to it.

To illustrate just how powerful these target_... commands are, let’s consider what the CMakeLists.txt file for the bar subdirectory might look like. In this case, let’s just assume bar needs to add a few sources files and that some of bar‘s sources or headers will include foo.h.

target_sources(myLib
    PRIVATE
        "${CMAKE_CURRENT_LIST_DIR}/bar.cpp"
        "${CMAKE_CURRENT_LIST_DIR}/bar.h"
        "${CMAKE_CURRENT_LIST_DIR}/gumby.cpp"
        "${CMAKE_CURRENT_LIST_DIR}/gumby.h"
)

Note the complete absence of anything other than simply listing the source files. All the work was done in the foo directory already, so there’s nothing left for us to do here. This highlights one of the biggest advantages of using target_sources(), namely that dependencies can be listed right where they are most relevant and all other directories don’t need to care. This localisation of dependency details leads to much more robust and more concise CMakeLists.txt files throughout a project. Without target_sources(), we would not be able to use target_compile_definitions(), target_compile_options(), target_include_directories() or target_link_libraries() in this way because the CMake target myLib would not be defined when we descend into each subdirectory.

Key points

Taking a step back, what target_sources() is doing for us is to remove the need for variables by allowing us to use CMake targets directly. This gives us these key advantages:

  • It allows the CMake target to be defined early, which in turn enables calling the various other target_... commands in any of the subdirectories pulled in with add_subdirectory() after the target is defined.
  • Subdirectories cannot inadvertently add sources to the wrong targets.
  • Dependency information can be fully and robustly defined at the point where those dependencies are introduced. The PRIVATE, PUBLIC and INTERFACE keywords give precise control over the nature of those dependencies and they also promote better integration with IDE environments able to take advantage of this information.

Remember, however, that you need to use include() rather than add_subdirectory() if you want to be able to use target_link_libraries() to add a library to a dependency for a target defined in a parent directory.

The target_sources() command also has one unique advantage over and above anything that a variable-based approach possesses, namely that it allows additional sources to be added to targets regardless of where they were defined (except for imported targets). This is especially useful when code from an external project is being incorporated into a build with add_subdirectory() or include() (e.g. see an earlier article showing how to incorporate GoogleTest directly into your main build). This can be used to add headers, images, etc. from the external project without affecting the way the target is built. For the really adventurous, you could even potentially use this technique to add your own implementation for a weak symbol such that your implementation overrides the one that the external project’s target would normally use. This may prove useful during testing or to provide a more efficient implementation of a specific function, etc.

Previous

Member Function Overloading: Choices You Didn’t Know You Had

Next

Scripting CMake builds

19 Comments

  1. forgge

    Hi! Are you using some kind of new version of cmake? I found that I can’t use target_*() set of commands in subdirs. I’m getting errors:

    CMake Error at targets/cc32xx/CMakeLists.txt:89 (target_link_libraries):
    Attempt to add link library “cc32xx_nonos” to target “demo_client” which is
    not built in this directory.

    cc32xx_nonos is a imported library in cc32xx subdir.
    demo_client is an application inside top-level CMakeList

    • Indeed you are correct, I thought I had checked that. I’ve modified the article to highlight how target_link_libraries() is treated differently by CMake. The short version is that you need to use include() instead of add_subdirectory() if you want to be able to use target_link_libraries() in a subdirectory for a target defined in a parent directory. Thanks for reporting the problem.

  2. Misairu

    Using include() instead of add_subdirectory() is slightly less convenient since you have to use full paths to sources anywhere you reference them, even when adding sources to a target defined in the same directory, but that’s the simplest workaround if you need to use target_link_libraries() to add a library to a target defined in a different directory.

    I don’t get this. If you use add_library() in the subdirectory’s CMakeLists.txt, you can then use that library inside parent’s CMakeLists.txt with target_link_libraries(). Why wound that be a problem? I’m using cmake 3.5.1

    • Let’s say you have a subdirectory sub and you add it from its parent using add_subdirectory(sub). Inside sub, you call add_library(foo ...). From the parent of sub, you cannot then call target_link_libraries(foo ...), you can only call that within the sub directory. If, however, you added sub using include(sub/CMakeLists.txt), then you can call target_link_libraries(foo ...) from the parent because sub and the parent are in the same directory scope as far as CMake is concerned. Calling include() does not introduce a new directory scope, whereas add_subdirectory() does.

      • Misairu

        But…I currently using cmake like what I said, and that is why I think that is not a problem.

        Root CMakeLists.txt:

        # ...
        set(EXTRA_LIBS ${EXTRA_LIBS} c01_hello)
        set(EXTRA_INCLUDE ${EXTRA_INCLUDE} chapter01)
        include_directories("${PROJECT_BINARY_DIR}")
        include_directories("${EXTRA_INCLUDE}")
        add_subdirectory(chapter01)

        target_link_libraries(<some executable here> ${EXTRA_LIBS})

        ...

        Sub CMakeLists.txt in chapter01:

        add_library(c01_hello SHARED HelloWorld.c)

        Or do I did this with some consequences?

        • Using your latest example, what I’m trying to highlight is that <some executable here> cannot be co1_hello. You cannot define co1_hello in one directory scope and then call target_link_libraries(co1_hello ...) in a different scope.

          • Misairu

            Okay, now I get it. Thank you.

          • Why would you call target_link_libraries(co1_hello …) in a different scope?
            That is a bad practice. It goes against modularity. You should aime to have your co1_hello entirely defined in the chapter01 CMakeLists.txt.
            Note that if you are worried that you may want to link co1_hello with something imported in the parent, you do not have worries there either, it will work.
            Example. I have a HPC engineering code I work on. It relies on the Eigen3 library and most sub-libraries that compose the main software need to include the Eigen3 directory.

            The main CMakeLists.txt has:

            […]
            find_package(Eigen3 3.3 REQUIRED NO_MODULES)
            […]
            add_subdirectory(SomeLibrary)
            […]
            add_executable(main_exec Main.cpp)
            […]

            target_link_library(main_exec some_lib […])

            And in SomeLibrary, CMakeLists.txt has:

            add_library(some_lib)
            target_sources(some_lib
            PRIVATE some_source1.cpp some_source2.cpp
            PUBLIC some_header.h)
            target_link_library(some_lib Eigen3::Eigen)

            ====================================
            Everything works fine and is modularized nicely.

          • “Why would you call target_link_libraries(co1_hello …) in a different scope?
            That is a bad practice. It goes against modularity. You should aime to have your co1_hello entirely defined in the chapter01 CMakeLists.txt.”

            This was just an example for illustration. For a real world example, consider a library or executable that has many source files and where some functionality is optional and/or depends on the availability of some external toolkit. Code related to such an optional feature can be put in its own sub directory and conditionally included in the library or executable. That sub directory can hold all the logic related to that feature, including any external libraries that need to be linked in. This is good modularisation since it localised the logic instead of polluting the main CMakeLists.txt file.

            It isn’t always possible or desirable to split out such an optional feature to its own library (eg to ensure aspects of the implementation are not revealed by exported symbol names or similar concerns). Thus, being able to incorporate it directly into the main library or executable could be a requirement. Being able to still isolate everything related to that feature in its own sub directory can help keep things organised and easy to manage. I work on projects which do exactly this and without the capabilities target_sources() provides, it would be much harder and messier.

  3. Thanks, fascinating.

    How are the headers for library BARRY found? I suppose lib BARRY is installed properly and the headers installed in a well-known (to cmake) location?

    I am trying to use a library that is not installed, just in a separate project. I need to do more reading and can probably figure it out.

    • You need to distinguish between the build tree and an installed set of files. In a build tree, assuming barry is built just like the rest of the project, then the barry target should define any PUBLIC include directories, compiler flags, etc. that consumers would need. This should take care of ensuring barry‘s headers can be found. Things are a little less clear-cut for an installed case because it’s really up to you where you install things, and then it’s up to whatever wants to use those installed things how they bring them into their own build. If you’ve used CMake’s install(EXPORT) functionality to provide support for other projects to use your installed package via find_package(), then that’s ideal and will behave very similarly to the build tree case. I’ve skipped over a lot of detail in that brief description, but that’s at least touching on a few of the important considerations.

      It also seems like your situation is somewhere in between. The library you want to use is not part of your main project but is also not installed somewhere. I guess you’ve just got its build tree? Maybe consider looking into the export() command from within the other library’s project as a way to make its build tree available to your main project (just one choice, there would be various other ways, each with their own advantages and disadvantages).

  4. Sébastien

    Thanks for the clear explanation. However, how can you do so and keep the same hierarchy in an IDE and on your filesystem? All the ways I have found on Google to keep the hierarchy in an IDE rely at some point on a variable listing all source files that your method avoid so neatly.

    • I think different IDEs group files differently by default. If you are using CMake 3.8 or later though, the source_group(TREE) command probably does what you want. You may be able to get the list of sources from the SOURCES target property but you’d need to get that list after you’ve added all the sources to the target.

      • Sébastien

        Thanks! I am somewhat new to CMake and didn’t know about properties; I’m indeed able to recover all the source files from a target using get_property(v TARGET my_target PROPERTY SOURCES).

  5. Ying

    This is useful. Now I am using target_sources in myLib, with a PUBLIC hello.h file. My other app testApp links to myLib. When cmake parses add_executable(testApp, “”), it reports error: “Cannot find source file hello.h”. So how to correctly understand and use the PUBLIC files in target_sources?

    • target_sources() doesn’t automatically convert files to absolute. If you list a file as a PUBLIC or INTERFACE, you should use an absolute path to avoid precisely the problem you’ve encountered. Note, however, that listing files as PUBLIC or INTERFACE should not be done if you intend to install/export the target, since the path when installed will be different.

  6. Thank you very much for the amazing post! Why do you use CMAKE_CURRENT_SOURCE_DIR for the bar catalog but not CMAKE_CURRENT_LIST_DIR as for the foo directory?

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Powered by WordPress & Theme by Anders Norén

%d bloggers like this: