Professional CMake:

A Practical Guide

Go beyond trivial examples and learn how to use CMake effectively with practical advice direct from a CMake co-maintainer

 

Enhanced source file handling with target_sources()

Updated December 2018: Parts of this article have been reworked to account for improvements made with the CMake 3.13.0 release. Key updates are noted within the article.

In all but trivial CMake projects, it is common to find targets built from a large number of source files. These files may be distributed across various subdirectories, which may themselves be nested multiple levels deep. In such projects, traditional approaches usually either list all source files at the top-most level or build up the list of source files in a variable and pass that to add_library(), add_executable(), etc. With CMake 3.1, a new command target_sources() was introduced which provides the missing piece among the various target_... commands. While the CMake documentation succintly describes what target_sources() does, it fails to highlight just how useful the new command is and why it promotes better CMake projects:

  • It can lead to cleaner and more concise CMakeLists.txt project files.
  • Dependency information can be specified closer to where the actual dependencies exist in the directory hierarchy.
  • Source files gain the ability to become part of a target’s interface.
  • Source files can be added to third party project targets without having to modify the third party project files.

Life Before target_sources()

Typically, developers first learn CMake in a very simple manner, defining a target by listing the source files directly in the add_executable() or add_library() command itself. Eg:

add_executable(myApp src1.cpp src2.cpp)

When the number of source files grows large and they get distributed over a number of subdirectories, possibly nested to multiple levels, this quickly becomes unwieldly. It also results in having to repeat the directory structure, which reduces the benefit of structuring source files into directories in the first place.

The logical improvement many developers then make is to build up the list of source files in a variable as each subdirectory is pulled in via include(). Then after all the subdirectories have been included, add_executable() or add_library() is called, but this time passing just the variable instead of an explicit list of files. The top level CMakeLists.txt file then looks something like this:

# The name of the included file could be anything,
# it doesn't have to be called CMakeLists.txt
include(foo/CMakeLists.txt)
include(bar/CMakeLists.txt)

add_executable(myApp ${myApp_SOURCES})

with the subdirectory files structured something like this:

list(APPEND myApp_SOURCES
    ${CMAKE_CURRENT_LIST_DIR}/foo.cpp
    ${CMAKE_CURRENT_LIST_DIR}/foo_p.cpp
)

This allows each subdirectory to define just the sources it provides and to delegate any further nested subdirectories with another include(). It also keeps the top level CMakeLists.txt file quite small and the CMakeLists.txt in each subdirectory also tends to be reasonably uncomplicated and focussed just on the things in that directory.

As an alternative to explicitly building up a list of source files in a variable, some developers instead choose to let CMake find the source files and generate the contents of that variable automatically with the file(GLOB_RECURSE ...) command. While at first this may seem very attractive for its simplicity, this technique has a number of drawbacks and is actively discouraged by the CMake documentation. Nevertheless, it is often used by developers new to CMake until they experience first hand the problems the technique introduces.

target_sources(): All The Advantages Without The Drawbacks

It may not be immediately obvious what the down sides are to the approach above where the source files are built up in a variable and then that variable is passed to an add_library() or add_executable() call at the top level CMakeLists.txt file. One drawback is that the variables themselves are not a particularly robust way to record the source file list. For example, if many targets are being built up throughout a directory heirarchy, then the number and naming of variables can get out of hand. This can be somewhat addressed by sticking to some kind of naming convention associated with the target a variable is used with, but this relies on all developers knowing and adhering to that convention. Furthermore, if a developer inadvertently tries to re-use a variable name in a deeper directory level, sources could end up being added to unintended targets and CMake won’t typically issue any sort of diagnostic message, since it won’t know you didn’t intend to do that.

But perhaps the bigger drawback of using variables is that it precludes having the CMake target defined when descending into the subdirectories. This in turn means that subdirectories cannot directly call target_compile_definitions(), target_compile_options(), target_include_directories() or target_link_libraries() either. In order to associate compiler flags, options, header search paths and other libraries to be linked, more variables have to be defined to pass this information back up to the top level. Extra care has to be taken to properly handle quoting when doing this too. If you want to take full advantage of the PUBLIC, PRIVATE and INTERFACE capabilities of these various target_... commands too, the number of variables required just for one target alone already starts getting a bit silly. You can imagine the explosion of variables if many targets are defined throughout your project’s directory structure!

NOTE: The advice and examples below have been updated from the original article to account for new capabilities added in CMake 3.13.0

An example should help to highlight why target_sources() leads to much more robust and concise CMakeLists.txt files. Let’s say we have a project with two subdirectories foo and bar. The top level CMakeLists.txt file can be as simple as this:

cmake_minimum_required(VERSION 3.13)
project(MyProj)

add_library(myLib "")

add_subdirectory(foo)
add_subdirectory(bar)

The empty quotes in the add_library() call are necessary because that command requires a list of source files, even if that list is empty. If there were sources to be added from this top level directory, they could be listed there.

Let’s now assume the source files in the foo subdirectory use features from some external third party library called barry. This requires myLib to link against the barry library. For the sake of discussion, let’s also assume we need to define a compiler symbol called USE_BARRY both when building myLib and also for any code that includes headers from myLib. Assuming a minimum CMake version of 3.13.0 or later, the CMakeLists.txt file within the foo subdirectory might then look something like this:

target_sources(myLib
    PRIVATE
        foo.cpp
        foo_p.cpp
        foo_p.h
    PUBLIC
        foo.h
)

find_library(BARRY_LIB barry)
# This call requires CMake 3.13 or later, see next section
target_link_libraries(myLib PUBLIC ${BARRY_LIB})

target_compile_definitions(myLib PUBLIC USE_BARRY)
target_include_directories(myLib PUBLIC ${CMAKE_CURRENT_LIST_DIR})

In the above example, note that .h header files were specified as sources too, not just the .cpp implementation files. Headers listed as sources don’t get compiled directly on their own, but the effect of adding them is for the benefit of IDE generators like Visual Studio, Xcode, Qt Creator, etc. This causes those headers to be listed in the project’s file list within the IDE, even if no source file refers to it via #include. This can make those headers easier to find during development and potentially aid things like refactoring functionality, etc.

The PRIVATE and PUBLIC keywords specify where those corresponding sources should be used. PRIVATE simply means those sources should only be added to myLib, whereas PUBLIC means those sources should be added to myLib and to any target that links to myLib. An INTERFACE keyword can be used for sources that should not be added to myLib but should be added to anything that links to myLib. Headers that are part of a public interface for a library might be listed in a PUBLIC or INTERFACE section to improve the generated IDE projects, as mentioned above. Private headers not part of the API can be listed under the PRIVATE entries of target_sources() so that they don’t appear under IDE targets other than the target they are private to. There are also some less common cases where some files (eg resources, images, data files) may need to be compiled directly into targets linking against a library for them to be found at runtime. Listing such sources as PUBLIC or INTERFACE can help address such situations.

The same meaning for PRIVATE, PUBLIC and INTERFACE apply to the other target_...() commands too. The above example shows how easy it is to specify that myLib and any target that links to it also needs to link to the barry library. Similarly, with just that one target_compile_definitions() call, both myLib and anything linking against it will have the USE_BARRY symbol defined. Any other source file in another directory that needs to #include the foo.h header will also be able to find it thanks to the target_include_directories() command adding the foo subdirectory to the header search path for both myLib and anything linking to it.

To illustrate just how powerful these target_...() commands are, let’s consider what the CMakeLists.txt file for the bar subdirectory might look like. In this case, let’s just assume bar needs to add a few sources files and that some of bar‘s sources or headers will include foo.h.

target_sources(myLib
    PRIVATE
        bar.cpp
        bar.h
        gumby.cpp
        gumby.h
)

Note the complete absence of anything other than simply listing the source files. All the work was done in the foo directory already, so there’s nothing left for us to do here. This highlights one of the biggest advantages of using target_sources(), namely that dependencies can be listed right where they are most relevant and all other directories don’t need to care. This localisation of dependency details leads to much more robust and more concise CMakeLists.txt files throughout a project. Without target_sources(), we would not be able to use target_compile_definitions(), target_compile_options(), target_include_directories() or target_link_libraries() in this way because the CMake target myLib would not be defined when we descend into each subdirectory.

Supporting CMake 3.12 And Earlier

The above comments about using CMake 3.13.0 or later relate to restrictions that were removed in that release. Prior to 3.13.0, target_link_libraries() could only be called on a target that was created in the same directory scope. This means that in the example for the foo subdirectory, the target_link_libraries(myLib ...) call would cause an error with CMake 3.12 or earlier because the myLb target was created in the parent scope. None of the other target_...() commands have ever had this restriction, only target_link_libraries(). We will address this point shortly.

Another change in CMake 3.13.0 relates to how target_sources() interprets relative paths to source files. In CMake 3.12 or earlier, relative paths were treated as being relative to the target to which sources were being added. This was unintuitive, so CMake 3.13.0 changed the default behavior to treating relative paths as being relative to the current source directory instead. If the project sets 3.13.0 as its minimum CMake version requirement, it automatically gets the new behavior by default.

For projects that need to support CMake 3.12 or earlier, they can use absolute paths to source files to avoid the change in behavior and to avoid any policy warnings. For example:

target_sources(myLib
    PRIVATE
        ${CMAKE_CURRENT_LIST_DIR}/foo.cpp
        ${CMAKE_CURRENT_LIST_DIR}/foo_p.cpp
        ${CMAKE_CURRENT_LIST_DIR}/foo_p.h
    PUBLIC
        ${CMAKE_CURRENT_LIST_DIR}/foo.h
)

This is less convenient and less readable, so it may be helpful to define a helper function to give something close to the new behavior, but which also works for earlier CMake versions. CMake has a strong requirement on preserving backward compatibility, so the change in behavior of how relative paths are treated is guarded by a policy, CMP0076. We can take advantage of that in the helper function:

# NOTE: This helper function assumes no generator expressions are used
#       for the source files
function(target_sources_local target)
  if(POLICY CMP0076)
    # New behavior is available, so just forward to it by ensuring
    # that we have the policy set to request the new behavior, but
    # don't change the policy setting for the calling scope
    cmake_policy(PUSH)
    cmake_policy(SET CMP0076 NEW)
    target_sources(${target} ${ARGN})
    cmake_policy(POP)
    return()
  endif()

  # Must be using CMake 3.12 or earlier, so simulate the new behavior
  unset(_srcList)
  get_target_property(_targetSourceDir ${target} SOURCE_DIR)

  foreach(src ${ARGN})
    if(NOT src STREQUAL "PRIVATE" AND
       NOT src STREQUAL "PUBLIC" AND
       NOT src STREQUAL "INTERFACE" AND
       NOT IS_ABSOLUTE "${src}")
      # Relative path to source, prepend relative to where target was defined
      file(RELATIVE_PATH src "${_targetSourceDir}" "${CMAKE_CURRENT_LIST_DIR}/${src}")
    endif()
    list(APPEND _srcList ${src})
  endforeach()
  target_sources(${target} ${_srcList})
endfunction()

Now we can call the above helper function just like the builtin command and get the CMake 3.13 behavior even with CMake 3.12 or earlier:

target_sources_local(myLib
    PRIVATE
        foo.cpp
        foo_p.cpp
        foo_p.h
    PUBLIC
        foo.h
)

When using CMake 3.12 or earlier, working around the restriction with target_link_libraries() is harder. The choices are either to move the target_link_libraries() call up to the same directory in which the target is defined, or avoid creating new directory scopes by using include() instead of add_subdirectory(). The second of these options would only require changing the top level CMakeLists.txt file to something like the following (which is the original method suggested by this article before it was updated for CMake 3.13.0):

cmake_minimum_required(VERSION 3.1)
project(MyProj)

add_library(myLib "")

# Using include() avoids creating a new directory scope, so these directories
# are able to call target_link_libraries(myLib ...)
include(foo/CMakeLists.txt)
include(bar/CMakeLists.txt)

The target_sources_local() helper function is defined in such a way that it will work inside foo or any other directory, regardless of whether we use add_subdirectory() or include().

Most developers find add_subdirectory() more natural and it does tend to give more intuitive handling of variables like CMAKE_CURRENT_SOURCE_DIR, CMAKE_CURRENT_BINARY_DIR, etc. Therefore, if the subdirectories don’t need to call target_link_libraries(), prefer to use the add_subdirectory() approach rather than the above include() workaround.

Complications For Installing

Specifying PRIVATE sources is relatively easy and has few difficulties. The location of each source file is clear and only needs to be considered within the build. Any PUBLIC or INTERFACE sources give rise to additional factors which must be considered. Within the build, paths to non-private sources are handled just like private ones, but when the project is installed, non-private paths have to make sense not only in the project’s own build, but also in the build of anything consuming the installed project. The paths used for the project’s own build will be specific to the machine on which that build is performed and to the directory in which it is built. Once installed, those directories will likely not be accessible, so the paths would need to be replaced with something else that makes sense for the installed set of files. This is made possible using the BUILD_INTERFACE and INSTALL_INTERFACE generator expressions. The following example shows how to provide different paths for the same file in the two different contexts:

target_sources(myHeaderOnly
    PUBLIC
        $<BUILD_INTERFACE:${CMAKE_CURRENT_LIST_DIR}/algo.h>
        $<INSTALL_INTERFACE:include/algo.h>
)
install(FILES algo.h DESTINATION include)

While the above solves the build/install differences, it also has the drawback of requiring us to spell out every BUILD_INTERFACE path as absolute once again. We can no longer use our target_sources_local() helper function that we defined earlier. This is the cost of supporting the installation of non-private source files. Projects should weigh up the loss of convenience and added complexity against the expected benefits to be gained by making sources non-private.

Key Points

Taking a step back, what target_sources() is doing for us is to remove the need for variables by allowing us to use CMake targets directly. This gives us these key advantages:

  • It allows the CMake target to be defined early, which in turn enables calling the various other target_... commands in any of the subdirectories pulled in with add_subdirectory() after the target is defined.
  • Subdirectories cannot inadvertently add sources to the wrong targets.
  • Dependency information can be fully and robustly defined at the point where those dependencies are introduced. The PRIVATE, PUBLIC and INTERFACE keywords give precise control over the nature of those dependencies and they also promote better integration with IDE environments able to take advantage of this information.

The following points should also be kept in mind:

  • Non-private sources require much more verbose and less convenient syntax, so consider whether the gains from making them non-private are worth it.
  • If you need to support CMake 3.12 or older, you will need to either pull up any target_link_libraries() calls to the same directory as the target they operate on, or else use include() rather than add_subdirectory() to avoid introducing a new directory scope. Prefer the former where possible since it is likely to be more intuitive for developers.

The target_sources() command also has one unique advantage over and above anything that a variable-based approach possesses, namely that it allows additional sources to be added to targets regardless of where they were defined (except for imported targets). This is especially useful when code from an external project is being incorporated into a build with add_subdirectory() or include() (e.g. see an earlier article showing how to incorporate GoogleTest directly into your main build). This can be used to add headers, images, etc. from the external project without affecting the way the target is built. For the really adventurous, you could even potentially use this technique to add your own implementation for a weak symbol such that your implementation overrides the one that the external project’s target would normally use. This may prove useful during testing or to provide a more efficient implementation of a specific function, etc.


Get the book for more CMake content

25 thoughts on “Enhanced source file handling with target_sources()”

  1. Hi! Are you using some kind of new version of cmake? I found that I can’t use target_*() set of commands in subdirs. I’m getting errors:

    CMake Error at targets/cc32xx/CMakeLists.txt:89 (target_link_libraries):
    Attempt to add link library “cc32xx_nonos” to target “demo_client” which is
    not built in this directory.

    cc32xx_nonos is a imported library in cc32xx subdir.
    demo_client is an application inside top-level CMakeList

    • Indeed you are correct, I thought I had checked that. I’ve modified the article to highlight how target_link_libraries() is treated differently by CMake. The short version is that you need to use include() instead of add_subdirectory() if you want to be able to use target_link_libraries() in a subdirectory for a target defined in a parent directory. Thanks for reporting the problem.

  2. Using include() instead of add_subdirectory() is slightly less convenient since you have to use full paths to sources anywhere you reference them, even when adding sources to a target defined in the same directory, but that’s the simplest workaround if you need to use target_link_libraries() to add a library to a target defined in a different directory.

    I don’t get this. If you use add_library() in the subdirectory’s CMakeLists.txt, you can then use that library inside parent’s CMakeLists.txt with target_link_libraries(). Why wound that be a problem? I’m using cmake 3.5.1

    • Let’s say you have a subdirectory sub and you add it from its parent using add_subdirectory(sub). Inside sub, you call add_library(foo ...). From the parent of sub, you cannot then call target_link_libraries(foo ...), you can only call that within the sub directory. If, however, you added sub using include(sub/CMakeLists.txt), then you can call target_link_libraries(foo ...) from the parent because sub and the parent are in the same directory scope as far as CMake is concerned. Calling include() does not introduce a new directory scope, whereas add_subdirectory() does.

      • But…I currently using cmake like what I said, and that is why I think that is not a problem.

        Root CMakeLists.txt:

        # ...
        set(EXTRA_LIBS ${EXTRA_LIBS} c01_hello)
        set(EXTRA_INCLUDE ${EXTRA_INCLUDE} chapter01)
        include_directories("${PROJECT_BINARY_DIR}")
        include_directories("${EXTRA_INCLUDE}")
        add_subdirectory(chapter01)

        target_link_libraries(<some executable here> ${EXTRA_LIBS})

        ...

        Sub CMakeLists.txt in chapter01:

        add_library(c01_hello SHARED HelloWorld.c)

        Or do I did this with some consequences?

        • Using your latest example, what I’m trying to highlight is that <some executable here> cannot be co1_hello. You cannot define co1_hello in one directory scope and then call target_link_libraries(co1_hello ...) in a different scope.

          • Why would you call target_link_libraries(co1_hello …) in a different scope?
            That is a bad practice. It goes against modularity. You should aime to have your co1_hello entirely defined in the chapter01 CMakeLists.txt.
            Note that if you are worried that you may want to link co1_hello with something imported in the parent, you do not have worries there either, it will work.
            Example. I have a HPC engineering code I work on. It relies on the Eigen3 library and most sub-libraries that compose the main software need to include the Eigen3 directory.

            The main CMakeLists.txt has:

            […]
            find_package(Eigen3 3.3 REQUIRED NO_MODULES)
            […]
            add_subdirectory(SomeLibrary)
            […]
            add_executable(main_exec Main.cpp)
            […]

            target_link_library(main_exec some_lib […])

            And in SomeLibrary, CMakeLists.txt has:

            add_library(some_lib)
            target_sources(some_lib
            PRIVATE some_source1.cpp some_source2.cpp
            PUBLIC some_header.h)
            target_link_library(some_lib Eigen3::Eigen)

            ====================================
            Everything works fine and is modularized nicely.

          • “Why would you call target_link_libraries(co1_hello …) in a different scope?
            That is a bad practice. It goes against modularity. You should aime to have your co1_hello entirely defined in the chapter01 CMakeLists.txt.”

            This was just an example for illustration. For a real world example, consider a library or executable that has many source files and where some functionality is optional and/or depends on the availability of some external toolkit. Code related to such an optional feature can be put in its own sub directory and conditionally included in the library or executable. That sub directory can hold all the logic related to that feature, including any external libraries that need to be linked in. This is good modularisation since it localised the logic instead of polluting the main CMakeLists.txt file.

            It isn’t always possible or desirable to split out such an optional feature to its own library (eg to ensure aspects of the implementation are not revealed by exported symbol names or similar concerns). Thus, being able to incorporate it directly into the main library or executable could be a requirement. Being able to still isolate everything related to that feature in its own sub directory can help keep things organised and easy to manage. I work on projects which do exactly this and without the capabilities target_sources() provides, it would be much harder and messier.

  3. Thanks, fascinating.

    How are the headers for library BARRY found? I suppose lib BARRY is installed properly and the headers installed in a well-known (to cmake) location?

    I am trying to use a library that is not installed, just in a separate project. I need to do more reading and can probably figure it out.

    • You need to distinguish between the build tree and an installed set of files. In a build tree, assuming barry is built just like the rest of the project, then the barry target should define any PUBLIC include directories, compiler flags, etc. that consumers would need. This should take care of ensuring barry‘s headers can be found. Things are a little less clear-cut for an installed case because it’s really up to you where you install things, and then it’s up to whatever wants to use those installed things how they bring them into their own build. If you’ve used CMake’s install(EXPORT) functionality to provide support for other projects to use your installed package via find_package(), then that’s ideal and will behave very similarly to the build tree case. I’ve skipped over a lot of detail in that brief description, but that’s at least touching on a few of the important considerations.

      It also seems like your situation is somewhere in between. The library you want to use is not part of your main project but is also not installed somewhere. I guess you’ve just got its build tree? Maybe consider looking into the export() command from within the other library’s project as a way to make its build tree available to your main project (just one choice, there would be various other ways, each with their own advantages and disadvantages).

  4. Thanks for the clear explanation. However, how can you do so and keep the same hierarchy in an IDE and on your filesystem? All the ways I have found on Google to keep the hierarchy in an IDE rely at some point on a variable listing all source files that your method avoid so neatly.

    • I think different IDEs group files differently by default. If you are using CMake 3.8 or later though, the source_group(TREE) command probably does what you want. You may be able to get the list of sources from the SOURCES target property but you’d need to get that list after you’ve added all the sources to the target.

      • Thanks! I am somewhat new to CMake and didn’t know about properties; I’m indeed able to recover all the source files from a target using get_property(v TARGET my_target PROPERTY SOURCES).

  5. This is useful. Now I am using target_sources in myLib, with a PUBLIC hello.h file. My other app testApp links to myLib. When cmake parses add_executable(testApp, “”), it reports error: “Cannot find source file hello.h”. So how to correctly understand and use the PUBLIC files in target_sources?

    • target_sources() doesn’t automatically convert files to absolute. If you list a file as a PUBLIC or INTERFACE, you should use an absolute path to avoid precisely the problem you’ve encountered. Note, however, that listing files as PUBLIC or INTERFACE should not be done if you intend to install/export the target, since the path when installed will be different.

      • I want to point out that, while your second sentence works for “included targets”, the third sentence is plain wrong. PUBLIC and INTERFACE can be used together with install/export but care needs to be taken. In particular, you want to utilize the generator expressions to properly reference the path in the right “phase”:

        $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}>$<INSTALL_INTERFACE:include/mylib>/relative/path/to/hello.h

        this properly resolves the path in both cases, noting that the file should be installed properly. My own experimentation shows that the export will assume the include directory.

        • Thanks for highlighting the inaccuracy. I’ve just done a fairly significant update to the article to account for changes in CMake 3.13, so while I was at it I also added a new section specifically discussing the issues around installation of non-private sources. Hopefully it makes this area a bit clearer.

    • I’ve already stated how you can fix this particular case, but I want to point out that it’s generally more preferred to place the include files in a separate directory, and reference them through target_include_directories. This has the benefit of simplifying the header install by simply saying “install(DIRECTORY include …)”. Doing all that will ensure your library can be installed and referenced externally no matter where it comes from.

      Now, I’m assuming you might be using an IDE which benefits from listing the header files among the sources. If that’s the case, you might want to look at source_group to list your headers in an IDE friendly way.

  6. Thank you very much for the amazing post! Why do you use CMAKE_CURRENT_SOURCE_DIR for the bar catalog but not CMAKE_CURRENT_LIST_DIR as for the foo directory?

    • Thanks, yes, the CMake 3.13 release removes the remaining minor inconveniences for target_sources() and really makes the approach discussed here that much better. Once the 3.13 reaches official status, I intend to update the article to account for these improvements.

      • Now that CMake 3.13 is officially out, I’ve updated the article to account for the changes in behavior. Not only can target_link_libraries() operate on targets defined elsewhere, the target_sources() command also handles relative paths to sources in a more intuitive manner.

Leave a Reply (all comments are moderated)

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: