Professional CMake:

A Practical Guide

Learn to use CMake effectively with practical advice from a CMake co-maintainer. You can also have the author work directly with your team!

Using ccache with CMake

Updated 1st February 2017

Working with very large C/C++ code bases will sometimes test your patience. Build times in particular can be a sore point, especially if the development team do not have a great understanding of how to minimise dependencies between source files, headers, etc. The problem gets worse if the developer frequently switches between branches of their source control system, resulting in source files often changing their contents and/or timestamps. The ccache tool aims to minimise that pain by caching compilation outputs (i.e. object files) and reusing them instead of compiling the source again if it gets a compilation request it has seen before and cached. It supports GCC or any compiler that looks like GCC (eg clang). When rebuilding a large project that ccache has mostly compiled before already, the time saving can be significant, even sometimes reducing many minutes down to seconds.

Getting ccache to work with CMake is not overly complicated, but the way to do it is not always obvious. This is especially true with Xcode builds. This article demonstrates how to set up a CMake project to use ccache with Unix Makefiles, Ninja or Xcode generators, with Xcode receiving special attention. The techniques presented do not require any changes to the host system. Specifically, no symlinks need to be set up for ccache, making it suitable for use in CI systems, etc. where the developer may not be in control of how/where ccache is installed.

Unix Makefiles and Ninja

Update: Don’t use RULE_LAUNCH_COMPILE as discussed in this section, use the CMAKE_<LANG>_COMPILER_LAUNCHER  approach discussed further below in Improved functionality from CMake 3.4 instead. The other aspects discussed here in this section are still relevant though.

A less well known feature of CMake is the global property called RULE_LAUNCH_COMPILE. When this is not empty, CMake inserts it before the compilation command, essentially wrapping the compiler that would have been run with whatever RULE_LAUNCH_COMPILE contains. This works for the Unix Makefiles generator with all CMake versions from 2.8.0 onwards and with the Ninja generator for CMake 3.4 and later. All other CMake generators simply ignore RULE_LAUNCH_COMPILE and build the project normally. By setting RULE_LAUNCH_COMPILE to our ccache command, compilation is re-routed via ccache and build times decrease when rebuilding previously compiled sources with the same settings and source contents.

This global property should be set as early as possible, ideally before even the project() command is called, since project() performs checking on the compilers to be used. When the property setting is placed before project(), these checks will confirm that the compiler can be launched through ccache and report problems early. With this in mind, a recommended structure for the top level CMakeLists.txt file would be to start it like this:

cmake_minimum_required(VERSION 2.8)

find_program(CCACHE_PROGRAM ccache)
if(CCACHE_PROGRAM)
  set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE "${CCACHE_PROGRAM}")
endif()

project(SomeProject)

The above will use ccache if it is available and fall back to normal non-ccache usage otherwise. Developers can opt-in simply by installing ccache and making it available on their PATH with no further setup required.

The studious developer may note that CMake also provides the RULE_LAUNCH_LINK global property which serves an analogous role for the linker. While it may be tempting to set this to route linker calls through ccache as well, there is no real benefit to doing so. This is because ccache doesn’t deal with link commands and will simply pass them through to the underlying linker anyway (this is mentioned as a limitation on the ccache website).

Xcode

Developers could reasonably expect that it should be possible to use ccache with the Xcode generator too. Afterall, the underlying compiler is going to be GCC-compatible, so ccache should work just fine. The CMake documentation for RULE_LAUNCH_COMPILE suggests we are out of luck though, stating the following:

Other generators ignore this property because their underlying build systems provide no hook to wrap individual commands with a launcher.

Happily for us, this is not quite true for Xcode. While we cannot use RULE_LAUNCH_COMPILE with the Xcode generator directly, we can use the support for CMAKE_XCODE_ATTRIBUTE_… variables added in CMake 3.1 to achieve the same effect, albeit with a little help. When Xcode projects contain user-defined build settings with the names CC and CXX, they override what Xcode uses as the compilers for C and C++ sources respectively. As far as I can tell, these user-defined settings are not documented by Apple, but it is straightforward enough to verify that they have the effect described. All we then need to do is to create two small scripts (one for C and the other for C++) which redirects the compile command to ccache and use the CC and CXX attributes to tell Xcode about the scripts. Glossing over a few details for the moment, the script would be something like this (shown for C, replace clang with clang++ for C++):

#!/bin/sh

exec ccache clang "$@"

Saving the scripts at the top of the source tree with names like ccache-c and ccache-cxx, the following lines can be added to the CMakeLists.txt file and that would be sufficient to get Xcode using ccache:

set(CMAKE_XCODE_ATTRIBUTE_CC  "${CMAKE_SOURCE_DIR}/ccache-c")
set(CMAKE_XCODE_ATTRIBUTE_CXX "${CMAKE_SOURCE_DIR}/ccache-cxx")

Annoyingly, it would appear that Xcode contains a bug related to how it chooses the linker when CC and CXX are defined. When building from within the Xcode IDE, setting CC and CXX results in the program specified for CC being used for linking as well, even for C++ projects. This results in linker errors because the C++ standard libraries are not included in the link. Oddly, building from the command line with xcodebuild yields different behaviour, with the clang or clang++ executables being called directly rather than using either of the programs specified in CC or CXX. Programs do build successfully from the command line as a result, but the different behaviour is inconsistent. Thankfully, a simple workaround is to also define the analogous LD and LDPLUSPLUS variables for specifying the linkers. This causes the specified linkers to be used both within the Xcode IDE and when building from the command line with xcodebuild, so the behaviour is correct and consistent in both cases. The CMake syntax to achieve this is simply:

set(CMAKE_XCODE_ATTRIBUTE_LD         "${CMAKE_SOURCE_DIR}/ccache-c")
set(CMAKE_XCODE_ATTRIBUTE_LDPLUSPLUS "${CMAKE_SOURCE_DIR}/ccache-cxx")

Or if you don’t want to route linking via the launchers:

set(CMAKE_XCODE_ATTRIBUTE_LD         "${CMAKE_C_COMPILER}")
set(CMAKE_XCODE_ATTRIBUTE_LDPLUSPLUS "${CMAKE_CXX_COMPILER}")

General implementation

The above explanation for Xcode glossed over a few details which should not be left unaddressed. The scripts contained hard-coded assumptions about where to find ccache and what underlying compiler to use. These are the sorts of things CMake is meant to handle for us more robustly. Indeed, we can do better by creating a pair of script template files and letting CMake populate tool paths for us. We can even make it general enough to support any compiler wrapper set in RULE_LAUNCH_COMPILE, not just ccache. First, create two files at the top of the source tree as follows:

launch-c.in:

#!/bin/sh

export CCACHE_CPP2=true
exec "${RULE_LAUNCH_COMPILE}" "${CMAKE_C_COMPILER}" "$@"

launch-cxx.in:

#!/bin/sh

export CCACHE_CPP2=true
exec "${RULE_LAUNCH_COMPILE}" "${CMAKE_CXX_COMPILER}" "$@"

Ignore the CCACHE_CPP2 lines for now, we will come to that shortly. First, note how the exec command now uses the RULE_LAUNCH_COMPILE and CMAKE_..._COMPILER variables to specify the compiler wrapper (ccache in our case) and compiler to use. In CMakeLists.txt, the configure_file() command is used to copy these two script templates to the build directory, substituting the two variables along the way. Making these modifications and putting in the support for Unix Makefiles and Ninja as well, the CMakeLists.txt file looks like this:

cmake_minimum_required(VERSION 2.8)

find_program(CCACHE_PROGRAM ccache)
if(CCACHE_PROGRAM)
  # Support Unix Makefiles and Ninja
  set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE "${CCACHE_PROGRAM}")
endif()

project(SomeProject)

get_property(RULE_LAUNCH_COMPILE GLOBAL PROPERTY RULE_LAUNCH_COMPILE)
if(RULE_LAUNCH_COMPILE AND CMAKE_GENERATOR STREQUAL "Xcode")
  # Set up wrapper scripts
configure_file(launch-c.in launch-c)
configure_file(launch-cxx.in launch-cxx)
execute_process(COMMAND chmod a+rx "${CMAKE_BINARY_DIR}/launch-c" "${CMAKE_BINARY_DIR}/launch-cxx" )

  # Set Xcode project attributes to route compilation and linking
  # through our scripts
  set(CMAKE_XCODE_ATTRIBUTE_CC "${CMAKE_BINARY_DIR}/launch-c")
  set(CMAKE_XCODE_ATTRIBUTE_CXX "${CMAKE_BINARY_DIR}/launch-cxx")
  set(CMAKE_XCODE_ATTRIBUTE_LD "${CMAKE_BINARY_DIR}/launch-c")
  set(CMAKE_XCODE_ATTRIBUTE_LDPLUSPLUS "${CMAKE_BINARY_DIR}/launch-cxx")
endif()

Note that the project() command comes after setting RULE_LAUNCH_COMPILE for Unix Makefiles and Ninja support, but before the calls to configure_file() to copy our launcher scripts. This is because we want CMake to work out the underlying C and C++ compilers, which is done by project(). If the configure_file() commands are put before project(), the CMAKE_C_COMPILER and CMAKE_CXX_COMPILER variables would not yet be set. Since Xcode can supply multiple toolchains, SDKs, etc., we should not simply assume clang can be found on the path, or indeed that the developer wants to use clang rather than gcc or some other compiler. The above arrangement still gives the developer the ability to select their compiler toolchain as before, even when cross-compiling to iOS, tvOS, etc. CMake will select the correct compiler and the re-routing scripts will use it rather than relying on some hard-coded location. A minor drawback is that the project() command is not testing the compiler through the launcher script when using Xcode, but the benefit of getting ccache working at all far outweighs that.

Returning to the scripts, they both contained the following line:

export CCACHE_CPP2=true

This is to handle certain pathalogical cases related to the preprocessor and enabling it is strongly recommended when using ccache with clang. Including this option may slightly increase build times for cache misses only (in the order of around 10%). The interested reader is directed to this article for a more complete discussion of the subject.

Improved functionality from CMake 3.4

From version 3.4, CMake also provides the <LANG>_COMPILER_LAUNCHER target properties (<LANG> may be C or CXX). The functionality is very similar to RULE_LAUNCH_COMPILE, but these properties can be set on a per-target basis and they also allow different launchers for different languages. If the same launchers should be used for all targets, then the CMAKE_<LANG>_COMPILER_LAUNCHER variables can be used to set the defaults for the corresponding target properties. Because the launchers can be language-specific, this is a better choice than RULE_LAUNCH_COMPILE if requiring CMake 3.4 or later is okay for your builds. The use of RULE_LAUNCH_COMPILE by projects is not recommended, as CMake expects to be able to use that for its own internal purposes in some situations.

For the most consistent behavior across all the CMake generators that support launchers, Ninja and Makefile generators can also be made to use the launcher scripts, as was done above for Xcode, rather than invoking ccache directly. Making this change, the launcher scripts and project file would then look something like this:

launch-c.in:

#!/bin/sh

# Xcode generator doesn't include the compiler as the
# first argument, Ninja and Makefiles do. Handle both cases.
if [[ "$1" = "${CMAKE_C_COMPILER}" ]] ; then
shift
fi

export CCACHE_CPP2=true
exec "${C_LAUNCHER}" "${CMAKE_C_COMPILER}" "$@"

launch-cxx.in:

#!/bin/sh

# Xcode generator doesn't include the compiler as the
# first argument, Ninja and Makefiles do. Handle both cases.
if [[ "$1" = "${CMAKE_CXX_COMPILER}" ]] ; then
shift
fi

export CCACHE_CPP2=true
exec "${CXX_LAUNCHER}" "${CMAKE_CXX_COMPILER}" "$@"

CMakeLists.txt:

cmake_minimum_required(VERSION 3.4)
project(SomeProject)

find_program(CCACHE_PROGRAM ccache)
if(CCACHE_PROGRAM)
  # Set up wrapper scripts
set(C_LAUNCHER "${CCACHE_PROGRAM}")
set(CXX_LAUNCHER "${CCACHE_PROGRAM}")
configure_file(launch-c.in launch-c)
configure_file(launch-cxx.in launch-cxx)
execute_process(COMMAND chmod a+rx "${CMAKE_BINARY_DIR}/launch-c" "${CMAKE_BINARY_DIR}/launch-cxx" )

  if(CMAKE_GENERATOR STREQUAL "Xcode")
  # Set Xcode project attributes to route compilation and linking
  # through our scripts
  set(CMAKE_XCODE_ATTRIBUTE_CC "${CMAKE_BINARY_DIR}/launch-c")
  set(CMAKE_XCODE_ATTRIBUTE_CXX "${CMAKE_BINARY_DIR}/launch-cxx")
  set(CMAKE_XCODE_ATTRIBUTE_LD "${CMAKE_BINARY_DIR}/launch-c")
  set(CMAKE_XCODE_ATTRIBUTE_LDPLUSPLUS "${CMAKE_BINARY_DIR}/launch-cxx")
  else()
  # Support Unix Makefiles and Ninja
  set(CMAKE_C_COMPILER_LAUNCHER "${CMAKE_BINARY_DIR}/launch-c")
  set(CMAKE_CXX_COMPILER_LAUNCHER "${CMAKE_BINARY_DIR}/launch-cxx")
  endif()
endif()

All three generators now behave consistently, using the same language-specific launch scripts. The above example places all of the ccache-specific logic after the project() command, so it could easily be factored out into a separate file and brought in with an include() or similar. The project() command won’t perform its compiler and linker tests with the launchers now that RULE_LAUNCH_COMPILE isn’t being set before project() is called, only the real underlying compilers will be tested by project(). This is a minor loss which only affects the Ninja and Makefiles generators and one could argue the consistency in behaviour across all generators is a fair tradeoff.

Acknowledgements

Credit to Peter Steinberger for highlighting the technique of setting CC and CXX in Xcode project files to override the compiler used. His article also contains a few other useful observations for using ccache with clang which may be of interest to some readers.


Have a CMake maintainer work on your project

Get the book for more CMake content

18 thoughts on “Using ccache with CMake”

    • The original article didn’t route Ninja or Makefiles via the launch scripts, they instead always invoked ccache directly. I’ve updated the article to show how to route them via the same launch scripts as Xcode (see the section “Improved functionality from CMake 3.4”).

      Reply
  1. Hi, I’m having a problem.

    If I call find_program(CCACHE_PROGRAM ccache) after project(SomeProject), It can’t find ccache.

    If i call find_program(CCACHE_PROGRAM ccache) before project(SomeProject), it works.

    Any ideia?
    I was trying to use the Improved functionality from CMake 3.4.

    Thanks

    Reply
    • I cannot reproduce your problem, it works fine for me for both cases, i.e. if I call find_program(CCACHE_PROGRAM ccache) before or after project(). Try a fresh build directory to see if that fixes your problem. If you still see an issue, check your toolchain file if you are using one. Other than that, I can’t make any further comment on what your issue might be. I suggest you post a question on stackoverflow with full details of your project if you still have problems.

      Reply
    • Henry that’s really useful! I had been intending to put together something like that to contribute as a new module to CMake, but hadn’t considered adding support for CUDA as well. Would you be willing to contribute it to CMake? It should just need the appropriate copyright block and docs added at the top and the rest shouldn’t be far off being okay. I’d suggest using the name CompilerCache.cmake (I am thinking ahead to the possibility of adding support for things like clcache for Visual Studio, so best to avoid specifically using ccache in the module name).

      Reply
  2. For some reason I receive gcc warning:
    c++: warning: /usr/bin/c++: linker input file unused because linking not done

    when
    # Support Unix Makefiles and Ninja
    set(CMAKE_C_COMPILER_LAUNCHER “${CMAKE_BINARY_DIR}/launch-c”)
    set(CMAKE_CXX_COMPILER_LAUNCHER “${CMAKE_BINARY_DIR}/launch-cxx”)

    so I fixed this by replacing laucher with ccache binary:

    # Support Unix Makefiles and Ninja
    set(CMAKE_C_COMPILER_LAUNCHER "${CCACHE_PROGRAM}")
    set(CMAKE_CXX_COMPILER_LAUNCHER "${CCACHE_PROGRAM}")

    Reply
    • Not sure why you are getting that warning, but note that using CCACHE_PROGRAM directly means the build will end up using whatever compiler is found via the PATH at build time, not necessarily the one CMake found and tested. Perhaps get the build to show the actual compiler and linker command lines and check that they look correct. If I had to guess, I’d say you either have a typo in one of your scripts or there’s a stray -c option creeping into your link command line somehow.

      Reply
  3. I had a similar issue as described by hoouiBogdan.

    Using the “older” solution solves the problem for Unix Makefiles and Ninja:
    set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE “${CCACHE_PROGRAM}”)

    Reply
  4. Turning ccache on / off causes cmake to do a complete rebuild for us, which doesn’t seem necessary. Have others found any way around this?

    Reply
  5. If you turn ccache on or off, it changes the command line used for compilation, so that will trigger the dependency checking to force recompilation. If you turn it off and back on without recompiling in between (or on and back off), it shouldn’t rebuild because the build will see the same command line as before.

    I personally haven’t hit this because I almost never turn it off. There’s generally little reason to do so, in my experience.

    Reply
  6. Is it possible to achieve the effect of setting CMAKE_CXX_COMPILER_LAUNCHER but using target options? Working with Microsoft COM #import statements, I’d like to remove the complications of an additional command. Thanks

    Reply
    • You can control which launcher is used on a per-target basis. The CMAKE_CXX_COMPILER_LAUNCHER variable is just a default for the CXX_COMPILER_LAUNCHER target property. The variable’s value is used to initialise the target property when the target is defined by add_executable() or add_library(). Normally, you want the same launcher used throughout the whole project, so the CMAKE_CXX_COMPILER_LAUNCHER variable is the most convenient way to set that. But if you need different behavior for a particular target, you can always set the CXX_COMPILER_LAUNCHER property on that target to whatever you need.

      If you want to avoid a launcher altogether, I don’t know what you’re trying to do. If the compiler supports what you want through its own command-line options, then you can set those with target_compile_options(). Otherwise, it isn’t clear what you’re trying to achieve. You may want to ask for help in the CMake forums for better exposure and more discussion.

      Reply

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.