Modules.rst

Modules (EXPERIMENTAL)

Warning

The functionality described on this page is still experimental! Please try it out and send us bug reports!

• Introduction
  • Problems with the current model
  • Semantic import
  • Problems modules do not solve
• Using Modules
  • Import declaration
  • Includes as imports
  • Module maps
  • Compilation model
  • Command-line parameters
• Module Map Language
  • Lexical structure
  • Module map file
  • Module declaration
    • Requires declaration
    • Header declaration
    • Umbrella directory declaration
    • Submodule declaration
    • Export declaration
    • Use declaration
    • Link declaration
    • Configuration macros declaration
    • Conflict declarations
  • Attributes
• Modularizing a Platform
• Future Directions
• Where To Learn More About Modules

Introduction

Most software is built using a number of software libraries, including libraries supplied by the platform, internal libraries built as part of the software itself to provide structure, and third-party libraries. For each library, one needs to access both its interface (API) and its implementation. In the C family of languages, the interface to a library is accessed by including the appropriate header files(s):

#include <SomeLib.h>

The implementation is handled separately by linking against the appropriate library. For example, by passing -lSomeLib to the linker.

Modules provide an alternative, simpler way to use software libraries that provides better compile-time scalability and eliminates many of the problems inherent to using the C preprocessor to access the API of a library.

Problems with the current model

The #include mechanism provided by the C preprocessor is a very poor way to access the API of a library, for a number of reasons:

• Compile-time scalability: Each time a header is included, the compiler must preprocess and parse the text in that header and every header it includes, transitively. This process must be repeated for every translation unit in the application, which involves a huge amount of redundant work. In a project with N translation units and M headers included in each translation unit, the compiler is performing M x N work even though most of the M headers are shared among multiple translation units. C++ is particularly bad, because the compilation model for templates forces a huge amount of code into headers.
• Fragility: #include directives are treated as textual inclusion by the preprocessor, and are therefore subject to any active macro definitions at the time of inclusion. If any of the active macro definitions happens to collide with a name in the library, it can break the library API or cause compilation failures in the library header itself. For an extreme example, #define std "The C++ Standard" and then include a standard library header: the result is a horrific cascade of failures in the C++ Standard Library's implementation. More subtle real-world problems occur when the headers for two different libraries interact due to macro collisions, and users are forced to reorder #include directives or introduce #undef directives to break the (unintended) dependency.
• Conventional workarounds: C programmers have adopted a number of conventions to work around the fragility of the C preprocessor model. Include guards, for example, are required for the vast majority of headers to ensure that multiple inclusion doesn't break the compile. Macro names are written with LONG_PREFIXED_UPPERCASE_IDENTIFIERS to avoid collisions, and some library/framework developers even use __underscored names in headers to avoid collisions with "normal" names that (by convention) shouldn't even be macros. These conventions are a barrier to entry for developers coming from non-C languages, are boilerplate for more experienced developers, and make our headers far uglier than they should be.
• Tool confusion: In a C-based language, it is hard to build tools that work well with software libraries, because the boundaries of the libraries are not clear. Which headers belong to a particular library, and in what order should those headers be included to guarantee that they compile correctly? Are the headers C, C++, Objective-C++, or one of the variants of these languages? What declarations in those headers are actually meant to be part of the API, and what declarations are present only because they had to be written as part of the header file?

Semantic import

Modules improve access to the API of software libraries by replacing the textual preprocessor inclusion model with a more robust, more efficient semantic model. From the user's perspective, the code looks only slightly different, because one uses an import declaration rather than a #include preprocessor directive:

import std.io; // pseudo-code; see below for syntax discussion

However, this module import behaves quite differently from the corresponding #include <stdio.h>: when the compiler sees the module import above, it loads a binary representation of the std.io module and makes its API available to the application directly. Preprocessor definitions that precede the import declaration have no impact on the API provided by std.io, because the module itself was compiled as a separate, standalone module. Additionally, any linker flags required to use the std.io module will automatically be provided when the module is imported [1] This semantic import model addresses many of the problems of the preprocessor inclusion model:

• Compile-time scalability: The std.io module is only compiled once, and importing the module into a translation unit is a constant-time operation (independent of module system). Thus, the API of each software library is only parsed once, reducing the M x N compilation problem to an M + N problem.
• Fragility: Each module is parsed as a standalone entity, so it has a consistent preprocessor environment. This completely eliminates the need for __underscored names and similarly defensive tricks. Moreover, the current preprocessor definitions when an import declaration is encountered are ignored, so one software library can not affect how another software library is compiled, eliminating include-order dependencies.
• Tool confusion: Modules describe the API of software libraries, and tools can reason about and present a module as a representation of that API. Because modules can only be built standalone, tools can rely on the module definition to ensure that they get the complete API for the library. Moreover, modules can specify which languages they work with, so, e.g., one can not accidentally attempt to load a C++ module into a C program.

Problems modules do not solve

Many programming languages have a module or package system, and because of the variety of features provided by these languages it is important to define what modules do not do. In particular, all of the following are considered out-of-scope for modules:

• Rewrite the world's code: It is not realistic to require applications or software libraries to make drastic or non-backward-compatible changes, nor is it feasible to completely eliminate headers. Modules must interoperate with existing software libraries and allow a gradual transition.
• Versioning: Modules have no notion of version information. Programmers must still rely on the existing versioning mechanisms of the underlying language (if any exist) to version software libraries.
• Namespaces: Unlike in some languages, modules do not imply any notion of namespaces. Thus, a struct declared in one module will still conflict with a struct of the same name declared in a different module, just as they would if declared in two different headers. This aspect is important for backward compatibility, because (for example) the mangled names of entities in software libraries must not change when introducing modules.
• Binary distribution of modules: Headers (particularly C++ headers) expose the full complexity of the language. Maintaining a stable binary module format across architectures, compiler versions, and compiler vendors is technically infeasible.

Using Modules

To enable modules, pass the command-line flag -fmodules [2]. This will make any modules-enabled software libraries available as modules as well as introducing any modules-specific syntax. Additional command-line parameters are described in a separate section later.

Import declaration

The most direct way to import a module is with an import declaration, which imports the named module:

import std;

The import declaration above imports the entire contents of the std module (which would contain, e.g., the entire C or C++ standard library) and make its API available within the current translation unit. To import only part of a module, one may use dot syntax to specific a particular submodule, e.g.,

import std.io;

Redundant import declarations are ignored, and one is free to import modules at any point within the translation unit, so long as the import declaration is at global scope.

Warning

The import declaration syntax described here does not actually exist. Rather, it is a straw man proposal that may very well change when modules are discussed in the C and C++ committees. See the section Includes as imports to see how modules get imported today.

Includes as imports

The primary user-level feature of modules is the import operation, which provides access to the API of software libraries. However, today's programs make extensive use of #include, and it is unrealistic to assume that all of this code will change overnight. Instead, modules automatically translate #include directives into the corresponding module import. For example, the include directive

#include <stdio.h>

will be automatically mapped to an import of the module std.io. Even with specific import syntax in the language, this particular feature is important for both adoption and backward compatibility: automatic translation of #include to import allows an application to get the benefits of modules (for all modules-enabled libraries) without any changes to the application itself. Thus, users can easily use modules with one compiler while falling back to the preprocessor-inclusion mechanism with other compilers.

Note

The automatic mapping of #include to import also solves an implementation problem: importing a module with a definition of some entity (say, a struct Point) and then parsing a header containing another definition of struct Point would cause a redefinition error, even if it is the same struct Point. By mapping #include to import, the compiler can guarantee that it always sees just the already-parsed definition from the module.

Module maps

The crucial link between modules and headers is described by a module map, which describes how a collection of existing headers maps on to the (logical) structure of a module. For example, one could imagine a module std covering the C standard library. Each of the C standard library headers (<stdio.h>, <stdlib.h>, <math.h>, etc.) would contribute to the std module, by placing their respective APIs into the corresponding submodule (std.io, std.lib, std.math, etc.). Having a list of the headers that are part of the std module allows the compiler to build the std module as a standalone entity, and having the mapping from header names to (sub)modules allows the automatic translation of #include directives to module imports.

Module maps are specified as separate files (each named module.map) alongside the headers they describe, which allows them to be added to existing software libraries without having to change the library headers themselves (in most cases [3]). The actual Module map language is described in a later section.

Note

To actually see any benefits from modules, one first has to introduce module maps for the underlying C standard library and the libraries and headers on which it depends. The section Modularizing a Platform describes the steps one must take to write these module maps.

One can use module maps without modules to check the integrity of the use of header files. To do this, use the -fmodule-maps option instead of the -fmodules option.

Compilation model

The binary representation of modules is automatically generated by the compiler on an as-needed basis. When a module is imported (e.g., by an #include of one of the module's headers), the compiler will spawn a second instance of itself [4], with a fresh preprocessing context [5], to parse just the headers in that module. The resulting Abstract Syntax Tree (AST) is then persisted into the binary representation of the module that is then loaded into translation unit where the module import was encountered.

The binary representation of modules is persisted in the module cache. Imports of a module will first query the module cache and, if a binary representation of the required module is already available, will load that representation directly. Thus, a module's headers will only be parsed once per language configuration, rather than once per translation unit that uses the module.

Modules maintain references to each of the headers that were part of the module build. If any of those headers changes, or if any of the modules on which a module depends change, then the module will be (automatically) recompiled. The process should never require any user intervention.

Command-line parameters

-fmodules

Enable the modules feature (EXPERIMENTAL).

-fcxx-modules

Enable the modules feature for C++ (EXPERIMENTAL and VERY BROKEN).

-fmodule-maps

Enable interpretation of module maps (EXPERIMENTAL). This option is implied by -fmodules.

-fmodules-cache-path=<directory>

Specify the path to the modules cache. If not provided, Clang will select a system-appropriate default.

-fno-autolink

Disable automatic linking against the libraries associated with imported modules.

-fmodules-ignore-macro=macroname

Instruct modules to ignore the named macro when selecting an appropriate module variant. Use this for macros defined on the command line that don't affect how modules are built, to improve sharing of compiled module files.

-fmodules-prune-interval=seconds

Specify the minimum delay (in seconds) between attempts to prune the module cache. Module cache pruning attempts to clear out old, unused module files so that the module cache itself does not grow without bound. The default delay is large (604,800 seconds, or 7 days) because this is an expensive operation. Set this value to 0 to turn off pruning.

-fmodules-prune-after=seconds

Specify the minimum time (in seconds) for which a file in the module cache must be unused (according to access time) before module pruning will remove it. The default delay is large (2,678,400 seconds, or 31 days) to avoid excessive module rebuilding.

-module-file-info <module file name>

Debugging aid that prints information about a given module file (with a .pcm extension), including the language and preprocessor options that particular module variant was built with.

-fmodules-decluse

Enable checking of module use declarations.

-fmodule-name=module-id

Consider a source file as a part of the given module.

Module Map Language

The module map language describes the mapping from header files to the logical structure of modules. To enable support for using a library as a module, one must write a module.map file for that library. The module.map file is placed alongside the header files themselves, and is written in the module map language described below.

As an example, the module map file for the C standard library might look a bit like this:

module std [system] {
  module complex {
    header "complex.h"
    export *
  }

  module ctype {
    header "ctype.h"
    export *
  }

  module errno {
    header "errno.h"
    header "sys/errno.h"
    export *
  }

  module fenv {
    header "fenv.h"
    export *
  }

  // ...more headers follow...
}

Here, the top-level module std encompasses the whole C standard library. It has a number of submodules containing different parts of the standard library: complex for complex numbers, ctype for character types, etc. Each submodule lists one of more headers that provide the contents for that submodule. Finally, the export * command specifies that anything included by that submodule will be automatically re-exported.

Lexical structure

Module map files use a simplified form of the C99 lexer, with the same rules for identifiers, tokens, string literals, /* */ and // comments. The module map language has the following reserved words; all other C identifiers are valid identifiers.

config_macros export     module
conflict      framework  requires
exclude       header     private
explicit      link       umbrella
extern        use

Module map file

A module map file consists of a series of module declarations:

module-map-file:
  module-declaration*

Within a module map file, modules are referred to by a module-id, which uses periods to separate each part of a module's name:

module-id:
  identifier ('.' identifier)*

Module declaration

A module declaration describes a module, including the headers that contribute to that module, its submodules, and other aspects of the module.

module-declaration:
  explicitopt frameworkopt module module-id attributesopt '{' module-member* '}'
  extern module module-id string-literal

The module-id should consist of only a single identifier, which provides the name of the module being defined. Each module shall have a single definition.

The explicit qualifier can only be applied to a submodule, i.e., a module that is nested within another module. The contents of explicit submodules are only made available when the submodule itself was explicitly named in an import declaration or was re-exported from an imported module.

The framework qualifier specifies that this module corresponds to a Darwin-style framework. A Darwin-style framework (used primarily on Mac OS X and iOS) is contained entirely in directory Name.framework, where Name is the name of the framework (and, therefore, the name of the module). That directory has the following layout:

Name.framework/
  module.map                Module map for the framework
  Headers/                  Subdirectory containing framework headers
  Frameworks/               Subdirectory containing embedded frameworks
  Resources/                Subdirectory containing additional resources
  Name                      Symbolic link to the shared library for the framework

The system attribute specifies that the module is a system module. When a system module is rebuilt, all of the module's header will be considered system headers, which suppresses warnings. This is equivalent to placing #pragma GCC system_header in each of the module's headers. The form of attributes is described in the section Attributes, below.

Modules can have a number of different kinds of members, each of which is described below:

module-member:
  requires-declaration
  header-declaration
  umbrella-dir-declaration
  submodule-declaration
  export-declaration
  use-declaration
  link-declaration
  config-macros-declaration
  conflict-declaration

An extern module references a module defined by the module-id in a file given by the string-literal. The file can be referenced either by an absolute path or by a path relative to the current map file.

Requires declaration

A requires-declaration specifies the requirements that an importing translation unit must satisfy to use the module.

requires-declaration:
  requires feature-list

feature-list:
  identifier (',' identifier)*

The requirements clause allows specific modules or submodules to specify that they are only accessible with certain language dialects or on certain platforms. The feature list is a set of identifiers, defined below. If any of the features is not available in a given translation unit, that translation unit shall not import the module.

The following features are defined:

altivec

The target supports AltiVec.

blocks

The "blocks" language feature is available.

cplusplus

C++ support is available.

cplusplus11

C++11 support is available.

objc

Objective-C support is available.

objc_arc

Objective-C Automatic Reference Counting (ARC) is available

opencl

OpenCL is available

tls

Thread local storage is available.

target feature

A specific target feature (e.g., sse4, avx, neon) is available.

Example: The std module can be extended to also include C++ and C++11 headers using a requires-declaration:

module std {
   // C standard library...

   module vector {
     requires cplusplus
     header "vector"
   }

   module type_traits {
     requires cplusplus11
     header "type_traits"
   }
 }

Header declaration

A header declaration specifies that a particular header is associated with the enclosing module.

header-declaration:
  umbrellaopt header string-literal
  private header string-literal
  exclude header string-literal

A header declaration that does not contain exclude specifies a header that contributes to the enclosing module. Specifically, when the module is built, the named header will be parsed and its declarations will be (logically) placed into the enclosing submodule.

A header with the umbrella specifier is called an umbrella header. An umbrella header includes all of the headers within its directory (and any subdirectories), and is typically used (in the #include world) to easily access the full API provided by a particular library. With modules, an umbrella header is a convenient shortcut that eliminates the need to write out header declarations for every library header. A given directory can only contain a single umbrella header.

Note

Any headers not included by the umbrella header should have explicit header declarations. Use the -Wincomplete-umbrella warning option to ask Clang to complain about headers not covered by the umbrella header or the module map.

A header with the private specifier may not be included from outside the module itself.

A header with the exclude specifier is excluded from the module. It will not be included when the module is built, nor will it be considered to be part of the module.

Example: The C header assert.h is an excellent candidate for an excluded header, because it is meant to be included multiple times (possibly with different NDEBUG settings).

module std [system] {
  exclude header "assert.h"
}

A given header shall not be referenced by more than one header-declaration.

Umbrella directory declaration

An umbrella directory declaration specifies that all of the headers in the specified directory should be included within the module.

umbrella-dir-declaration:
  umbrella string-literal

The string-literal refers to a directory. When the module is built, all of the header files in that directory (and its subdirectories) are included in the module.

An umbrella-dir-declaration shall not refer to the same directory as the location of an umbrella header-declaration. In other words, only a single kind of umbrella can be specified for a given directory.

Note

Umbrella directories are useful for libraries that have a large number of headers but do not have an umbrella header.

Submodule declaration

Submodule declarations describe modules that are nested within their enclosing module.

submodule-declaration:
  module-declaration
  inferred-submodule-declaration

A submodule-declaration that is a module-declaration is a nested module. If the module-declaration has a framework specifier, the enclosing module shall have a framework specifier; the submodule's contents shall be contained within the subdirectory Frameworks/SubName.framework, where SubName is the name of the submodule.

A submodule-declaration that is an inferred-submodule-declaration describes a set of submodules that correspond to any headers that are part of the module but are not explicitly described by a header-declaration.

inferred-submodule-declaration:
  explicitopt frameworkopt module '*' attributesopt '{' inferred-submodule-member* '}'

inferred-submodule-member:
  export '*'

A module containing an inferred-submodule-declaration shall have either an umbrella header or an umbrella directory. The headers to which the inferred-submodule-declaration applies are exactly those headers included by the umbrella header (transitively) or included in the module because they reside within the umbrella directory (or its subdirectories).

For each header included by the umbrella header or in the umbrella directory that is not named by a header-declaration, a module declaration is implicitly generated from the inferred-submodule-declaration. The module will:

• Have the same name as the header (without the file extension)
• Have the explicit specifier, if the inferred-submodule-declaration has the explicit specifier
• Have the framework specifier, if the inferred-submodule-declaration has the framework specifier
• Have the attributes specified by the inferred-submodule-declaration
• Contain a single header-declaration naming that header
• Contain a single export-declaration export *, if the inferred-submodule-declaration contains the inferred-submodule-member export *

Example: If the subdirectory "MyLib" contains the headers A.h and B.h, then the following module map:

module MyLib {
  umbrella "MyLib"
  explicit module * {
    export *
  }
}

is equivalent to the (more verbose) module map:

module MyLib {
  explicit module A {
    header "A.h"
    export *
  }

  explicit module B {
    header "B.h"
    export *
  }
}

Export declaration

An export-declaration specifies which imported modules will automatically be re-exported as part of a given module's API.

export-declaration:
  export wildcard-module-id

wildcard-module-id:
  identifier
  '*'
  identifier '.' wildcard-module-id

The export-declaration names a module or a set of modules that will be re-exported to any translation unit that imports the enclosing module. Each imported module that matches the wildcard-module-id up to, but not including, the first * will be re-exported.

Example:: In the following example, importing MyLib.Derived also provides the API for MyLib.Base:

module MyLib {
  module Base {
    header "Base.h"
  }

  module Derived {
    header "Derived.h"
    export Base
  }
}

Note that, if Derived.h includes Base.h, one can simply use a wildcard export to re-export everything Derived.h includes:

module MyLib {
  module Base {
    header "Base.h"
  }

  module Derived {
    header "Derived.h"
    export *
  }
}

Note

The wildcard export syntax export * re-exports all of the modules that were imported in the actual header file. Because #include directives are automatically mapped to module imports, export * provides the same transitive-inclusion behavior provided by the C preprocessor, e.g., importing a given module implicitly imports all of the modules on which it depends. Therefore, liberal use of export * provides excellent backward compatibility for programs that rely on transitive inclusion (i.e., all of them).

Use declaration

A use-declaration specifies one of the other modules that the module is allowed to use. An import or include not matching one of these is rejected when the option -fmodules-decluse.

use-declaration:
  use module-id

Example:: In the following example, use of A from C is not declared, so will trigger a warning.

module A {
  header "a.h"
}

module B {
  header "b.h"
}

module C {
  header "c.h"
  use B
}

When compiling a source file that implements a module, use the option ``-fmodule-name=``module-id to indicate that the source file is logically part of that module.

The compiler at present only applies restrictions to the module directly being built.

Link declaration

A link-declaration specifies a library or framework against which a program should be linked if the enclosing module is imported in any translation unit in that program.

link-declaration:
  link frameworkopt string-literal

The string-literal specifies the name of the library or framework against which the program should be linked. For example, specifying "clangBasic" would instruct the linker to link with -lclangBasic for a Unix-style linker.

A link-declaration with the framework specifies that the linker should link against the named framework, e.g., with -framework MyFramework.

Note

Automatic linking with the link directive is not yet widely implemented, because it requires support from both the object file format and the linker. The notion is similar to Microsoft Visual Studio's #pragma comment(lib...).

Configuration macros declaration

The config-macros-declaration specifies the set of configuration macros that have an effect on the the API of the enclosing module.

config-macros-declaration:
  config_macros attributesopt config-macro-listopt

config-macro-list:
  identifier (',' identifier)*

Each identifier in the config-macro-list specifies the name of a macro. The compiler is required to maintain different variants of the given module for differing definitions of any of the named macros.

A config-macros-declaration shall only be present on a top-level module, i.e., a module that is not nested within an enclosing module.

The exhaustive attribute specifies that the list of macros in the config-macros-declaration is exhaustive, meaning that no other macro definition is intended to have an effect on the API of that module.

Note

The exhaustive attribute implies that any macro definitions for macros not listed as configuration macros should be ignored completely when building the module. As an optimization, the compiler could reduce the number of unique module variants by not considering these non-configuration macros. This optimization is not yet implemented in Clang.

A translation unit shall not import the same module under different definitions of the configuration macros.

Note

Clang implements a weak form of this requirement: the definitions used for configuration macros are fixed based on the definitions provided by the command line. If an import occurs and the definition of any configuration macro has changed, the compiler will produce a warning (under the control of -Wconfig-macros).

Example: A logging library might provide different API (e.g., in the form of different definitions for a logging macro) based on the NDEBUG macro setting: