This doc aims to describe internal-facing implementation concepts. For external-facing, see https://android.googlesource.com/platform/build/bazel/+/refs/heads/master/docs/concepts.md.
| Soong/Ninja | Bazel | Remarks |
|---|---|---|
| make phony goal, e.g. "dist", "sdk", "apps_only", "droidcore" | Top level filegroup rule target | Details |
| Ninja build target (phony) | (readable) alias to a file target | |
| Ninja build target (non-phony) | File target | |
ModuleFactory | RuleConfiguredTargetFactory | |
Module type (e.g. cc_library) | Rule class (e.g. cc_library) | |
| Module object instance | Target (instance of a rule) | Details |
| Module properties | Rule attributes | Details |
| Module name | Target label | |
| Module variant | (Split) configured target | |
| LoadHooks | macros (ish) | |
| Top-down mutators on modules | Split configuration on targets | Allows building multiple "variants" of the same build artifact in the same build. |
| Bottom-up mutators on modules | Aspects on targets | |
| Build statement (Ninja) | Action (result of ctx.actions.run) | |
| Rule statement (Ninja) | ctx.actions.run() API | |
out/soong/build.ninja and out/build-<target>.ninja | Action graph (serialized) | |
| Pool (ninja) | Thread pools / ExecutorService | |
Blueprint's Registration and Parse, ResolveDependencies phase | Loading phase | |
| Blueprint's Generate and Write phases | Analysis Phase | |
| Ninja execution | Execution phase | |
Blueprints/Android.bp files | BUILD/BUILD.bazel files | |
| Namespaces | Packages | Most Soong modules are within the global namespace |
| Mutators | Configuration keys (ish) | |
| Variation | Configuration value | |
| Singleton | Aspect-ish | |
| Target (system + vendor + product) | Platform | |
Bash scripts e.g. envsetup functions, soong_ui.bash) | Repository rule | |
| Product and board configuration makefile and env variables | Configuration in Bazel (ish) | Details |
| Dependency Tags | Provider names |
Soong maintains the make terminology of goals to denote what should be built. All modules can be specified by name as a goal, in addition, phony goals are supported.
A Phony goal creates a Make-style phony rule, a rule with no commands that can depend on other phony rules or real files. Phony can be called on the same name multiple times to add additional dependencies. These are often used to build many targets at once. The default goal for Android's build system is droid. Some other common phony goals include: nothing (perform loading/analysis), docs, checkbuild, apps_only.
Some common phony goals are defined in build/make/core/main.mk The purpose is to help soong_ui to determine what top level files to build.
When a Module is instantiated by Blueprint (which calls the appropriate ModuleFactory), the property structs are populated by Blueprint.
Blueprint performs no additional operations on these properties, such that dependencies on other modules and references to source files are unresolved initially. Mutators then introspect the values of properties to specify dependencies between modules, which Blueprint resolves. Source files (including globs) and output paths for references to other modules are resolved during blueprint analysis via the various Path[s]ForModuleSrc[Excludes] functions within build/soong/android/paths.go.
For a Bazel target instance, the dependencies and source file references within attrs have been resolved by Bazel.
Bazel implementation to collect deps.
Within Soong/Blueprint, properties are represented as Go structs, which can be nested, with no depth limit. Properties can be primitive or pointer types, but they must be one of these types: int64, string, bool, list.
These properties can be defined from various structs within the module type factory itself (via AddProperties) or from common helper functions such as:
InitAndroidModule: specifies name-related, common, and dist properties.InitAndroidArchModule: adds host/device properiesGo comments for a property will be treated as documentation to describe the property. In some cases, these comments describe a default value for the property. However, the default value is not based on the comment or field definition but resolved somewhere within the module's mutators or build. These defaults are often determined using Blueprint proptools *Default functions. For example, cc modules have a property include_build_directory, which is described in the comments. The default value is resolved when compiler flags are being determined.
In general, these can be set in an Android.bp file. However, if the property is tagged with `blueprint:"mutated"`, it can only be set programmatically within Blueprint/Soong. Additionally, mutated tagged properties also support map and int types in addition to those mentioned above. These mutated properties are used to propagate data that gets set during mutations, which ensures that the information is copied successfully to module variants during mutation.
Soong supports additional property tags to provide additional functionality/information about a property:
`android:arch_variant`: This specifies that a property can be configured for different architectures, operating systems, targets, etc. The arch mutator, will merge target-specific properties into the correct variant for properties with this tag.
Note: if a nested property is arch-variant, all recursively nesting structs that can be specified in an Android.bp file must also be tagged as arch-variant.
`android:variant_prepend`: When merging properties for the arch variant, the arch-specific values should be prepended rather than appended to existing property values.
`android:path`: This specifies that this property will contain some combination of:
":<name>{.<tag>}", where {.<tag>} is optional to specify a non-default output file, specific to the module type"<namespace>:<name>{.<tag>}""Note: Dependencies to other modules for these properties will be automatically added by the pathdeps mutator.
Similar to properties, attributes only support a few types. The difference is that Bazel attributes cannot be nested .
Some attributes are common across many/all rule classes, including (but not limited to) name, tag, visibility.
The definition of an attribute can contain settings, such as: its default value, whether it is mandatory ot have a value, and its documentation.
To specify a source file or reference to another module, use label or label_list attribute types (rather than regular string or string_list types). These support additional restrictions (as compared to string* types), such as:
Unlike Soong, when accessing this attribute within the rule's implementation (at anlysis time), the label(s) will be resolved to the file or target they refer to.
Attributes do not need to specify whether they accept configurable attribute. However, the rule definition can specify the configuration or specify a configuration transition.
However, not all target definitions within a BUILD file are invoking a rule. Instead, they may invoke a Starlark macro, which is a load-time wrapper around rules. Arguments for a macro are not typed. If macros are used, their arguments would have to be wrangled into an attribute-compatible type.
LoadHooks provide access to :
AppendProperties/PrependProperties)CreateModuleLoadHooks make it easier to extend existing module factories to always specify certain properties or to split a single Android.bp definition into multiple Module instances .
Ninja build statements can be expanded from Ninja rules, which are like templates.
# rule
rule cattool
depfile = out/test/depfile.d
command = ${in} ${out}
# build statement
build out/test/output.txt: cattool test/cattool.sh test/one test/two
# build statement
build out/test/other_output.txt: cattool test/cattool.sh test/three test/four
Rules for Android.mk modules (out/build-<target>.ninja) and build statements are 1:1. That is every rule is only used once by a single build statement.
Soong (out/soong/build.ninja) rules are reused extensively in build statements (1:many). For example the Cp rule is a commonly used rule for creating build statements which copy files.
In Soong, Ninja rules can be defined in two ways:
ResolveDependencies Running a series of Mutators, to add dependencies, split modules with variations, etc
GenerateBuildActions to generate Ninja statements, which in turn calls each module's GenerateAndroidBuildActions.Module Namespaces can import other namespaces, and there’s a module name lookup algorithm which terminates in the global namespace.
Note: this is not widely used and most Soong modules are in the global namespace.
Packages can nest subpackages recursively, but they are independent containers of Bazel targets. This means that Bazel target names only need to be unique within a package.
blueprint invokes mutators are invoking in the order they are registered (e.g. top-down and bottom-up can be interleaved). Each mutator applys a single visitation to every module in the graph.
Mutators visiting module can parallelized, while maintaining their ordering, by calling .Parallel().
While top-down and bottom-up mutators differ in their purposes, the interface available to each contains many similarities. Both have access to: BaseModuleContext and BaseMutatorContext.
In addition to the registration order, Soong supports phase-based ordering of mutators:
Pre-Arch: mutators that need to run before arch-variation. For example, defaults are handled at this stage such properties from defaults are correctly propagated to arch-variants later.
(Hard-coded) archMutator splits a module into the appropriate target(s). Next, the arch- and OS-specific properties are merged into the appropriate variant.
Pre-Deps: mutators that can/need to run before deps have been resolved, for instance, creating variations that have an impact on dependency resolution.
(Hard-coded) depsMutator, which calls the DepsMutator function that must be part of a Soong Module's interface.
Post-Deps: mutators that need to run after deps have been resolved
Final-Deps like post-deps but variations cannot be created
A top-down mutator is invoked on a module before its dependencies.
The general purpose is to propagate dependency info from a module to its dependencies.
A bottom-up mutator is invoked on a module only after the mutator has been invoked on all its dependencies.
The general purpose of a bottom-up mutator is to split modules into variants.
A tuple (name of mutator, variation / config value) passed to CreateVariations.
Soong's config process encompasses both what should build and how it should build. This section focuses on the how aspect.
We do not cover how Soong's configuration will be implemented in Bazel, but the general capabilities of Bazel to configure builds.
Android users can configure their builds based on:
Some environment variables or command line options are used directly to alter the build. However, specification of target product encompasses many aspects of both what and how things are built. This configuration is currently handled within Make but is in the process of being migrated to Starlark.
Soong invokes Kati to run in a "config" mode, also commonly known as "product config". This mode limits the scope of what .mk files are parsed. The product-specific handlers are largely in:
product_config.mk: this subset of functionality is also commonly referred to as "product config"board_config.mkHowever, these cover only a subset of config.mk. This ensures that all values have appropriate defaults and specify details necessary to the build. Some examples:
.bazelrc file.PRODUCT_SHIPPING_API_LEVEL is defaulted if not specified by the target.Finally, Kati dumps variables to be consumed by Soong:
soong.variables, a JSON fileThroughout Soong, environment variables can be accessed to alter the build via the Config:
Soong loads the soong.variables config file, stored as productVariables. These variables are used in three ways:
Config, for example: paths can be opted out of specific sanitizersAndroid.bp file to control what is built or perform variable replacement. variableProperties limits which configuration variables can be specified within an Android.bp file and which properties they can apply to. The values specified within an Android.bp file, are merged/replaced by the VariableMutator, which appends performs string replacement if requested and merges the properties into the modules.Android.bp file for the module type and properties they request. Soong config variable structs are dynamically generated via reflection. In the factory, the properties to merge into the module instance are identified based on the config variable's type.The product configuration also provides information about architecture and operating system, both for target(s) and host. This is used within the archMutator to split a module into the required variants and merge target-specific properties into the appropriate variant. Only properties which have been tagged with android:"arch_variant" can be specified within an Android.bp as arch/os/target-specific. For example:
type properties struct { // this property will be arch-variant Arch_variant_not_nested *string `android:"arch_variant"` Nested_with_arch_variant struct { // this property is arch-variant Arch_variant_nested *string `android:"arch_variant"` // this property is **not** arch-variant Not_arch_variant_nested *string } `android:"arch_variant"` Nested_no_arch_variant struct { // this property is **NOT** arch-variant No_arch_variant_nested_not_arch_variant *string `android:"arch_variant"` // this property is **not** arch-variant No_arch_variant_nested *string } }
The arch/os/target-specific structs are dynamically generated based on the tags using reflection.
Bazel documentation covers configurable builds fairly extensively, so this is a short overview that primarily links to existing Bazel documentation rather than repeating it here.
Configurable attributes, (aka select()) allows users to toggle values of build rule attributes on the command line.
Within a rule, the value of a select will have been resolved based on the configuration at analysis phase. However, within a macro (at loading phase, before analysis phase), a select() is an opaque type that cannot be inspected. This restricts what operations are possible on the arguments passed to a macro.
The conditions within a select statement are one of:
A config_setting is a collection of build settings, whether defined by Bazel, or user-defined.
User-defined build settings allow users to specify additional configuration, which optionally can be specified as a flag. In addition to specifying build settings within a config_setting, rules can depend directly on them.
In addition, Bazel supports platforms, which is a named collection of constraints. Both a target and host platform can be specified on the command line. More about platforms.
There are many mechanisms to communicate between Soong modules. Because of this, it can be difficult to trace the information communicated between modules.
Dependency tags are the primary way to filter module dependencies by what purpose the dependency serves. For example, to filter for annotation processor plugins in the deps of a Java library module, use ctx.VisitDirectDeps and check the tags:
ctx.VisitDirectDeps(func(module android.Module) {
tag := ctx.OtherModuleDependencyTag(module)
if tag == pluginTag { patchPaths += ":" + strings.Split(ctx.OtherModuleDir(module), "/")[0] }
}
)
At this point the module managing the dependency, may have enough information to cast it to a specific type or interface and perform more specific operations.
For instance, shared libraries and executables have special handling for static library dependencies: where the coverage files and source based ABI dump files are needed explicitly. Based on the dependency tag, the module is cast to a concrete type, like cc.Module, where internal fields are accessed and used to obtain the desired data.
Usage of dependency tags can be more evident when used between module types representing different langauges, as the functions must be exported in Go due to Soong's language-based package layout. For example, rust uses cc module's HasStubVariants.
A common mechanism for a module to communicate information about itself is to define or implement a Go interface.
Some interfaces are common throughout Soong:
SourceFileProducer, by implementing Srcs() PathsOutputFileProducer by implementing OutputFiles(string) (Paths, error)HostToolProvider by implementing HostToolPath() OptionalPathSourceFileProducer and OutputFileProducer are used to resolve references to other modules via android:"path" references.
Modules may define additional interfaces. For example, genrule defines a SourceFileGenerator interface.
Soong has Bazel-inspired providers, but providers are not used in all cases yet.
Usages of providers are the easiest, simplest, and cleanest communication approach in Soong.
In the module providing information, these are specified via SetProvider and SetVariationProvider.
In the module retrieving information, HasProvider and Provider or OtherModuleHasProvider and OtherModuleProvider are used to test existence and retrieve a provider.
Targets primarily communicate with each other via providers in Bazel rule implementations. All rules have access to any of the providers but rules will pick and choose which ones to access based on their needs. For example, all rules can access JavaInfo provider, which provides information about compile and rolled-up runtime jars for javac and java invocations downstream. However, the JavaInfo provider is only useful to java_* rules or rules that need jvm information.
Providers are pieces of information exposed to other modules.
One such provider is DefaultInfo, which contains the default output files and runfiles.
Rule authors can also create custom providers or implement existing providers to communicate information specific to their rule logic. For instance, in Android Starlark cc_object rule implementation, we return a CcInfo provider and a custom CcObjectInfo provider.
For implementation of native rules in Java, ruleContext.getPrerequisite is used to extract providers from dependencies.
depset constructiondepset are used in conjunction with providers to accumulate data efficiently from transitive dependencies. used to accumulate data from transitive dependencies.
exportsSome target have an exports attribute by convention, like java_library.exports. This attribute is commonly used to propagate transitive dependencies to the dependent as though the dependent has a direct edge to the transitive dependencies.