emacs/cross/ndk-build

NDK BUILD SYSTEM IMPLEMENTATION

Copyright (C) 2023 Free Software Foundation, Inc.
See the end of the file for license conditions.

Emacs implements ndk-build itself, because the version that comes with
the Android NDK is not easy to use from another Makefile, and keeps
accumulating incompatible changes.

The Emacs implementation of ndk-build consists of one m4 file:

  m4/ndk-build.m4

four Makefiles in build-aux, run during configure:

  build-aux/ndk-build-helper-1.mk
  build-aux/ndk-build-helper-2.mk
  build-aux/ndk-build-helper-3.mk
  build-aux/ndk-build-helper.mk

one awk script in build-awx, run during configure:

  build-aux/ndk-module-extract.awk

seven Makefiles in cross/ndk-build,

  cross/ndk-build/ndk-build-shared-library.mk
  cross/ndk-build/ndk-build-static-library.mk
  cross/ndk-build/ndk-build-executable.mk
  cross/ndk-build/ndk-clear-vars.mk
  cross/ndk-build/ndk-prebuilt-shared-library.mk
  cross/ndk-build/ndk-prebuilt-static-library.mk
  cross/ndk-build/ndk-resolve.mk

and finally, two more Makefiles in cross/ndk-build, generated by
configure:

  cross/ndk-build/Makefile     (generated from cross/ndk-build/Makefile.in)
  cross/ndk-build/ndk-build.mk (generated from cross/ndk-build/ndk-build.mk.in)

m4/ndk-build.m4 is a collection of macros which are used by the
configure script to set up the ndk-build system, look for modules, add
the appropriate options to LIBS and CFLAGS, and generate the Makefiles
necessary to build the rest of Emacs.

Immediately after determining the list of directories in which to look
for ``Android.mk'' files, the version and type of Android system being
built for, configure calls:

  ndk_INIT([$android_abi], [$ANDROID_SDK], [cross/ndk-build])

This expands to a sequence of shell script that enumerates all of the
Android.mk files specified in "$with_ndk_path", sets up some shell
functions used by the rest of the ndk-build code run by the configure
script, and teaches the ndk-build system that the Makefiles to be
generated are found in the directory "cross/ndk-build/Makefile".

When configure is cross-compiling for Android, the macro
EMACS_CHECK_MODULES will expand to the macro ndk_CHECK_MODULES,
instead of pkg-config.m4's PKG_CHECK_MODULES.  Thus, the following
code:

    EMACS_CHECK_MODULES([PNG], [libpng >= 1.0.0])

will actually expand to:

    ndk_CHECK_MODULES([PNG], [libpng >= 1.0.0], [HAVE_PNG=yes],
		      [HAVE_PNG=no])

which in turn expands to a sequence shell script that first invokes:

    make -f build-aux/ndk-build-helper.mk

for each ``Android.mk'' file found by ndk_INIT, with the following
variables given to Make:

    EMACS_SRCDIR=.  # the source directory (in which configure is running)
    BUILD_AUXDIR=$ndk_AUX_DIR # the build-aux directory
    EMACS_ABI=$ndk_ABI # this is the $android_abi given to ndk_INIT
    ANDROID_MAKEFILE="/opt/android/libpng/Android.mk"
    ANDROID_MODULE_DIRECTORY="/opt/android/libpng"
    NDK_BUILD_DIR="$ndk_DIR" # this is the directory given as to ndk_INIT

build-aux/ndk-build-helper.mk will then evaluate the contents
$(ANDROID_MAKEFILE), the ``Android.mk'' file, for the first time.  The
purpose of this evaluation is to establish a list of packages (or
modules) provided by the ``Android.mk'' file, and the corresponding
Makefile targets and compiler and linker flags required to build and
link to those tagets.

Before doing so, build-aux/ndk-build-helper.mk will define several
variables and functions required by all ``Android.mk'' files.  The
most important of these are:

  my-dir # the directory containing the Android.mk file.
  BUILD_SHARED_LIBRARY # build-aux/ndk-build-helper-1.mk
  BUILD_STATIC_LIBRARY # build-aux/ndk-build-helper-2.mk
  BUILD_EXECUTABLE # build-aux/ndk-build-helper-3.mk
  CLEAR_VARS # build-aux/ndk-build-helper-4.mk

Then, ``Android.mk'' will include $(CLEAN_VARS), possibly other
``Android.mk'' files, (to clear variables previously set), set several
variables describing each module to the ndk-build system, and include
one of $(BUILD_SHARED_LIBRARY), $(BUILD_STATIC_LIBRARY) and
$(BUILD_EXECUTABLE).

Each one of those three scripts will then read from the variables set
by ``Android.mk'', resolve dependencies, and print out some text
describing the module to Emacs.	 For example, the shared library
module "libpng" results in the following text being printed:

Building shared
libpng
/opt/android/libpng/png.c /opt/android/libpng/pngerror.c /opt/android/libpng/pngget.c /opt/android/libpng/pngmem.c /opt/android/libpng/pngpread.c /opt/android/libpng/pngread.c /opt/android/libpng/pngrio.c /opt/android/libpng/pngrtran.c /opt/android/libpng/pngrutil.c /opt/android/libpng/pngset.c /opt/android/libpng/pngtrans.c /opt/android/libpng/pngwio.c /opt/android/libpng/pngwrite.c /opt/android/libpng/pngwtran.c /opt/android/libpng/pngwutil.c
-I/opt/android/libpng

  -L/opt/emacs/cross/ndk-build -l:libpng_emacs.so
libpng_emacs.so
End

The output is arranged as follows:

  - The first line consists of the word ``Building'', followed by
    either ``shared'', ``static'', or ``executable'', depending on
    what type of module being built.

  - The second line consists of the name of the module currently being
    built.

  - The third line consists of all of the source code files comprising
    the module.

  - The fourth line consists of the text that has to be added to
    CFLAGS in order to find the includes associated with the module.

  - The fifth line consists of the text that has to be added to LIBS
    in order to link with this module and all of its dependencies.

  - The sixth line consists of the Make targets (more on this later)
    that will build the final shared object or library archive of this
    module, along with all of its dependencies.

  - The seventh line is either empty, or the name of a dependency on
    the C++ standard library.  This is used to determine whether or
    not Emacs will include the C++ standard library in the application
    package.

The output from Make is given to an awk script,
build-aux/ndk-module-extract.awk.  This is responsible for parsing the
that output and filtering out modules other than what is being built:

  awk -f build-aux/ndk-module-extract.awk MODULE=libpng

eventually generating this section of shell script:

module_name=libpng
module_kind=shared
module_src="/opt/android/libpng/png.c /opt/android/libpng/pngerror.c /opt/android/libpng/pngget.c /opt/android/libpng/pngmem.c /opt/android/libpng/pngpread.c /opt/android/libpng/pngread.c /opt/android/libpng/pngrio.c /opt/android/libpng/pngrtran.c /opt/android/libpng/pngrutil.c /opt/android/libpng/pngset.c /opt/android/libpng/pngtrans.c /opt/android/libpng/pngwio.c /opt/android/libpng/pngwrite.c /opt/android/libpng/pngwtran.c /opt/android/libpng/pngwutil.c"
module_includes="-I/opt/android/libpng"
module_cflags=""
module_ldflags="  -L/opt/emacs/cross/ndk-build -l:libpng_emacs.so"
module_target="libpng_emacs.so"
module_cxx_deps=""
module_imports=""

which is then evaluated by `configure'.	 Once the variable
`module_name' is set, configure apends the remaining
$(module_includes), $(module_cflags) and $(module_ldflags) to the
module's CFLAGS and LIBS variables, and appends the list of Makefile
targets specified to the variable NDK_BUILD_MODULES.

In some cases, an ``Android.mk'' file may chose to import a module
defined in ``--with-ndk-path'', but not defined inside its own
``Android.mk'' file.  build-aux/ndk-build-helper.mk defines the
`import-module' function to add the modules being imported to a
variable, which is then printed out after ``ndk-build-helper.mk''
completes.  For example, libxml2 imports the ``libicucc'' module,
which results in the following text being printed:

Building shared
libxml2
/home/oldosfan/libxml2/SAX.c /home/oldosfan/libxml2/entities.c /home/oldosfan/libxml2/encoding.c /home/oldosfan/libxml2/error.c /home/oldosfan/libxml2/parserInternals.c /home/oldosfan/libxml2/parser.c /home/oldosfan/libxml2/tree.c /home/oldosfan/libxml2/hash.c /home/oldosfan/libxml2/list.c /home/oldosfan/libxml2/xmlIO.c /home/oldosfan/libxml2/xmlmemory.c /home/oldosfan/libxml2/uri.c /home/oldosfan/libxml2/valid.c /home/oldosfan/libxml2/xlink.c /home/oldosfan/libxml2/debugXML.c /home/oldosfan/libxml2/xpath.c /home/oldosfan/libxml2/xpointer.c /home/oldosfan/libxml2/xinclude.c /home/oldosfan/libxml2/DOCBparser.c /home/oldosfan/libxml2/catalog.c /home/oldosfan/libxml2/globals.c /home/oldosfan/libxml2/threads.c /home/oldosfan/libxml2/c14n.c /home/oldosfan/libxml2/xmlstring.c /home/oldosfan/libxml2/buf.c /home/oldosfan/libxml2/xmlregexp.c /home/oldosfan/libxml2/xmlschemas.c /home/oldosfan/libxml2/xmlschemastypes.c /home/oldosfan/libxml2/xmlunicode.c /home/oldosfan/libxml2/xmlreader.c /home/oldosfan/libxml2/relaxng.c /home/oldosfan/libxml2/dict.c /home/oldosfan/libxml2/SAX2.c /home/oldosfan/libxml2/xmlwriter.c /home/oldosfan/libxml2/legacy.c /home/oldosfan/libxml2/chvalid.c /home/oldosfan/libxml2/pattern.c /home/oldosfan/libxml2/xmlsave.c /home/oldosfan/libxml2/xmlmodule.c /home/oldosfan/libxml2/schematron.c /home/oldosfan/libxml2/SAX.c /home/oldosfan/libxml2/entities.c /home/oldosfan/libxml2/encoding.c /home/oldosfan/libxml2/error.c /home/oldosfan/libxml2/parserInternals.c /home/oldosfan/libxml2/parser.c /home/oldosfan/libxml2/tree.c /home/oldosfan/libxml2/hash.c /home/oldosfan/libxml2/list.c /home/oldosfan/libxml2/xmlIO.c /home/oldosfan/libxml2/xmlmemory.c /home/oldosfan/libxml2/uri.c /home/oldosfan/libxml2/valid.c /home/oldosfan/libxml2/xlink.c /home/oldosfan/libxml2/debugXML.c /home/oldosfan/libxml2/xpath.c /home/oldosfan/libxml2/xpointer.c /home/oldosfan/libxml2/xinclude.c /home/oldosfan/libxml2/DOCBparser.c /home/oldosfan/libxml2/catalog.c /home/oldosfan/libxml2/globals.c /home/oldosfan/libxml2/threads.c /home/oldosfan/libxml2/c14n.c /home/oldosfan/libxml2/xmlstring.c /home/oldosfan/libxml2/buf.c /home/oldosfan/libxml2/xmlregexp.c /home/oldosfan/libxml2/xmlschemas.c /home/oldosfan/libxml2/xmlschemastypes.c /home/oldosfan/libxml2/xmlunicode.c /home/oldosfan/libxml2/xmlreader.c /home/oldosfan/libxml2/relaxng.c /home/oldosfan/libxml2/dict.c /home/oldosfan/libxml2/SAX2.c /home/oldosfan/libxml2/xmlwriter.c /home/oldosfan/libxml2/legacy.c /home/oldosfan/libxml2/chvalid.c /home/oldosfan/libxml2/pattern.c /home/oldosfan/libxml2/xmlsave.c /home/oldosfan/libxml2/xmlmodule.c /home/oldosfan/libxml2/schematron.c


  -L/home/oldosfan/emacs-dev/emacs-android/cross/ndk-build -l:libxml2_emacs.so -l:libicuuc_emacs.so
libxml2_emacs.so libicuuc_emacs.so
End
Start Imports
libicuuc
End Imports

Upon encountering the ``Start Imports'' section,
build-aux/ndk-module-extract.awk collects all imports until it
encounters the line ``End Imports'', at which point it prints:

module_imports="libicuuc"

Then, if the list of imports is not empty, ndk_CHECK_MODULES
additionally calls itself for each import before appending the
module's own ``Android.mk'', ensuring that the module's imported
dependencies are included by $ndk_DIR/Makefile before itself.

Finally, immediately before generating src/Makefile.android, configure
expands:

  ndk_CONFIG_FILES

to generate $ndk_DIR/Makefile and $ndk_DIR/ndk-build.mk.

Now, the $ndk_DIR directory is set up to build all modules upon which
depends, and $ndk_DIR/ndk-build.mk includes a list of files required
to link Emacs, along with the rules to chdir into $ndk_DIR in order to
build them.

$ndk_DIR/ndk-build.mk is included by cross/src/Makefile
(Makefile.android) and java/Makefile.  It defines three different
variables:

  NDK_BUILD_MODULES	the file names of all modules to be built.
  NDK_BUILD_STATIC	absolute names of all library archives
			to be built.
  NDK_BUILD_SHARED	absolute names of all shared libraries to
			be built.

and then proceeds to define rules to build each of the modules in
$(NDK_BUILD_MODULES).

cross/src/Makefile arranges to have all dependencies of Emacs not
already built built before linking ``libemacs.so'' with them.

java/Makefile additionally arranges to have all shared object
dependencies built before the application package is built, which is
normally redundant because they should have already been built before
linking ``libemacs.so''.

Building the modules is performed through $ndk_DIR/Makefile, which
contains the actual implementation of the ``ndk-build'' build system.
First, it defines certain variables constant within the ``ndk-build''
build system, such as the files included by ``Android.mk'' to build
shared or static libraries, and CLEAR_VARS.  The most important of
these are:

  CLEAR_VARS			cross/ndk-build/ndk-clear-vars.mk
  BUILD_EXECUTABLE		cross/ndk-build/ndk-build-executable.mk
  BUILD_SHARED_LIBRARY		cross/ndk-build/ndk-build-shared-library.mk
  BUILD_STATIC_LIBRARY		cross/ndk-build/ndk-build-static-library.mk
  PREBUILT_SHARED_LIBRARY	cross/ndk-build/ndk-prebuilt-shared-library.mk
  PREBUILT_STATIC_LIBRARY	cross/ndk-build/ndk-prebuilt-static-library.mk

Then, it loads each Emacs dependency's ``Android.mk'' file.  For each
module defined there, ``Android.mk'' includes $(CLEAR_VARS) to unset
all variables specific to each module, and then includes
$(BUILD_SHARED_LIBRARY) or $(BUILD_STATIC_LIBRARY) for each shared or
static library module.

This results in cross/ndk-build/ndk-build-shared-library.mk or
cross/ndk-build/ndk-build-static-library being included, just like the
Makefiles in build-aux were inside the configure script.

Each one of those two scripts then defines rules to build all of the
object files associated with the module, and then link or archive
them.  The name under which the module is linked is the same as the
Make target found on the sixth line of output from
build-aux/ndk-build-helper.mk.

In doing so, they both include the file ndk-resolve.mk.
ndk-resolve.mk is expected to recursively add all of the exported
CFLAGS and includes of any dependencies to the compiler and linker
command lines for the module being built.

When building a shared library module, ndk-resolve.mk is also expected
to define the variables NDK_LOCAL_A_NAMES_$(LOCAL_MODULE) and
NDK_WHOLE_A_NAMES_$(LOCAL_MODULE), containing all static library
dependencies' archive files.  They are to be linked in to the
resulting shared object file.

This is done by including cross/ndk-build/ndk-resolve.mk each time a
shared or static library module is going to be built.  How is this
done?

First, ndk-resolve.mk saves the LOCAL_PATH, LOCAL_STATIC_LIBRARIES,
LOCAL_SHARED_LIBRARIES, LOCAL_EXPORT_CFLAGS and
LOCAL_EXPORT_C_INCLUDES from the module.

Next, ndk-resolve loops through the dependencies the module has
specified, appending its CFLAGS and includes to the command line for
the current module.

Then, that process is repeated for each such dependency which has not
already been resolved, until all dependencies have been resolved.

libpng is a very simple module, providing only a single shared object
module.	 This module is named libpng_emacs.so and is eventually built
and packaged into the library directory of the Emacs application
package.  Now, let us look at a more complex module, libwebp:



When built with libwebp, Emacs depends on a single library,
libwebpdemux.  This library is named ``libwebpdemux'' on Unix systems,
and that is the name by which it is found with pkg-config.

However, the library's module is only named ``webpdemux'' on Android.
When ndk_CHECK_MODULES begins to look for a module, it first tries to
see if its name is found in the variable `ndk_package_map', which was
set inside ndk_INIT.  In this case, it finds the following word:

  libwebpdemux:webpdemux

and immediately replaces ``libwebpdemux'' with ``webpdemux''.

Then, it locates the ``Android.mk'' file containing a static library
module named webpdemux and gives the output from
build-aux/ndk-build-helper.mk to the awk script, resulting in:

module_name=webpdemux
module_kind=static
module_src="/opt/android/webp/src/demux/anim_decode.c /opt/android/webp/src/demux/demux.c"
module_includes="-I/opt/android/webp/src"
module_cflags=""
module_ldflags=" cross/ndk-build/libwebpdemux.a cross/ndk-build/libwebp.a cross/ndk-build/libwebpdecoder_static.a "
module_target="libwebpdemux.a libwebp.a libwebpdecoder_static.a"

The attentive reader will notice that in addition to the
``libwebpdemux.a'' archive associated with the ``webpdemux'' library,
Emacs has been made to link with two additional libraries.  This is
because the ``webpdemux'' module specifies a dependency on the
``webp'' module (defined in the same Android.mk).
build-aux/ndk-build-helper.mk resolved that dependency, noticing that
it in turn specified another dependency on ``webpdecoder_static'',
which in turn was added to the linker command line and list of targets
to build.

As a result, all three dependencies will be built and linked to Emacs,
instead of just the single ``webpdemux'' dependency that was
specified.



This file is part of GNU Emacs.

GNU Emacs is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

GNU Emacs is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.