1
0
mirror of https://git.FreeBSD.org/src.git synced 2024-12-06 09:17:25 +00:00
freebsd/share/mk/suite.test.mk

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

127 lines
4.0 KiB
Makefile
Raw Normal View History

# $FreeBSD$
#
# You must include bsd.test.mk instead of this file from your Makefile.
#
# Internal glue for the build of /usr/tests/.
.if !target(__<bsd.test.mk>__)
.error suite.test.mk cannot be included directly.
.endif
Add limited sandbox capability to "make check" == Rationale == r295380 introduced "make check" and consolidated means for running test code in an attempt to simplify running tests. One could either install files/libraries/programs and run "make check", or run "make check" with an explicit CHECKDIR, e.g., `make check CHECKDIR=$(make -V.OBJDIR)``. One criticism that was received is that "make check" should be run with the intent of making dev->test->commit easier, which means that the target audience's workflow should be developers. One developer pattern available in other opensource projects is to run test code from a developer sandbox, instead of installing to a system. == Method == This approach is slightly different from the standard approach, in the sense that it builds and installs into a deterministic directory under .OBJDIR (as I call it, the "sandbox"), then runs "make check" against that. In the event the test run is successful, the deterministic directory is removed to save space. == Approach == bsd.lib.mk, bsd.prog.mk: To support this functionality, a new variable `HAS_TESTS` is being added. HAS_TESTS enables appropriate behavior with bsd.lib.mk and bsd.prog.mk, as follows: - Add "make check" as an available target from the directory. - Pass down appropriate variables via ${TESTS_ENV}, i.e., ${TESTS_LD_LIBRARY_PATH} and ${TESTS_PATH}. One should add "HAS_TESTS" to directories containing tests in them, e.g. from bin/sh/Makefile, HAS_TESTS= SUBDIR.${MK_TESTS}+= tests HAS_TESTS doesn't automatically add the tests subdirectory for flexibility reasons. bsd.opts.mk, src.opts.mk: - The knob ${MK_MAKE_CHECK_USE_SANDBOX} has been added, both to explicitly direct (internally) when to set a deterministic ${DESTDIR} and to also allow users to disable this behavior globally, i.e., via src.conf. - MK_TESTS has been promoted from src.opts.mk to bsd.opts.mk to leverage syntactic sugar for having MK_TESTS be a dependency for MK_MAKE_CHECK_USE_SANDBOX, but to also ensure that src.opts.mk isn't required to use suite.test.mk (which is a dependency of bsd.test.mk). suite.test.mk: - beforecheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified from a no-op to: -- Build. -- Run "make hierarchy" on the sandbox dir. -- Install the tests/files to the sandbox dir. - aftercheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified from a no-op to: -- Remove the sandbox dir. Again, because the dependency order set in bsd.test.mk is beforecheck -> check -> aftercheck, "make check" will not be run unless "beforecheck" completes successfully, and "aftercheck" will not be run unless "beforecheck" and "check" complete successfully. == Caveats == - This target must either be run with MK_INSTALL_AS_USER or as root. Otherwise it will fail when running "make install" as the default user/group for many makefiles when calling INSTALL is root/wheel. - This target must be run from a suitable top-level directory. For example, running tests from `tests/sys/fs/tmpfs` won't work, but `tests/sys/fs` will, because `tests/sys/fs/tmpfs` relies on files installed by `tests/sys/fs`. - Running MK_INSTALL_AS_USER may introduce determinism issues. However, using it could identify deficiences in tests in terms of needing to be run as root, which are not properly articulated in the test requirements. - The doesn't negate the need for running "make installworld" and "make checkworld", etc. Again, this just is intended to simplify the dev->test->commit workflow. == Cleanup done == - CHECKDIR is removed; one can use "MK_MAKE_CHECK_USE_SANDBOX=no" to enable "legacy" (r295380) behavior. MFC after: 2 months Relnotes: yes (CHECKDIR removed; "make check" behavior changed) Requested by: jhb Reviewed by: arch (silence), testing (silence) Differential Revision: D11905
2017-08-14 19:03:05 +00:00
.include <bsd.opts.mk>
# Name of the test suite these tests belong to. Should rarely be changed for
# Makefiles built into the FreeBSD src tree.
TESTSUITE?= FreeBSD
# Knob to control the handling of the Kyuafile for this Makefile.
#
# If 'yes', a Kyuafile exists in the source tree and is installed into
# TESTSDIR.
#
# If 'auto', a Kyuafile is automatically generated based on the list of test
# programs built by the Makefile and is installed into TESTSDIR. This is the
# default and is sufficient in the majority of the cases.
#
# If 'no', no Kyuafile is installed.
KYUAFILE?= auto
# Per-test program interface definition.
#
# The name provided here must match one of the interface names supported by
# Kyua as this is later encoded in the Kyuafile test program definitions.
#TEST_INTERFACE.<test-program>= interface-name
# Metadata properties applicable to all test programs.
#
# All the variables for a test program defined in the Makefile are appended
# to the test program's definition in the Kyuafile. This feature can be
# used to avoid having to explicitly supply a Kyuafile in the source
# directory, allowing the caller Makefile to rely on the KYUAFILE=auto
# behavior defined here.
#TEST_METADATA+= key="value"
# Per-test program metadata properties as a list of key/value pairs.
#
# These per-test program settings _extend_ the values provided in the
# unqualified TEST_METADATA variable.
#TEST_METADATA.<test-program>+= key="value"
.if ${KYUAFILE:tl} != "no"
Merge ^/user/ngie/release-pkg-fix-tests to unbreak how test files are installed after r298107 Summary of changes: - Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that namespacing is kept with FILES appropriately, and that this shouldn't need to be repeated if the namespace changes -- only the definition of PACKAGE needs to be changed - Allow PACKAGE to be overridden by callers instead of forcing it to always be `tests`. In the event we get to the point where things can be split up enough in the base system, it would make more sense to group the tests with the blocks they're a part of, e.g. byacc with byacc-tests, etc - Remove PACKAGE definitions where possible, i.e. where FILES wasn't used previously. - Remove unnecessary TESTSPACKAGE definitions; this has been elided into bsd.tests.mk - Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES; ${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk. - Fix installation of files under data/ subdirectories in lib/libc/tests/hash and lib/libc/tests/net/getaddrinfo - Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup) Document the proposed changes in share/examples/tests/tests/... via examples so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of replacing FILES. share/mk/bsd.README didn't seem like the appropriate method of communicating that info. MFC after: never probably X-MFC with: r298107 PR: 209114 Relnotes: yes Tested with: buildworld, installworld, checkworld; buildworld, packageworld Sponsored by: EMC / Isilon Storage Division
2016-05-04 23:20:53 +00:00
${PACKAGE}FILES+= Kyuafile
${PACKAGE}FILESDIR_Kyuafile= ${TESTSDIR}
.endif
.for _T in ${_TESTS}
_TEST_METADATA.${_T}= ${TEST_METADATA} ${TEST_METADATA.${_T}}
.endfor
.if ${KYUAFILE:tl} == "auto"
CLEANFILES+= Kyuafile Kyuafile.tmp
Kyuafile: Makefile
@{ \
echo '-- Automatically generated by bsd.test.mk.'; \
echo; \
echo 'syntax(2)'; \
echo; \
echo 'test_suite("${TESTSUITE}")'; \
echo; \
} > ${.TARGET}.tmp
.for _T in ${_TESTS}
@echo '${TEST_INTERFACE.${_T}}_test_program{name="${_T}"${_TEST_METADATA.${_T}:C/$/,/:tW:C/^/, /W:C/,$//W}}' \
>>${.TARGET}.tmp
.endfor
.for _T in ${TESTS_SUBDIRS:N.WAIT}
@echo "include(\"${_T}/${.TARGET}\")" >>${.TARGET}.tmp
.endfor
@mv ${.TARGET}.tmp ${.TARGET}
.endif
KYUA?= kyua
# Definition of the "make check" target and supporting variables.
#
# This target, by necessity, can only work for native builds (i.e. a FreeBSD
# host building a release for the same system). The target runs Kyua, which is
# not in the toolchain, and the tests execute code built for the target host.
#
# Due to the dependencies of the binaries built by the source tree and how they
# are used by tests, it is highly possible for a execution of "make test" to
# report bogus results unless the new binaries are put in place.
realcheck: .PHONY
@if ! which -s "${KYUA}"; then \
echo; \
echo "kyua binary not installed at expected location (${.TARGET})"; \
echo; \
echo "Please install via pkg install, or specify the path to the kyua"; \
echo "binary via the \$${KYUA} variable, e.g., "; \
echo "KYUA=\"${LOCALBASE}/bin/kyua\""; \
false; \
fi
Add limited sandbox capability to "make check" == Rationale == r295380 introduced "make check" and consolidated means for running test code in an attempt to simplify running tests. One could either install files/libraries/programs and run "make check", or run "make check" with an explicit CHECKDIR, e.g., `make check CHECKDIR=$(make -V.OBJDIR)``. One criticism that was received is that "make check" should be run with the intent of making dev->test->commit easier, which means that the target audience's workflow should be developers. One developer pattern available in other opensource projects is to run test code from a developer sandbox, instead of installing to a system. == Method == This approach is slightly different from the standard approach, in the sense that it builds and installs into a deterministic directory under .OBJDIR (as I call it, the "sandbox"), then runs "make check" against that. In the event the test run is successful, the deterministic directory is removed to save space. == Approach == bsd.lib.mk, bsd.prog.mk: To support this functionality, a new variable `HAS_TESTS` is being added. HAS_TESTS enables appropriate behavior with bsd.lib.mk and bsd.prog.mk, as follows: - Add "make check" as an available target from the directory. - Pass down appropriate variables via ${TESTS_ENV}, i.e., ${TESTS_LD_LIBRARY_PATH} and ${TESTS_PATH}. One should add "HAS_TESTS" to directories containing tests in them, e.g. from bin/sh/Makefile, HAS_TESTS= SUBDIR.${MK_TESTS}+= tests HAS_TESTS doesn't automatically add the tests subdirectory for flexibility reasons. bsd.opts.mk, src.opts.mk: - The knob ${MK_MAKE_CHECK_USE_SANDBOX} has been added, both to explicitly direct (internally) when to set a deterministic ${DESTDIR} and to also allow users to disable this behavior globally, i.e., via src.conf. - MK_TESTS has been promoted from src.opts.mk to bsd.opts.mk to leverage syntactic sugar for having MK_TESTS be a dependency for MK_MAKE_CHECK_USE_SANDBOX, but to also ensure that src.opts.mk isn't required to use suite.test.mk (which is a dependency of bsd.test.mk). suite.test.mk: - beforecheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified from a no-op to: -- Build. -- Run "make hierarchy" on the sandbox dir. -- Install the tests/files to the sandbox dir. - aftercheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified from a no-op to: -- Remove the sandbox dir. Again, because the dependency order set in bsd.test.mk is beforecheck -> check -> aftercheck, "make check" will not be run unless "beforecheck" completes successfully, and "aftercheck" will not be run unless "beforecheck" and "check" complete successfully. == Caveats == - This target must either be run with MK_INSTALL_AS_USER or as root. Otherwise it will fail when running "make install" as the default user/group for many makefiles when calling INSTALL is root/wheel. - This target must be run from a suitable top-level directory. For example, running tests from `tests/sys/fs/tmpfs` won't work, but `tests/sys/fs` will, because `tests/sys/fs/tmpfs` relies on files installed by `tests/sys/fs`. - Running MK_INSTALL_AS_USER may introduce determinism issues. However, using it could identify deficiences in tests in terms of needing to be run as root, which are not properly articulated in the test requirements. - The doesn't negate the need for running "make installworld" and "make checkworld", etc. Again, this just is intended to simplify the dev->test->commit workflow. == Cleanup done == - CHECKDIR is removed; one can use "MK_MAKE_CHECK_USE_SANDBOX=no" to enable "legacy" (r295380) behavior. MFC after: 2 months Relnotes: yes (CHECKDIR removed; "make check" behavior changed) Requested by: jhb Reviewed by: arch (silence), testing (silence) Differential Revision: D11905
2017-08-14 19:03:05 +00:00
@env ${TESTS_ENV:Q} ${KYUA} test -k ${DESTDIR}${TESTSDIR}/Kyuafile
MAKE_CHECK_SANDBOX_DIR= checkdir
Add limited sandbox capability to "make check" == Rationale == r295380 introduced "make check" and consolidated means for running test code in an attempt to simplify running tests. One could either install files/libraries/programs and run "make check", or run "make check" with an explicit CHECKDIR, e.g., `make check CHECKDIR=$(make -V.OBJDIR)``. One criticism that was received is that "make check" should be run with the intent of making dev->test->commit easier, which means that the target audience's workflow should be developers. One developer pattern available in other opensource projects is to run test code from a developer sandbox, instead of installing to a system. == Method == This approach is slightly different from the standard approach, in the sense that it builds and installs into a deterministic directory under .OBJDIR (as I call it, the "sandbox"), then runs "make check" against that. In the event the test run is successful, the deterministic directory is removed to save space. == Approach == bsd.lib.mk, bsd.prog.mk: To support this functionality, a new variable `HAS_TESTS` is being added. HAS_TESTS enables appropriate behavior with bsd.lib.mk and bsd.prog.mk, as follows: - Add "make check" as an available target from the directory. - Pass down appropriate variables via ${TESTS_ENV}, i.e., ${TESTS_LD_LIBRARY_PATH} and ${TESTS_PATH}. One should add "HAS_TESTS" to directories containing tests in them, e.g. from bin/sh/Makefile, HAS_TESTS= SUBDIR.${MK_TESTS}+= tests HAS_TESTS doesn't automatically add the tests subdirectory for flexibility reasons. bsd.opts.mk, src.opts.mk: - The knob ${MK_MAKE_CHECK_USE_SANDBOX} has been added, both to explicitly direct (internally) when to set a deterministic ${DESTDIR} and to also allow users to disable this behavior globally, i.e., via src.conf. - MK_TESTS has been promoted from src.opts.mk to bsd.opts.mk to leverage syntactic sugar for having MK_TESTS be a dependency for MK_MAKE_CHECK_USE_SANDBOX, but to also ensure that src.opts.mk isn't required to use suite.test.mk (which is a dependency of bsd.test.mk). suite.test.mk: - beforecheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified from a no-op to: -- Build. -- Run "make hierarchy" on the sandbox dir. -- Install the tests/files to the sandbox dir. - aftercheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified from a no-op to: -- Remove the sandbox dir. Again, because the dependency order set in bsd.test.mk is beforecheck -> check -> aftercheck, "make check" will not be run unless "beforecheck" completes successfully, and "aftercheck" will not be run unless "beforecheck" and "check" complete successfully. == Caveats == - This target must either be run with MK_INSTALL_AS_USER or as root. Otherwise it will fail when running "make install" as the default user/group for many makefiles when calling INSTALL is root/wheel. - This target must be run from a suitable top-level directory. For example, running tests from `tests/sys/fs/tmpfs` won't work, but `tests/sys/fs` will, because `tests/sys/fs/tmpfs` relies on files installed by `tests/sys/fs`. - Running MK_INSTALL_AS_USER may introduce determinism issues. However, using it could identify deficiences in tests in terms of needing to be run as root, which are not properly articulated in the test requirements. - The doesn't negate the need for running "make installworld" and "make checkworld", etc. Again, this just is intended to simplify the dev->test->commit workflow. == Cleanup done == - CHECKDIR is removed; one can use "MK_MAKE_CHECK_USE_SANDBOX=no" to enable "legacy" (r295380) behavior. MFC after: 2 months Relnotes: yes (CHECKDIR removed; "make check" behavior changed) Requested by: jhb Reviewed by: arch (silence), testing (silence) Differential Revision: D11905
2017-08-14 19:03:05 +00:00
CLEANDIRS+= ${MAKE_CHECK_SANDBOX_DIR}
.if ${MK_MAKE_CHECK_USE_SANDBOX} != "no" && make(check)
DESTDIR:= ${.OBJDIR}/${MAKE_CHECK_SANDBOX_DIR}
Add limited sandbox capability to "make check" == Rationale == r295380 introduced "make check" and consolidated means for running test code in an attempt to simplify running tests. One could either install files/libraries/programs and run "make check", or run "make check" with an explicit CHECKDIR, e.g., `make check CHECKDIR=$(make -V.OBJDIR)``. One criticism that was received is that "make check" should be run with the intent of making dev->test->commit easier, which means that the target audience's workflow should be developers. One developer pattern available in other opensource projects is to run test code from a developer sandbox, instead of installing to a system. == Method == This approach is slightly different from the standard approach, in the sense that it builds and installs into a deterministic directory under .OBJDIR (as I call it, the "sandbox"), then runs "make check" against that. In the event the test run is successful, the deterministic directory is removed to save space. == Approach == bsd.lib.mk, bsd.prog.mk: To support this functionality, a new variable `HAS_TESTS` is being added. HAS_TESTS enables appropriate behavior with bsd.lib.mk and bsd.prog.mk, as follows: - Add "make check" as an available target from the directory. - Pass down appropriate variables via ${TESTS_ENV}, i.e., ${TESTS_LD_LIBRARY_PATH} and ${TESTS_PATH}. One should add "HAS_TESTS" to directories containing tests in them, e.g. from bin/sh/Makefile, HAS_TESTS= SUBDIR.${MK_TESTS}+= tests HAS_TESTS doesn't automatically add the tests subdirectory for flexibility reasons. bsd.opts.mk, src.opts.mk: - The knob ${MK_MAKE_CHECK_USE_SANDBOX} has been added, both to explicitly direct (internally) when to set a deterministic ${DESTDIR} and to also allow users to disable this behavior globally, i.e., via src.conf. - MK_TESTS has been promoted from src.opts.mk to bsd.opts.mk to leverage syntactic sugar for having MK_TESTS be a dependency for MK_MAKE_CHECK_USE_SANDBOX, but to also ensure that src.opts.mk isn't required to use suite.test.mk (which is a dependency of bsd.test.mk). suite.test.mk: - beforecheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified from a no-op to: -- Build. -- Run "make hierarchy" on the sandbox dir. -- Install the tests/files to the sandbox dir. - aftercheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified from a no-op to: -- Remove the sandbox dir. Again, because the dependency order set in bsd.test.mk is beforecheck -> check -> aftercheck, "make check" will not be run unless "beforecheck" completes successfully, and "aftercheck" will not be run unless "beforecheck" and "check" complete successfully. == Caveats == - This target must either be run with MK_INSTALL_AS_USER or as root. Otherwise it will fail when running "make install" as the default user/group for many makefiles when calling INSTALL is root/wheel. - This target must be run from a suitable top-level directory. For example, running tests from `tests/sys/fs/tmpfs` won't work, but `tests/sys/fs` will, because `tests/sys/fs/tmpfs` relies on files installed by `tests/sys/fs`. - Running MK_INSTALL_AS_USER may introduce determinism issues. However, using it could identify deficiences in tests in terms of needing to be run as root, which are not properly articulated in the test requirements. - The doesn't negate the need for running "make installworld" and "make checkworld", etc. Again, this just is intended to simplify the dev->test->commit workflow. == Cleanup done == - CHECKDIR is removed; one can use "MK_MAKE_CHECK_USE_SANDBOX=no" to enable "legacy" (r295380) behavior. MFC after: 2 months Relnotes: yes (CHECKDIR removed; "make check" behavior changed) Requested by: jhb Reviewed by: arch (silence), testing (silence) Differential Revision: D11905
2017-08-14 19:03:05 +00:00
beforecheck:
.for t in clean depend all
@cd ${.CURDIR} && ${MAKE} $t
.endfor
@cd ${SRCTOP} && ${MAKE} hierarchy DESTDIR=${DESTDIR}
@cd ${.CURDIR} && ${MAKE} install \
DESTDIR=${DESTDIR}
# NOTE: this is intentional to ensure that "make check" can be run multiple
# times. "aftercheck" won't be run if "make check" fails, is interrupted,
# etc.
aftercheck:
@cd ${.CURDIR} && ${MAKE} clean
@test ! -e ${DESTDIR} || chflags -R 0 "${DESTDIR}"
@rm -Rf "${DESTDIR}"
Add limited sandbox capability to "make check" == Rationale == r295380 introduced "make check" and consolidated means for running test code in an attempt to simplify running tests. One could either install files/libraries/programs and run "make check", or run "make check" with an explicit CHECKDIR, e.g., `make check CHECKDIR=$(make -V.OBJDIR)``. One criticism that was received is that "make check" should be run with the intent of making dev->test->commit easier, which means that the target audience's workflow should be developers. One developer pattern available in other opensource projects is to run test code from a developer sandbox, instead of installing to a system. == Method == This approach is slightly different from the standard approach, in the sense that it builds and installs into a deterministic directory under .OBJDIR (as I call it, the "sandbox"), then runs "make check" against that. In the event the test run is successful, the deterministic directory is removed to save space. == Approach == bsd.lib.mk, bsd.prog.mk: To support this functionality, a new variable `HAS_TESTS` is being added. HAS_TESTS enables appropriate behavior with bsd.lib.mk and bsd.prog.mk, as follows: - Add "make check" as an available target from the directory. - Pass down appropriate variables via ${TESTS_ENV}, i.e., ${TESTS_LD_LIBRARY_PATH} and ${TESTS_PATH}. One should add "HAS_TESTS" to directories containing tests in them, e.g. from bin/sh/Makefile, HAS_TESTS= SUBDIR.${MK_TESTS}+= tests HAS_TESTS doesn't automatically add the tests subdirectory for flexibility reasons. bsd.opts.mk, src.opts.mk: - The knob ${MK_MAKE_CHECK_USE_SANDBOX} has been added, both to explicitly direct (internally) when to set a deterministic ${DESTDIR} and to also allow users to disable this behavior globally, i.e., via src.conf. - MK_TESTS has been promoted from src.opts.mk to bsd.opts.mk to leverage syntactic sugar for having MK_TESTS be a dependency for MK_MAKE_CHECK_USE_SANDBOX, but to also ensure that src.opts.mk isn't required to use suite.test.mk (which is a dependency of bsd.test.mk). suite.test.mk: - beforecheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified from a no-op to: -- Build. -- Run "make hierarchy" on the sandbox dir. -- Install the tests/files to the sandbox dir. - aftercheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified from a no-op to: -- Remove the sandbox dir. Again, because the dependency order set in bsd.test.mk is beforecheck -> check -> aftercheck, "make check" will not be run unless "beforecheck" completes successfully, and "aftercheck" will not be run unless "beforecheck" and "check" complete successfully. == Caveats == - This target must either be run with MK_INSTALL_AS_USER or as root. Otherwise it will fail when running "make install" as the default user/group for many makefiles when calling INSTALL is root/wheel. - This target must be run from a suitable top-level directory. For example, running tests from `tests/sys/fs/tmpfs` won't work, but `tests/sys/fs` will, because `tests/sys/fs/tmpfs` relies on files installed by `tests/sys/fs`. - Running MK_INSTALL_AS_USER may introduce determinism issues. However, using it could identify deficiences in tests in terms of needing to be run as root, which are not properly articulated in the test requirements. - The doesn't negate the need for running "make installworld" and "make checkworld", etc. Again, this just is intended to simplify the dev->test->commit workflow. == Cleanup done == - CHECKDIR is removed; one can use "MK_MAKE_CHECK_USE_SANDBOX=no" to enable "legacy" (r295380) behavior. MFC after: 2 months Relnotes: yes (CHECKDIR removed; "make check" behavior changed) Requested by: jhb Reviewed by: arch (silence), testing (silence) Differential Revision: D11905
2017-08-14 19:03:05 +00:00
.endif