INDEX_JOBS currently defaults to 2 for the make index target.
MAKE_JOBS_NUMBER in bsd.port.mk currently uses kern.smp.cpus to select
a default. Modify INDEX_JOBS to match.
Nominal and repeatable performance improvements of ~20-30% were observed.
CR: D381
Reviewed by: swills, bdrewery
Approved by: portmgr
if available, for 'make index'. This removes the need to have
perl installed. [1]
- Fix output of the file to have the proper /usr/ports/ORIGIN regardless
of real PORTSDIR or CURDIR [2]
PR: ports/183095 [1]
Submitted by: ak [2]
With hat: portmgr
- bsd.port.mk: add INDEX_PORTS, to support INDEX creation for a subset of the ports tree [1]
- bsd.port.mk: call target "install-rc-script" before "post-install" [2]
- [patch] ports/Mk bsd.port.mk order if groups/users are created by package [3]
- [bsd.port.mk] [patch] reaper of the dead: md5 has been in /sbin for a while [4]
- [bsd.port.mk] [patch] remove support for pre 7.x systems (b.*.m) [5]
- [patch] [bsd.port.mk] reaper of the dead: are three variable defintions needed [6]
PR: ports/156575 [1],
ports/139116 [2],
ports/152498 [3],
ports/155983 [4],
ports/155510 [5],
ports/156340 [6]
Submitted by: Florent Thoumie <flz@xbsd.org> [1],
Sergey Skvortsov <skv@freebsd.org> [2],
Olli Hauer <ohauer@FreeBSD.org> [3],
Eitan Adler <lists@eitanadler.com> [4],
Eitan Adler <lists@eitanadler.com> [5],
Eitan Adler <lists@eitanadler.com> [6]
targets.
* Use /rescue/sh for index builds instead of /bin/sh, when it exists.
The former is statically linked and faster to execute, which becomes
significant when executing it tens of thousands of times. This
trick can be used with other recursive targets by passing in
__MAKE_SHELL.
* Get rid of make variable assignments that use != command invocations
in the critical path, using several methods:
- rewriting logic to use shell or make builtins instead of external command executions
- macroizing commands and executing them in the targets where they
are needed instead of with every invocation of make
- precomputing the results of invariant commands in
bsd.port.subdir.mk and passing them in explicitly to child makes,
and using this to avoid recalculation in all the children. NB: the
commands are still run one per top-level subdirectory but this
does not currently seem to be a major issue. They could be moved
further up into the top-level Makefile at the cost of some
cleanliness.
- Committers are strongly discouraged from adding further "bare" !=
assignments to the ports tree, even in their own ports. One of
the above strategies should be used to avoid future bloat.
* Rewrite the core 'describe' target to work entirely within a single
shell process using only builtin commands. The old version is
retained as a backup for use on systems older than 603104, which
does not have the make :u modifier. This cuts down the number of
processes executed during the course of a 'make index' by an order
of magnitude, and we are essentially now amortized to the minimum of
a single make + sh instance per port, plus whatever commands the
port makefile itself executes (which are usually unnecessary and
bogus).
* Less validation of the WWW: target is performed; this can become
policed at a port level by portlint. Specifically we look at the
second word of the first line beginning with "WWW:" in pkg-descr,
and append "http://" to it unless it already begins with "http://",
"https://" or "ftp://". Thanks to dougb for the idea of how to
extract WWW: using shell builtins.
* Use the "true" shell builtin instead of echo > /dev/null for a
measurable decrease in CPU use.
* Add a note about dubious escaping strategy in bsd.port.subdir.mk
* Minor change in output of 'make describe': it no longer strips
trailing CR characters from pkg-descr files with MSDOS CR/LF
termination. Instead the makeindex perl script that post-processes
make describe into the INDEX is tweaked to strip on input.
The bottom line is that on my test hardware INDEX builds are now
faster by more than a factor of 2 and with a reduction in system time
by a factor of 4-8 depending on configuration.
to use csup for both ports and src. It should be possible to use csup
for src and portsnap for ports.
If you define SUP_UPDATE without PORTSSUPFILE defined, you'll get
portsnap by default now.
PR: ports/113819
Submitted by: ale
Requested by: many
Hat: portmgr
you had to manually select one of 3 methods: SUP_UPDATE, CVS_UPDATE,
or PORTSNAP_UPDATE. The latter is now obsolete.
This should help some novice administrators just-start their system.
PR: ports/105835
Submitted by: erwin
[1] Remove DEPENDS
[2] Add a new pkg-plist keyword @stopdaemon
[3] Provide rc.subr script suffix as RC_SUBR_SUFFIX
[4] Add USE_CDRTOOLS to abstract dependency on sysutils/cdrtool[-cjk]
[5] Add DOS2UNIX_REGEX, allowing to pass a filename regex to CRLF replacement
routine
[6] Support partially translated manpages in MAN* variables
[7] Fix USE_LDCONFIG on ports with @cwd in pkg-plist
[8] Remove support for XFree86 3.X
[9] Add user settable WITH_DEBUG flag that turns on -g in CFLAGS and resets
STRIP
[9] Fix `make install' to refuse overwriting of older version of the port
[10] New category ports-mgmt
[11] New category gnustep
[12] Removed category picobsd
bsd.java.mk
[13] Remove deprecated syntax
bsd.tcl.mk
[14] Provide TCL_VER when USE_TK[_BUILD] is defined. Fixes cad/netgen
ports/Makefile
[15] Make csup default `make update' method on FreeBSD 6.2 and up
PR: ports/99742 [1], ports/93373 [2], ports/100915 [3],
ports/105161 [4], ports/106029 [5], ports/106252 [6],
ports/106235 [7], ports/100939 [9], ports/97507 [10],
ports/103931 [11], ports/106921 [12], ports/104136 [13],
ports/105215 [15]
Submitted by: sem [1] [2], Matthias Andree <matthias.andree@gmx.de> [3],
nork [4], pav [5] [7] [14], Nick Barkas <snb@threerings.net> [6],
flz [8], gabor [9], shaun [10], erwin [12], hq [13],
Gurkan Sengun <gurkan@linuks.mine.nu> [11]
symlink. [1]
- Allow use of alternate http download utilities with 'make
fetchindex'. [2]
PR: ports/35767 [1], ports/92588 [2]
Submitted by: Peter Jeremy <PeterJeremy at optushome dot com dot au> [1],
Alexander Logvinov <user at blg dot akavia dot ru> [2]
EMACS ports. [1]
- Allow building a port as root using an NFS-mounted /usr/ports if the
server maps root to a UID other than root. [2]
- Make 'BROKEN' and 'IGNORED' ports exit their "make install" with a fail
status rather than success. [3]
- Improve behavior when dealing with versioned dependencies. [4]
- Fix false positives in check-conflicts target. [5]
- Remove obsolete bzip2 code. [6]
- Add physical category net-p2p. [7]
- Don't fetch INDEXFILE if not necessary; respect FETCH_ENV. [8], [11]
- INDEX can now be moved outside of ports tree. [9]
- Add ghostscript-gpl. [10]
- Remove obsolete USE_MESA. [12]
- Force pkg_install tools from ports on FreeBSD 4.10 and older. [13]
- Document ALWAYS_KEEP_DISTFILES. [14]
- Remove USE_REINPLACE from bsd.port.mk USE_DOS2UNIX patch. [15]
PR: ports/37596 [1], ports/57259 [2], ports/63216 [3],
ports/89448 [4], ports/89710 [5], ports/88996 [6],
ports/89260 [7], ports/89363 [8], ports/89809 [9],
ports/89853 [10], ports/91086 [11], ports/91710 [12],
ports/91727 [13], ports/92111 [14], ports/92124 [15]
Submitted by: Jay Sachs <jay at eziba dot com> [1], sem [1, 3, 8, 12],
Andrew Heybey <ath at niksun dot com> [2], Jamie Jones
<jamie at thompson dot bishopston dot net>, tobez [4], Mark
Andrews <Mark_Andrews at isc dot org> [5], edwin [6, 11, 15],
pav [7, 13], Peter Jeremy <PeterJeremy at optushome dot com
dot au> [9], Ulrich Spoerlein <q at galgenberg dot net> [10],
netchild [11], erwin [14]
Reviewed by: kris, clement (partially)
very long and if some dependencies can't be added into the list.
[1]
- Fix FAM support. Make gamin the default FAM system. [2]
- Introduce new 'quicksearch' target to show only port, path and
info section of the matching ports. [3]
- Introduce new category - rubygems. [4]
- Fix stale dependencies while installing qmail slaveport and
another port that depends on qmail. [5]
- Add commentary for describes target in bsd.port.mk. [6]
- Fix warning issued during make index on archs !368. [7]
- Add USE_DOS2UNIX variable. If set to "YES", remove the ^M from
all files under ${WRKSRC}. If set to a string, remove in all files
under ${WRKSRC} with one of these names the ^Ms. [8]
- Add new variables PERL_RUN_DEPENDS and PERL_BUILD_DEPENDS by
checking the existance of the Perl modules with the "perl -e 'use
module;'" command. [9]
- Fix bsd.port.mk variable quoting issues. No quoting is necessary
anymore either in the Makefile or on the command line. Affected
variables include:
BROKEN
FORBIDDEN
IGNORE
MANUAL_PACKAGE_BUILD
NO_CDROM
NO_PACKAGE
RESTRICTED
[10]
- Add NOFETCHFILES variable. If set, don't download these files
from the ${MASTER_SITES} or ${MASTER_SITE_BACKUP} (but do from
${MASTER_SITE_OVERRIDE}). [11]
- Improve 'search' target output. [12]
- Add a new virtual category for Amateur Radio - hamradio. [13]
- Cleanup some old/unused pathes in bsd.port.mk. [14]
- Add @dirrmtry for plists which does the same as:
"@unexec rmdir %D/foo 2>/dev/null || true" [15]
- Remove virtual category - offix. [16]
- Use portsnap instead of cvsup or cvs on "make update" in
/usr/ports. [17]
- Move location of bsd.autotools.mk within bsd.port.mk [18]
- Add bsd.linux-rpm.mk, fix INSTALLS_SHLIB for Linux ports [19]
- Use new USE_RC_SUBR format for FreeBSD version >= 700007 [20]
- Replace the string "FreeBSD" by "The FreeBSD Project" in the
security warning [21]
- Add bsd.local.mk for local modification to ports framework. [22]
- Replace rcNG spelling by rc.d [23]
- Remove superfluous USE_REINPLACE. [24]
Special thanks to: linimon for spending hours with all these patches
clement for fixes
kris for help with pointyhat
PR: ports/86310 [1], ports/89498 [2], ports/83530 [3],
ports/83789 [4], ports/84053 [5], ports/86281 [6],
ports/87214 [7], ports/87234 [8], ports/87318 [9],
ports/87396 [10], ports/87605 [11], ports/87840 [12],
ports/88230 [13], ports/88493 [14], ports/88711 [15],
ports/88751 [16], ports/89281 [17], ports/89999 [18],
ports/90031 [19], ports/90150 [20], ports/90668 [21],
ports/91433 [23], ports/88754 [24]
Submitted by: mi [1], marcus [2], Lars Engels <lars.engels@0x20.net> [3],
pav [4, 16, 20, 24], garga [5], cperciva [6], vd [7],
edwin [8, 9, 11, 15, 21],
fenner [10], Arseny Nasokin <tarc.po.cs.msu.su@tarc.po.cs.msu.su> [12],
Carl Makin <carl@stagecraft.cx> [13], arved [14],
NIIMI Satoshi <sa2c@sa2c.net> [17], thierry [18],
jylefort [19], linimon [22], dougb [23]
for INDEX builds [1]
* Remove the parallel target from Makefile; this is heavily tied to
the package build cluster and can be better done in the makeparallel
script (commit to follow) [2]
* Extend the format of INDEX to separately list the
EXTRACT/PATCH/FETCH_DEPENDS instead of lumping them all in together
with BUILD_DEPENDS. The three new fields are appended to the end of
the record in that order. [2]
* Change BROKEN to IGNORE in BROKEN_WITH_MYSQL failure code [3]
* Support non-default PREFIX for perl 5.00503 [5]
* Use pkg_info -I instead of ls when searching for conflicts [6]
* Allow local customization of the port subdirectories by including
${.CURDIR}/Makefile.local in bsd.subdir.mk if it exists [7]
* Fix 'make search' when ${PORTSDIR} is a symlink to a directory name
containing extended regexp metacharacters [8]
Submitted by: linimon [1] [3], kris [2], lth [4], sem [5], eik [5] [6],
Roman Neuhauser <neuhauser@chello.cz> [7]
PR: 68299 [1], 67705 [3], 67264 [4], 59696 [5], 66568 [6],
68072 [7]
* Don't assume root is using /bin/sh when switching credentials to
configure OPTIONS. [2]
* Support glob expressions in USE_GETTEXT to allow more flexibility
in the face of future gratuitous library version bumps by the gettext
developers [3]:
USE_GETTEXT=yEs # Works as before (case-insensitive)
USE_GETTEXT=[5-7] # Accepts any of those libintl.so.x versions
# in the LIB_DEPENDS
* Correctly register dependencies when a non-system perl port is used
on 4.x [4]
* Extend 'make search' support to allow much more flexible searching.
Syntax will be documented in CHANGES for brevity. [5]
* Reorder the post-install-script target to before add-plist-info for
consistency [6]
* Various fixes to support port operations when a port directory
exists under /usr/obj [7]
* Extend USE_PERL5_BUILD and USE_PERL5 to add EXTRACT and PATCH
dependencies since many ports require perl in those stages [8]
* Move info file deregistration later in the deinstallation process so
it works properly. [9]
* Improve wording in EXPIRATION_DATE message. [10]
* Fix dependencies for XFREE86_VERSION==3 (obtain imake from
x11/XFree86 now that the former port is gone) [11]
* While building index, treat non-existent dependencies as fatal.
Previously the error was being hidden by the stderr redirection. [12]
* Don't always retry BROKEN ports when package building (it is taking
too much time to continually rebuild ports that are usually going to
really be broken). Set TRYBROKEN if you want to attempt a build of
a BROKEN port. [12]
* Revert incorrect change from 1.487 relating to ALL-DEPENDS-LIST [13]
PR: 24214 [1], 67529 [2], 63937 [3], 65554 [4], 40699 [5],
59162 [6], 63372 66567 [7], 63394 [8], 65304 [9],
65931 [10], 66565 [11], 66743 [13]
Submitted by: roam [1], will [1], hrs [2], mi [3], ade [4],
Roman Neuhauser <roman@bellavista.cz> [5],
Sergey Matveychuk <sem@ciam.ru> [6], gad [7],
adamw [8], kris [8][12], dinoex [9],
Alexey Dokuchaev <danfe@regency.nsu.ru> [10],
eik [11][13]
INDEX-5).
A note about the implementation: I had to add the call to chmod to make
the index file readable to everyone. I'm a little bit surprised by this,
since my umask is 022. Either fetch(1) doesn't respect the umask or it
mirrors the permissions too (I haven't investigated this). Both
possibilities aren't documented in the man-page.
Approved by: kris (previous version without the chmod)
INFO_PATH variable may be used to specify their location. It defaults
to 'share/info' for the standard PREFIX, and 'info' when PREFIX ==
/usr. [1]
- Remove the <category>/pkg/COMMENT files in favour of a COMMENT
variable in <category>/Makefile [2]
- Prevent patch breakage with VERSION_CONTROL=numbered [3]
- Fix some instances of incorrect WRKDIRPREFIX handling. [4]
- remove useless ${MKDIR} ${WRKSRC} in config target [5]
- remove reference to OpenBSD [6]
- Exempt devel/p5-Module-Build from the self-dependency in
PERL_MODBUILD so that this port may use the option without getting an
infinite dependency list [7]
- The default PERL_ARCH is currently determined as a function of
OSVERSION. It should however be a function of PERL_LEVEL since the
correct value depends on what Perl version one has installed (older
Perl versions use ${ARCH}-freebsd, newer versions use mach). [8]
- Fix PORTDOCS on older (4.7, 5.0) systems [9]
- Allow 'make parallel' to generate a working makefile when not all
categories are present (this does not mean you'll be able to build all
ports, unless you make sure they don't have external dependencies) [10]
- Don't report symlinks as world-writable in the security check [11]
- Fix a comment that was broken by a mismerged patch [12]
- Clarify the meaning of USE_*, WANT_*, WITH_* and WITHOUT_* [13]
- Don't set _CHKSUMFILES/_IGNOREFILES if CKSUMFILES/IGNOREFILES is
empty and DIST_SUBDIR is set. [14]
- Fix comment for DISTDIR [15]
- Update the documentation of the USE_GL variable [16]
- Check to see if NONEXISTENT exists, and fail with an error if it does [17]
- Fix fetching of new distfiles in 'make makesum' when SIZE is set [18]
- Consistently set MAKE_ENV when USE_GCC=3.2 or 3.3 are set [19]
- Rework INDEX builds: [20]
* Fix the bsd.port.subdir.mk code that is supposed to report index
breakage (the fallback code wasn't actually being run because make
would halt immediately following the error). This should help with
INDEX error reports because it will immediately show the cause of
failure, so we won't have to pull teeth to extract it from the
submitter.
* Streamline the 'make describe' code a bit.
* Provide some basic instructions to the user when an index build
fails, on when and how to report index build failures (turn this off
with INDEX_QUIET=1)
* Removed INDEX_NOSORT, because I couldn't imagine it to be very
useful and it doesn't cost very much anyway.
* Don't prevent INDEX builds from seeing the local host environment.
Since a lot of users are using 'make index' thesedays they should get
an index that reflects their local settings and installed ports. If
you want to build a 'default' index that isn't influenced by local
settings (e.g. for release builds), set the INDEX_PRISTINE variable.
* Allows parallel INDEX builds (using make -j). The most obvious way
of doing this doesn't work, because I/O from child makes is broken up
into 2k chunks, and output lines from 'make describe' that exceed this
length (*cough* GNOME *cough*) will be intertwined with the output of
other makes, leading to a corrupted INDEX. The I/O interleaving
can be disabled using 'make -P', but this inserts extraneous output of
its own, and redirects stderr, making it useless for our purposes.
Instead, I collect the output from the child make processes in
temporary files and recombine them at the end.
* The number of concurrent make processes to spawn can be set using
INDEX_JOBS. By default this is set to 2, which seems to be a sweet
spot for both single and dual-processor systems. On my tests I do not
see any significant performance changes on UP, but on a dual 4.x
system the build time drops by 47% (6 minute index builds on one test
machine!). Depending on your disk and CPU hardware you might see
further gains with INDEX_JOBS=4 or higher, so you might like to
experiment to see what works best. On a dual 5.x system the
performance gains do not seem to be as great (20-30%), but this is
still a significant net win.
PR: 55493 [1], 59651 [2], 61552 [3], 62247 [4], 62329 [5],
62337 [6], 62422 [7], 62441 [8], 62627 [9], 62983 [10],
63112 [11], 63297 [12], 63335 [13], 64029 [14], 64069 [15],
64236 [16], 64519 [17], 62958 [18], 64237 [19]
Submitted by: lev [1],
Matthew Seaman <m.seaman@infracaninophile.co.uk> [2],
Joel Ray Holveck <joelh@piquan.org> [3],
ade [4], Sergey Matveychuk <sem@ciam.ru> [5],
markus [6], mat [7], des [8], eik [9],
Dmitry Morozovsky <marck@rinet.ru> [10],
Andrew <andrew@ugh.net.au> [11], vs [12], linimon [13],
edwin [14][15], gerald [16], marcus[17][18], kris [19][20]
