This would relax the load of www.freebsd.org and let user fetch INDEX
from a closer mirror.
Approved by: bapt (portmgr)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33871
Add INDEX_COMPRESSION_FORMAT as a variable a user can overwrite via
/etc/make.conf to allow the compression format used for INDEX to be
changed. Available compression formats are: bz2, xz and zst for now.
While here change the default compression format to be .xz (the
smallest), for environement where performance of unxz matters it can
switched to zst.
bz2 format is kept for compatibility with existing tooling.
In order to simplify the code, bsdcat(1) is now used for uncompressing
instead detecting the compression format manually and switch the
decompression tool manually
Note the bzip2 compression format is now considered as deprecated
When extracting the fetched index, do it in a temporary file, and only
when extraction succeed, move it to the final file, it makes it more
atomic and less likely to live a corrupted file behind
PR: 224871
When generating the INDEX file, we are already building it in a
temporary file and swap it last minute. make the index target now
directly depend on the INDEX file generation target.
Turn that target into a PHONY target so the file is always regenerated
when someone calls make index.
PR: 202070
Don't assume svn is installed, but use it if it's available, falling back to
svnlite if not (or using svn from base in the case where WITH_SVN was set in
base). This matches the logic in base for finding svn.
PR: 207479
Discussed with: gjb
Reported by: Torsten Zühlsdorff <ports@toco-domains.de>
INDEX_JOBS currently defaults to 2 for the make index target.
MAKE_JOBS_NUMBER in bsd.port.mk currently uses kern.smp.cpus to select
a default. Modify INDEX_JOBS to match.
Nominal and repeatable performance improvements of ~20-30% were observed.
CR: D381
Reviewed by: swills, bdrewery
Approved by: portmgr
if available, for 'make index'. This removes the need to have
perl installed. [1]
- Fix output of the file to have the proper /usr/ports/ORIGIN regardless
of real PORTSDIR or CURDIR [2]
PR: ports/183095 [1]
Submitted by: ak [2]
With hat: portmgr
- bsd.port.mk: add INDEX_PORTS, to support INDEX creation for a subset of the ports tree [1]
- bsd.port.mk: call target "install-rc-script" before "post-install" [2]
- [patch] ports/Mk bsd.port.mk order if groups/users are created by package [3]
- [bsd.port.mk] [patch] reaper of the dead: md5 has been in /sbin for a while [4]
- [bsd.port.mk] [patch] remove support for pre 7.x systems (b.*.m) [5]
- [patch] [bsd.port.mk] reaper of the dead: are three variable defintions needed [6]
PR: ports/156575 [1],
ports/139116 [2],
ports/152498 [3],
ports/155983 [4],
ports/155510 [5],
ports/156340 [6]
Submitted by: Florent Thoumie <flz@xbsd.org> [1],
Sergey Skvortsov <skv@freebsd.org> [2],
Olli Hauer <ohauer@FreeBSD.org> [3],
Eitan Adler <lists@eitanadler.com> [4],
Eitan Adler <lists@eitanadler.com> [5],
Eitan Adler <lists@eitanadler.com> [6]
targets.
* Use /rescue/sh for index builds instead of /bin/sh, when it exists.
The former is statically linked and faster to execute, which becomes
significant when executing it tens of thousands of times. This
trick can be used with other recursive targets by passing in
__MAKE_SHELL.
* Get rid of make variable assignments that use != command invocations
in the critical path, using several methods:
- rewriting logic to use shell or make builtins instead of external command executions
- macroizing commands and executing them in the targets where they
are needed instead of with every invocation of make
- precomputing the results of invariant commands in
bsd.port.subdir.mk and passing them in explicitly to child makes,
and using this to avoid recalculation in all the children. NB: the
commands are still run one per top-level subdirectory but this
does not currently seem to be a major issue. They could be moved
further up into the top-level Makefile at the cost of some
cleanliness.
- Committers are strongly discouraged from adding further "bare" !=
assignments to the ports tree, even in their own ports. One of
the above strategies should be used to avoid future bloat.
* Rewrite the core 'describe' target to work entirely within a single
shell process using only builtin commands. The old version is
retained as a backup for use on systems older than 603104, which
does not have the make :u modifier. This cuts down the number of
processes executed during the course of a 'make index' by an order
of magnitude, and we are essentially now amortized to the minimum of
a single make + sh instance per port, plus whatever commands the
port makefile itself executes (which are usually unnecessary and
bogus).
* Less validation of the WWW: target is performed; this can become
policed at a port level by portlint. Specifically we look at the
second word of the first line beginning with "WWW:" in pkg-descr,
and append "http://" to it unless it already begins with "http://",
"https://" or "ftp://". Thanks to dougb for the idea of how to
extract WWW: using shell builtins.
* Use the "true" shell builtin instead of echo > /dev/null for a
measurable decrease in CPU use.
* Add a note about dubious escaping strategy in bsd.port.subdir.mk
* Minor change in output of 'make describe': it no longer strips
trailing CR characters from pkg-descr files with MSDOS CR/LF
termination. Instead the makeindex perl script that post-processes
make describe into the INDEX is tweaked to strip on input.
The bottom line is that on my test hardware INDEX builds are now
faster by more than a factor of 2 and with a reduction in system time
by a factor of 4-8 depending on configuration.
to use csup for both ports and src. It should be possible to use csup
for src and portsnap for ports.
If you define SUP_UPDATE without PORTSSUPFILE defined, you'll get
portsnap by default now.
PR: ports/113819
Submitted by: ale
Requested by: many
Hat: portmgr
you had to manually select one of 3 methods: SUP_UPDATE, CVS_UPDATE,
or PORTSNAP_UPDATE. The latter is now obsolete.
This should help some novice administrators just-start their system.
PR: ports/105835
Submitted by: erwin
[1] Remove DEPENDS
[2] Add a new pkg-plist keyword @stopdaemon
[3] Provide rc.subr script suffix as RC_SUBR_SUFFIX
[4] Add USE_CDRTOOLS to abstract dependency on sysutils/cdrtool[-cjk]
[5] Add DOS2UNIX_REGEX, allowing to pass a filename regex to CRLF replacement
routine
[6] Support partially translated manpages in MAN* variables
[7] Fix USE_LDCONFIG on ports with @cwd in pkg-plist
[8] Remove support for XFree86 3.X
[9] Add user settable WITH_DEBUG flag that turns on -g in CFLAGS and resets
STRIP
[9] Fix `make install' to refuse overwriting of older version of the port
[10] New category ports-mgmt
[11] New category gnustep
[12] Removed category picobsd
bsd.java.mk
[13] Remove deprecated syntax
bsd.tcl.mk
[14] Provide TCL_VER when USE_TK[_BUILD] is defined. Fixes cad/netgen
ports/Makefile
[15] Make csup default `make update' method on FreeBSD 6.2 and up
PR: ports/99742 [1], ports/93373 [2], ports/100915 [3],
ports/105161 [4], ports/106029 [5], ports/106252 [6],
ports/106235 [7], ports/100939 [9], ports/97507 [10],
ports/103931 [11], ports/106921 [12], ports/104136 [13],
ports/105215 [15]
Submitted by: sem [1] [2], Matthias Andree <matthias.andree@gmx.de> [3],
nork [4], pav [5] [7] [14], Nick Barkas <snb@threerings.net> [6],
flz [8], gabor [9], shaun [10], erwin [12], hq [13],
Gurkan Sengun <gurkan@linuks.mine.nu> [11]
symlink. [1]
- Allow use of alternate http download utilities with 'make
fetchindex'. [2]
PR: ports/35767 [1], ports/92588 [2]
Submitted by: Peter Jeremy <PeterJeremy at optushome dot com dot au> [1],
Alexander Logvinov <user at blg dot akavia dot ru> [2]
EMACS ports. [1]
- Allow building a port as root using an NFS-mounted /usr/ports if the
server maps root to a UID other than root. [2]
- Make 'BROKEN' and 'IGNORED' ports exit their "make install" with a fail
status rather than success. [3]
- Improve behavior when dealing with versioned dependencies. [4]
- Fix false positives in check-conflicts target. [5]
- Remove obsolete bzip2 code. [6]
- Add physical category net-p2p. [7]
- Don't fetch INDEXFILE if not necessary; respect FETCH_ENV. [8], [11]
- INDEX can now be moved outside of ports tree. [9]
- Add ghostscript-gpl. [10]
- Remove obsolete USE_MESA. [12]
- Force pkg_install tools from ports on FreeBSD 4.10 and older. [13]
- Document ALWAYS_KEEP_DISTFILES. [14]
- Remove USE_REINPLACE from bsd.port.mk USE_DOS2UNIX patch. [15]
PR: ports/37596 [1], ports/57259 [2], ports/63216 [3],
ports/89448 [4], ports/89710 [5], ports/88996 [6],
ports/89260 [7], ports/89363 [8], ports/89809 [9],
ports/89853 [10], ports/91086 [11], ports/91710 [12],
ports/91727 [13], ports/92111 [14], ports/92124 [15]
Submitted by: Jay Sachs <jay at eziba dot com> [1], sem [1, 3, 8, 12],
Andrew Heybey <ath at niksun dot com> [2], Jamie Jones
<jamie at thompson dot bishopston dot net>, tobez [4], Mark
Andrews <Mark_Andrews at isc dot org> [5], edwin [6, 11, 15],
pav [7, 13], Peter Jeremy <PeterJeremy at optushome dot com
dot au> [9], Ulrich Spoerlein <q at galgenberg dot net> [10],
netchild [11], erwin [14]
Reviewed by: kris, clement (partially)
very long and if some dependencies can't be added into the list.
[1]
- Fix FAM support. Make gamin the default FAM system. [2]
- Introduce new 'quicksearch' target to show only port, path and
info section of the matching ports. [3]
- Introduce new category - rubygems. [4]
- Fix stale dependencies while installing qmail slaveport and
another port that depends on qmail. [5]
- Add commentary for describes target in bsd.port.mk. [6]
- Fix warning issued during make index on archs !368. [7]
- Add USE_DOS2UNIX variable. If set to "YES", remove the ^M from
all files under ${WRKSRC}. If set to a string, remove in all files
under ${WRKSRC} with one of these names the ^Ms. [8]
- Add new variables PERL_RUN_DEPENDS and PERL_BUILD_DEPENDS by
checking the existance of the Perl modules with the "perl -e 'use
module;'" command. [9]
- Fix bsd.port.mk variable quoting issues. No quoting is necessary
anymore either in the Makefile or on the command line. Affected
variables include:
BROKEN
FORBIDDEN
IGNORE
MANUAL_PACKAGE_BUILD
NO_CDROM
NO_PACKAGE
RESTRICTED
[10]
- Add NOFETCHFILES variable. If set, don't download these files
from the ${MASTER_SITES} or ${MASTER_SITE_BACKUP} (but do from
${MASTER_SITE_OVERRIDE}). [11]
- Improve 'search' target output. [12]
- Add a new virtual category for Amateur Radio - hamradio. [13]
- Cleanup some old/unused pathes in bsd.port.mk. [14]
- Add @dirrmtry for plists which does the same as:
"@unexec rmdir %D/foo 2>/dev/null || true" [15]
- Remove virtual category - offix. [16]
- Use portsnap instead of cvsup or cvs on "make update" in
/usr/ports. [17]
- Move location of bsd.autotools.mk within bsd.port.mk [18]
- Add bsd.linux-rpm.mk, fix INSTALLS_SHLIB for Linux ports [19]
- Use new USE_RC_SUBR format for FreeBSD version >= 700007 [20]
- Replace the string "FreeBSD" by "The FreeBSD Project" in the
security warning [21]
- Add bsd.local.mk for local modification to ports framework. [22]
- Replace rcNG spelling by rc.d [23]
- Remove superfluous USE_REINPLACE. [24]
Special thanks to: linimon for spending hours with all these patches
clement for fixes
kris for help with pointyhat
PR: ports/86310 [1], ports/89498 [2], ports/83530 [3],
ports/83789 [4], ports/84053 [5], ports/86281 [6],
ports/87214 [7], ports/87234 [8], ports/87318 [9],
ports/87396 [10], ports/87605 [11], ports/87840 [12],
ports/88230 [13], ports/88493 [14], ports/88711 [15],
ports/88751 [16], ports/89281 [17], ports/89999 [18],
ports/90031 [19], ports/90150 [20], ports/90668 [21],
ports/91433 [23], ports/88754 [24]
Submitted by: mi [1], marcus [2], Lars Engels <lars.engels@0x20.net> [3],
pav [4, 16, 20, 24], garga [5], cperciva [6], vd [7],
edwin [8, 9, 11, 15, 21],
fenner [10], Arseny Nasokin <tarc.po.cs.msu.su@tarc.po.cs.msu.su> [12],
Carl Makin <carl@stagecraft.cx> [13], arved [14],
NIIMI Satoshi <sa2c@sa2c.net> [17], thierry [18],
jylefort [19], linimon [22], dougb [23]
for INDEX builds [1]
* Remove the parallel target from Makefile; this is heavily tied to
the package build cluster and can be better done in the makeparallel
script (commit to follow) [2]
* Extend the format of INDEX to separately list the
EXTRACT/PATCH/FETCH_DEPENDS instead of lumping them all in together
with BUILD_DEPENDS. The three new fields are appended to the end of
the record in that order. [2]
* Change BROKEN to IGNORE in BROKEN_WITH_MYSQL failure code [3]
* Support non-default PREFIX for perl 5.00503 [5]
* Use pkg_info -I instead of ls when searching for conflicts [6]
* Allow local customization of the port subdirectories by including
${.CURDIR}/Makefile.local in bsd.subdir.mk if it exists [7]
* Fix 'make search' when ${PORTSDIR} is a symlink to a directory name
containing extended regexp metacharacters [8]
Submitted by: linimon [1] [3], kris [2], lth [4], sem [5], eik [5] [6],
Roman Neuhauser <neuhauser@chello.cz> [7]
PR: 68299 [1], 67705 [3], 67264 [4], 59696 [5], 66568 [6],
68072 [7]
