for INDEX builds [1]
* Remove the parallel target from Makefile; this is heavily tied to
the package build cluster and can be better done in the makeparallel
script (commit to follow) [2]
* Extend the format of INDEX to separately list the
EXTRACT/PATCH/FETCH_DEPENDS instead of lumping them all in together
with BUILD_DEPENDS. The three new fields are appended to the end of
the record in that order. [2]
* Change BROKEN to IGNORE in BROKEN_WITH_MYSQL failure code [3]
* Support non-default PREFIX for perl 5.00503 [5]
* Use pkg_info -I instead of ls when searching for conflicts [6]
* Allow local customization of the port subdirectories by including
${.CURDIR}/Makefile.local in bsd.subdir.mk if it exists [7]
* Fix 'make search' when ${PORTSDIR} is a symlink to a directory name
containing extended regexp metacharacters [8]
Submitted by: linimon [1] [3], kris [2], lth [4], sem [5], eik [5] [6],
Roman Neuhauser <neuhauser@chello.cz> [7]
PR: 68299 [1], 67705 [3], 67264 [4], 59696 [5], 66568 [6],
68072 [7]
build locking, log files, and cleans things up if a build fails.
This script is the primary starting point for a package build. Symlinks
should be created in the form of dopackages.${branch} -> dopackages.wrapper
where ${branch} is currently one of 4, 4-exp, or 5. This script takes the
place of the unofficial (i.e. uncommitted) dopackages.steveX scripts.
Ok'd by: kris
Tested by: 4.10-RELEASE package build
- CC committers and maintainer [1]
- include affected ports in the subject line [2]
- do a CVS log of the version checked out [3]
Suggsted by: Ade Lovett <ade@FreeBSD.org> [1]
Bjoern A. Zeeb <bzeeb-lists@lists.zabbadoz.net> [2]
Pav Lucistnik <pav@FreeBSD.org> [3]
You can even get notified of version changes in your favourite
perl modules by setting
WATCH_REGEX='p5-.*'
Plus, it has a nice configurable nagging option.
used in 20 minutes, as well as directories listed as 'in use' that have not been touched
in 24 hours (corresponding to port builds that have timed out or shut down uncleanly)
and prunes them to reclaim space. This is intended to be run as a cron job.
- Does not execute unknown commands nor call sh(1) at all.
- Does not overwrite existing files by default.
- Does not extract files into upper directories.
- Does have a dry run (-n) flag to see what would have been extracted.
- Does have a strip (-p N) flag to strip any number of levels from
pathnames.
It (so far) only supports shell archives made with BSD shar.
more accurate:
* The tests for "Cannot stat", "can't cd", and "tar command failed" are moved
to the bottom. This will sacrifice a little bit of performance for a gain
in accuracy
* The tests for "configuration errors" needed to be combined and tweaked to
better identify errors that are architecture-specific (usuallly the
"all pointers are ints" assumption)
* The detection of chown was buggy
* There were some duplicate test cases
* As an experiment, another test for dependency problems is added
PR: 61914
Submitted by: linimon
encountered again, nad having it in there slows performance. However,
I'm leaving it in the file in case a bad bindist slips back in, and it
becomes convenient to uncomment this for purposes of seeing what build
failures are cluster-related.
Requested by: kris
processonelog script, that contains all the per-errorlog logic [1]
* Add new error checks, one for detecting CPU problems, and the other for
checking for broken libgnugetopt support [2]
* Add a new "munmap" error type to check for broken bindists
* Add imake and pthread detection to processonelog [3]
* Add errortype column to the bento "New Build Failures" report [4]
PR: 50258 56859 [1]
54406 [2]
57067 [3]
59272 [4]
Submitted by: linimon [1] [2] [3] [4]
also print the exact location of the file to checkout.
This is for those who rarely adds new ports and do not remember the
location of Additional FreeBSD Contributors article.
Approved by: will
Perl before, sometimes we would word wrap too soon. Now we should never
generate a line more than 72 characters long (but as close to 72 characters
long as we can without breaking up a word).
Approved by: will (maintainer)
of master sites.
Before that %SUBDIR% was not expanded and the resulting URL was malformed.
L-o-o-o-ng maintainer timeout waiting for review of a slightly different
solution.
is full
* The newfailure file is no longer maintained, but generated as-needed
by the processfail script
* Update format of failure file (records both initial failure date and
most recent failure date)
of stale entries (removed ports, ports listed in duds that were mistakenly
built, malformed entries) and refreshes the version of entries to the
latest in INDEX. This must be run under lockf (see the comments in the
script) to avoid racing with portbuild which also tries to write to
these files.
processing (e.g. transfer to ftp site). It was taking up too much disk
space to collect distfiles for 14 architecture/version combinations
(including the backup set from the previous run).
Put the cvsdone files in the arch/branch subdirectory, not arch/.
of having the clients scp their loads to bento every 10 seconds. Fix
some indentation and add some sleeps to make sure the startup script
doesn't run too early in the boot process.
Add some new command-line options:
* -continue : restart the build without rebuilding ports that have
already failed
* -nofinish : don't run the post-processing steps when the build finishes
* -finish : run the post-processing steps only
Reorder some code sections appropriately
and removed files.
This can be used as the basis of a pkg-plist, or even just for
curiosity about what files something is touching.
Fairly raw at the moment, and doubtless inefficient, but it should
make a useful tool for port creators.
PR: ports/47424
Submitter: Daniel O'Connor <doconnor@gsoft.com.au>
- Set the UNAME_* variables to make use of the override support in uname(1)
This allows us to get rid of the uname shell script in the build chroots
- Collect some variables that are probably old and mouldy, for later GCing
- If FETCH_ORIGINAL is set in the environment, don't fetch from ftp-master
hosts (non-freefall)" bug: add '-A' argument to ssh(1) command line
for accessing host cointaining modules file. This should fix the
bug for ssh-agent(1) users. Others can just switch to using
ssh-agent(1).
Approved by: will (maintainer)
in portbuild.conf). [1]
* Build stage 2 builds in a jail instead of a chroot. This allows us to
detect ports that attempt IP communication during the build, as well as
allowing clean termination of the port build (packages built in a chroot
can leave processes hanging around after the build finishes). There
are some caveats with this approach which will be worked around in a
future commit.
Requested by: peter [1]
are now collected by polling a small server on the client (using netcat)
instead of having the client scp the load files to the master every 10
seconds (!!!)
- Call processfail to produce 'new port failures' page
- Comment out the 'comparelogs' calls for now until I make them aware
of architectures that don't have 4.x support
and 'buildfailure' and produce a HTML output listing ports with the date
they became broken. The output can surely be made more useful (e.g. it
always links to the ${branch}-latest logs, which may not exist when the
build-in-progress has not yet attempted the build of that port).
keep track of the number of times a port has failed to build, resetting
the counter when it builds successfully. There are still some bugs to be
worked out, but this will be used as the basis for maintainer notification
of port failures.
- Increase timeout to 2 hours (needs to be arch-specific)
- Mount nfs filesystems with nfsv3,intr (the latter so that clients do
not hang if bento panics)
- Run pnohang.${arch} on clients instead of pnohang
- Support builds as non-privileged user
- Increase timeout to 8 hours (this needs to be made per-arch so it
doesn't overly pessimize fast client machines)
- Support building as a non-privileged user
- Use $INDEXFILE
- Use makeparallel script instead of 'make parallel'
- Support building packages as a non-privileged user (needs root access
to client machines)
- Switch to using ${arch}/${branch} subdirectory instead of ${branch}
- export INDEXFILE, MACHINE_ARCH and ARCH variables
- Add (commented out) FETCH_BEFORE_ARGS setting
- Add MASTER_SITE_OVERRIDE to fetch from ftp://bento
- Zap some mouldy old unused variables
from recursive dependency builds a la:
make DEPENDS_TARGET='install package clean' all install package clean
The pkg-list script obtains a list of the packages in the dependency
directories; the pkg-stash script moves them away to a predefined
directory, adding a timestamp to the package file name. This is
convenient for keeping ready-built packages for system rescue
activities.
This fixes addport for me on -stable (cvs was erroring out), and also
fixes addport when you have the EDITOR environment variable set.
Approved by: will
body tag nor a closing html tag. In general, browsers work without
these, but Konqueror in particular was displaying some artifacts.
Also make sure empty cells have an nbsp directive; eliminate a bogus
case for Aff.; and change Date build to Build date.
Submitted by: Mark Linimon <linimon@lonesome.com>
PR: www/45021
* Update list of quickports to include kde3
* Add ability to restart builds (-restart)
* Change default to build packages for non-cdrom use (associated option name
inverted from -nocdrom to -cdrom)
* Cope with .tgz or .tbz packages
* Use buildenv framework
* Use cvs update -PdA instead of -Pd when updating ports/docs tree
* Always delete restricted packages after a build run, since they are
published on the website.
* Minor cosmetic changes
* generate an interim .logs file containing the data to be presented in
the html files (also useful for other consumers, like fenner's
per-maintainer error logs)
* Generate 5 different summaries of the error data, sorted by port name,
maintainer, category, build error, builddate
* Improve the table html code so it renders faster
* Add a few new failure reasons
Submitted by: edwin (based on)
PR: ports/43927
uname script. Mount the docs from the correct location. Remove the
hack to pkg_add perl into the chroot environment, now that the scripts
no longer rely on it.
actually simplifies some of the code and makes other parts more complicated.
Tested by: www/mod_filter && (after bugfixes) sysutils/sjog
Noticed by: ijliao (by way of peter)
table row, instead of making a new row for every single file. This
cuts down the generated page size drastically, improving rendering times.
Submitted by: Edwin Groothuis <edwin@mavetju.org>
PR: ports/40962
is it makes patch names longer, and many shells autoexpand it to "\:\:"
which makes them even longer.
* Note that this file uses ts=4.
Approved by: sobomax
Some work/cleanup here would probably be desirable. I'm committing them
now to give others the ability to reproduce these package splits and help
with making these scripts better. The README should have some more
documentation in the future.
A better solution to the package set size problem would be to teach
sysinstall to ask for the CD on which a particular package is when it
needs to add it, but for now this will do.
Requested by: murray
Written by: steve
Discussed with: steve, kris (some time ago)
* Add a trap handler to try and clean up the build if it is interrupted
by a signal (one problem with the previous version is that package builds
whice are interrupted by ptimeout because they are stuck, leave their
working files lying around in the chroot).
* Switch to NFS v3 mounts instead of v2
* Autogenerate the version string to report in uname within the chroot,
based on the version string in the head of the CVS branch being built.
* Copy packages via cp from the NFS mount, not scp.
* Require an additional <tmpdir> argument so the client knows where its
temp directory is.
* Mount the portbuild directory readonly via NFS, and copy files that
way instead of via scp, which has too much overhead
* Don't assume the script will be called from the ports directory
* Use buildenv to set environment variables
* Set LOCALBASE and X11BASE to dummy variables to prevent the ports tree
from picking up packages installed on the host system, and patch up
the generated index at the end
operations in one central place, instead of doing them piecemeal all over
the place. This also includes the ability to customize settings per
port branch (e.g. XFREE86_VERSION)
Also, instead of hardcoding values of OSVERSION and OSREL which are going
to get forgotten again, pull them out of the source tree in ${branch}/src
so they track the head of the branch.
tangled up.
* Sort the environment variables so that all of the commented out ones
are grouped together
* Reverse sense of NOPLISTCHECK -> PLISTCHECK, since it's not an option
we want enabled by default (it causes too many build failures). This
was too easy to forget when building packages 'by hand' using the parallel
makefile.
* Copy across UNAME_VERSION into the chroot area so that uname inside the
chroot reports the desired version
* Shorten timeout period from 12 hours to 4 hours to avoid delaying the builds
unnecessarily.
* Reverse sense of NOPLISTCHECK -> PLISTCHECK, since it's not an option
we want enabled by default (it causes too many build failures). This
was too easy to forget when building packages 'by hand' using the parallel
makefile.
we want enabled by default (it causes too many build failures). This
was too easy to forget when building packages 'by hand' using the parallel
makefile.
* Display correct pathnames in build progress messages
* Add a -nocdrom option to avoid the (time-consuming) clean-for-cdrom-list
target
* Remove x11/XFree86 from dummyports since we don't want to use an old
stale version of the package which never gets rebuilt
* Add comments noting that we should check for CVS conflicts in the cvs
update scripts and exit gracefully.
we want enabled by default (it causes too many build failures). This
was too easy to forget when building packages 'by hand' using the parallel
makefile.
* Don't require the script to be called from inside the branch directory;
follow standard practise and pass the branch to build as an argument.
* Populate the chroot with the BSD.local.dist mtree file
* Add whitespace for readability, and sprinkle liberally with comments
* Comment on some inscrutable parts of the script which were presumably
put there to work around a (perceived) problem, but which I can't
understand.
of just proclaiming "error(s) occurred". This should immediately identify
which port is causing the build to break (although it's a bit of a verbose
solution).
* Update the list of ports which takes a long time to build.
* Don't use a single ports collection for every port build, indirect through
${pb}/${branch}/ports so we can have different ports collections for
each branch. Add -noportscvs option to prevent cvs update of ports
collection.
* Remove 3.x as a valid branch, add 4-exp for the 4.x experimental builds
used for testing bsd.port.mk patches
* Update usage information
decide machines which should be handed new jobs. Also, at some point someone
added a note about a possible bug in the reportload script which causes
this one to fall over.
down to user support flaws in the FreeBSD ports system. The flaw in question
is related to the fact that dependencies are often "chained", which allows to
simplify maintenance of ports with large number of implied dependencies (a la
Evolution, Nautilus, you-name-it). Dependency chaining it's not a problem by
itself, but the fact that when building or installing a port the system doesn't
check chain integrity - it's only checks that dependencies explicitly
specified in port's Makefile are satisfied, which opens wide window for
various hard-trackable problems when one or more links in the middle of the
chain missed.
The idea behind the tool is quite simple - it should be executed right after
main dependency checking procedure, two times for each build - check build-time
chain before building the port (pre-pre-extract) and check run-time chain
before installing the port (pre-pre-install). When executed, the tool checks
integrity of the specified chain (build-time, run-time or both) and reports all
errors, both fatal (dependency isn't installed) and non-fatal (dependency is
installed, but different version).
I've wrote this tool mostly to simplify maintenance of the GNOME ports, but
it doesn't contain anything GNOME-specific, so that it could be used in the
other parts of tree as well.
As an example I've added GNOME_VALIDATE_DEPS_CHAIN knob into bsd.gnome.mk (off
by default), which enables automatic chain validation for all ports that
USE_GNOMELIBS. This is a bit hackish, because I've used pre-extract and
pre-install targets - what we probably need is a generic way to plug various
custom tasks specified in bsd.xxx.mk (where xxx is kde, gnome, python, etc.)
into various parts of the build process (something like {pre,post}-pre-foo,
{pre,post}-post-foo springs into my mind).
The code is quite raw, so that I would appreciate any bug reports, patches,
suggestions, constructive critiquie and so on.
file. For example I'm usually keeping all working directories in /tmp using
WRKDIRPREFIX, while for the quick access to a port's files creating a symlink
to this directory in skeleton's dir (i.e. ports/foo/bar/src -->
/tmp/usr/ports/foo/bar/work/bar-0.0) and with this patch the tool correctly
works when I'm specifying `src/foo.c' as an argument.
Add mkppackage, which will create a "fake"-style package, but not
really. It doesn't try fake PREFIX and such to the build (it's not
really possible anyway yet), but it uses pkg_create's '-s' option to
pretend the package generated was installed in ${PREFIX}.
a work/ dir in the repository placed there by addport without permission.
Also restore the rcsids from the predecessors of this script by removing
their $'s.
If I had more time, I'd just revamp parts of this script.
Submitted by: greid
Apologies to: greid && cvs@
tool has something in common with update-patches shell script, but has more
features and is more intelligent. See README.patchtool or source code for
details.
