invocations). It also fixes some edge cases that were not handled in
the previous version.
TODO: Correctly report IPv6 sockets (already in use by the sparc64 build)
ordering, which had become too limited.
We now build packages ordered by those that are part of the longest
dependency chains first. This has the effect of building the deepest
parts of the tree first and levelling out the tree height, hopefully
avoiding the situation we currently face where there appear
bottlenecks late in the build where the cluster becomes mostly idle
while waiting for a few long dependency chains to finish building
before the cluster can become fully loaded again.
The algorithm is that we sort the list of remaining packages according
to height (longest dependency chain), then add leaf packages from each
in order until we have filled a queue of length between 100 and 200,
to amortise the cost of this queue rebalancing while not losing the
height averaging property. Jobs are dispatched from this queue into
worker threads as machine slots become available.
Unlike the make-based solution that required a fixed -j concurrency
value and could not respond to addition/removal of build resources, we
now can dynamically add new machines as they become available to the
queue.
The other advantage of using python is that we have more
customisability and visibility into the build status, e.g. we
periodically report the number of remaining packages, as well as the
list of deepest packages that we are working on.
TODO:
* Implement mtime checking for parent package staleness, so that
parents are rebuilt if the dependencies are touched more recently.
Currently packages will not be rebuild if they exist, whether or not
they are "stale" wrt their dependencies.
* Offload the machine selection into an external queue manager.
Currently the queue manager used here doesn't interoperate with the
old one (getmachine/releasemachine) because it's not possible to use
the lockf()-based mutual exclusion within a multithreaded client.
Doing that will also allow for a more flexible job placement
algorithm as well as finer queue customization.
just the plist ones. If the log is less than 1000 lines after the header,
include it all; else, trim to last 1000 lines.
This should help when deciding where to forward logs.
Tested on: pointyhat
makes it possible to correctly analyze why packages were not built for a
specific run.
Add a beginning and ending email notification to help coordinate between
multiple portmgrs doing runs.
zfs:
* Enabled by use_zfs=1 in portbuild.conf
* Populate build chroots by cloning a zfs snapshot instead of maintaining
many duplicate copies. In principle this is very efficient since
everything is copy-on-write and zfs snapshot creation is almost
instantaneous. There might be additional overheads from building on zfs
though. Currently the snapshot base is hard-wired to y/${branch}@base
but should be parametrized. This also must be populated beforehand, e.g.
during machine startup
* Clean build chroots by just destroying the snapshot.
tmpfs:
* Enabled by use_tmpfs=1 and tmpfs_size in portbuild.conf
* The previous md strategy of mounting in used/, populating and then
remounting (to avoid possible races from multiple builds claiming the
same chroot) doesn't work here because tmpfs instances are destroyed at
umount. I am not entirely sure the simpler approach is free from races.
order to run certain host binaries that were kernel-dependent. We
now seem to be able to rely on the /rescue versions (and killall(1)
seems to be unused).
* Allow for ccache directories to be shared over NFS via the ccache_dir_nfs
portbuild.conf boolean
* Populate BSD.local.dist from ${PORTSDIR}/Templates and remove population
of BSD.x11-4.dist and support for XFree86 3.x
machine with the lowest number of running jobs. This worked when the
clients were all roughly equivalent, but schedules poorly when there
are some that are much more powerful (e.g. 8-core machines vs UP machines)
* We now compute the ratio of running jobs to maximum jobs and schedule on
the machine with lowest occupation fraction. This populates the machines
to equal fractions of their capacity.
* Only hardlink the old log files instead of anything else that might be
in the directories
* Add comment that old logfiles should be removed as well as packages, to
avoid duplicate versions of the same port log
do it in portbuild from outside the jail thesedays
* Ignore /var/db/fontconfig which does not get restored to pristine state
* Save copies of master.passwd and groups and check them after the build
for changes, to look for user/group additions that may not be correctly
registered in UIDs/GIDs. Future work will hopefully automatically
check against those files and make unregistered IDs a fatal condition
* Correct logic mistake that was keeping distfiles for collection when
the checksum mismatched
