Kris Kennaway 778518d745 NNew build scheduler written in python to replace the make+sh job ... ordering, which had become too limited. We now build packages ordered by those that are part of the longest dependency chains first. This has the effect of building the deepest parts of the tree first and levelling out the tree height, hopefully avoiding the situation we currently face where there appear bottlenecks late in the build where the cluster becomes mostly idle while waiting for a few long dependency chains to finish building before the cluster can become fully loaded again. The algorithm is that we sort the list of remaining packages according to height (longest dependency chain), then add leaf packages from each in order until we have filled a queue of length between 100 and 200, to amortise the cost of this queue rebalancing while not losing the height averaging property. Jobs are dispatched from this queue into worker threads as machine slots become available. Unlike the make-based solution that required a fixed -j concurrency value and could not respond to addition/removal of build resources, we now can dynamically add new machines as they become available to the queue. The other advantage of using python is that we have more customisability and visibility into the build status, e.g. we periodically report the number of remaining packages, as well as the list of deepest packages that we are working on. TODO: * Implement mtime checking for parent package staleness, so that parents are rebuilt if the dependencies are touched more recently. Currently packages will not be rebuild if they exist, whether or not they are "stale" wrt their dependencies. * Offload the machine selection into an external queue manager. Currently the queue manager used here doesn't interoperate with the old one (getmachine/releasemachine) because it's not possible to use the lockf()-based mutual exclusion within a multithreaded client. Doing that will also allow for a more flexible job placement algorithm as well as finer queue customization.		2008-05-10 13:22:51 +00:00
..
portbuild	NNew build scheduler written in python to replace the make+sh job	2008-05-10 13:22:51 +00:00
scripts	Add a tiny sh hack - ardiff - compares two archives.	2008-05-06 16:17:01 +00:00
make_index
make_readmes