1
0
mirror of https://git.FreeBSD.org/ports.git synced 2024-12-05 01:55:52 +00:00
Commit Graph

768 Commits

Author SHA1 Message Date
Kris Kennaway
5953f77e24 Rewrite this to make it more efficient (fewer external command
invocations).  It also fixes some edge cases that were not handled in
the previous version.

TODO: Correctly report IPv6 sockets (already in use by the sparc64 build)
2008-06-02 19:46:03 +00:00
Erwin Lansing
cdb5013c48 Remove support for FreeBSD 5.x 2008-06-01 09:26:12 +00:00
Mark Linimon
11ea3eeed1 Remove the force file, if it was used.
Forgotten by:	linimon
2008-05-29 14:14:54 +00:00
Mark Linimon
ed5147b294 Add a force function here, just like processlogs. 2008-05-29 02:34:41 +00:00
Mark Linimon
4cc67bd216 Add a note that processonelog and processlogs2 are finicky about the
header format of the log files.
2008-05-29 01:46:08 +00:00
Mark Linimon
f5c9292932 Fix these after the 1.31 update to buildscript.
Forgotten by:	pav
2008-05-29 01:45:39 +00:00
Kris Kennaway
9100b4ee0d Rewrite this in python instead of shell. Because we can read the
INDEX once and process internally instead of invoking many external
utilities, runtime is improved from ~20 minutes to <10 seconds.
2008-05-25 18:07:49 +00:00
Kris Kennaway
778518d745 NNew build scheduler written in python to replace the make+sh job
ordering, which had become too limited.

We now build packages ordered by those that are part of the longest
dependency chains first.  This has the effect of building the deepest
parts of the tree first and levelling out the tree height, hopefully
avoiding the situation we currently face where there appear
bottlenecks late in the build where the cluster becomes mostly idle
while waiting for a few long dependency chains to finish building
before the cluster can become fully loaded again.

The algorithm is that we sort the list of remaining packages according
to height (longest dependency chain), then add leaf packages from each
in order until we have filled a queue of length between 100 and 200,
to amortise the cost of this queue rebalancing while not losing the
height averaging property.  Jobs are dispatched from this queue into
worker threads as machine slots become available.

Unlike the make-based solution that required a fixed -j concurrency
value and could not respond to addition/removal of build resources, we
now can dynamically add new machines as they become available to the
queue.

The other advantage of using python is that we have more
customisability and visibility into the build status, e.g. we
periodically report the number of remaining packages, as well as the
list of deepest packages that we are working on.

TODO:

* Implement mtime checking for parent package staleness, so that
  parents are rebuilt if the dependencies are touched more recently.
  Currently packages will not be rebuild if they exist, whether or not
  they are "stale" wrt their dependencies.

* Offload the machine selection into an external queue manager.
  Currently the queue manager used here doesn't interoperate with the
  old one (getmachine/releasemachine) because it's not possible to use
  the lockf()-based mutual exclusion within a multithreaded client.
  Doing that will also allow for a more flexible job placement
  algorithm as well as finer queue customization.
2008-05-10 13:22:51 +00:00
Andrew Pantyukhin
0b5ca793be Add a tiny sh hack - ardiff - compares two archives. 2008-05-06 16:17:01 +00:00
Pav Lucistnik
989ac675fc Remove XFree86-4 from quickports 2008-04-11 11:34:30 +00:00
Pav Lucistnik
b88fea571b Parallelize to 4 concurrent jobs 2008-04-11 11:33:38 +00:00
Pav Lucistnik
0c02d135ff Include per-machine configuration and respect use_zfs flag 2008-04-11 11:32:29 +00:00
Pav Lucistnik
2545b7c79e Sync with pointyhat (reorg) 2008-04-11 11:31:33 +00:00
Pav Lucistnik
7b00cf5640 - Add the port's Makefile ident string to the build log headers
PR:		113234
Submitted by:	Andrej Zverev <andrey.zverev@electro-com.ru>

- Sync with actual script on pointyhat (whitespace, mostly)
2008-03-13 09:54:03 +00:00
Pav Lucistnik
d50ed44a19 - Don't write queue entry for machine that's not on mlist. This allows us to
gracefully remove a node from the workload, and also it prevents build from
  stucking when machine is deleted from mlist.
2008-03-12 16:04:59 +00:00
Pav Lucistnik
f7c728bb1e Make the INDEX_JOBS definition actually effective 2008-03-11 17:05:55 +00:00
Erwin Lansing
170314e4e2 Remove support for 5-exp and add 7-exp, 8 and 8-exp 2008-01-14 17:10:33 +00:00
Erwin Lansing
d32d4753c3 Change homebrew locking routines with lockf(1) based locking. 2008-01-01 13:25:01 +00:00
Mark Linimon
2c493361b0 Add the standard 6-line header and URL to all build failure logs, not
just the plist ones.  If the log is less than 1000 lines after the header,
include it all; else, trim to last 1000 lines.

This should help when deciding where to forward logs.

Tested on:	pointyhat
2007-12-24 14:25:42 +00:00
Mark Linimon
1a3e27b0eb Copy the duds file to a directory where the web server can find it. This
makes it possible to correctly analyze why packages were not built for a
specific run.

Add a beginning and ending email notification to help coordinate between
multiple portmgrs doing runs.
2007-12-24 01:56:22 +00:00
Mark Linimon
ece1562584 Make the script less whiny for unpopulated builds. 2007-12-23 17:50:29 +00:00
Renato Botelho
e89241beff Fix a problem when add new ports on ports-mgmt category. ports-mgmt/Makefile
lines has 3 spaces before SUBDIR word and all other categories has 4.

I've asked pav@ if there is a default format of category Makefiles and he said
the number of spaces doesn't matter, so, i fix addport to respect the current
number of spaces and/or tabs the file has.

Reported by:	miwi, erwin
2007-12-19 16:54:37 +00:00
Kris Kennaway
6db6b14a8e Add support for 7-exp and 8 branches
Approved by:	portmgr (self)
2007-11-04 14:28:25 +00:00
Kris Kennaway
7822c39f44 Correctly report active builds when ZFS is in use
Approved by:	 portmgr (self)
2007-11-04 14:28:04 +00:00
Kris Kennaway
c94b9291a0 * Add support for 7-exp and 8 branches
* XORG_UPGRADE and USA_RESIDENT are no longer required

Approved by:   portmgr (self)
2007-11-04 14:27:19 +00:00
Erwin Lansing
0a56075360 - Add support for FreeBSD 8.0
- Be more robust about param.h and allow space as well as tab as seperator
  while determining OSVERION
2007-10-25 21:51:07 +00:00
Mark Linimon
46f9abbbb3 Add 7-exp, 8. 2007-10-16 00:53:33 +00:00
Mark Linimon
51dbc0f2a0 Add 7-exp and 8. 2007-10-16 00:51:01 +00:00
Mark Linimon
31a7bd9451 Rename 'missing' column to 'not yet built' so as not to confuse terminology
with portsmon reports.

Noticed by:	erwin
2007-10-16 00:37:03 +00:00
Renato Botelho
d54475429d s/query-pr.real/query-pr/ to fix getpr after freefall migration
Discussed at:	#bsdports
2007-10-15 19:37:31 +00:00
Mark Linimon
0cc5febd80 Reflect 8-CURRENT. 2007-10-15 00:40:38 +00:00
Mark Linimon
4451674575 The index page used on pointyhat. 2007-10-15 00:20:29 +00:00
Mark Linimon
d7decd3410 Update this example file to reflect 7.0 branching. 2007-10-14 18:21:34 +00:00
Gerald Pfeifer
957d2a35c5 Replace two further instances of /var/db/pkg by $PKG_DBDIR. 2007-09-16 22:24:12 +00:00
Mark Linimon
ba4a6e6a48 Remove the following false positives, which show up as compiler warnings,
not errors:

  discards qualifiers
  [0-9]: implicit declaration of function
  ISO C\+\+ forbids
2007-09-13 02:30:08 +00:00
Gerald Pfeifer
df787b975d Use PKG_DBDIR instead of hard-coding /var/db/pkg/. Do not silently revert
to defaults for PORTSDIR and PKG_DBDIR if the respective directory does
not exist, but bail out.

Approved by:	netchild
2007-09-08 16:23:13 +00:00
Mark Linimon
6d3018e6fc Change to the latest names of the ooo ports; add jdk16; sort a bit.
Glanced at by:	kris
2007-09-04 21:14:45 +00:00
Alexander Leidinger
0d7a3e7e9f explicit_lib_depends.sh:
- check if an installed libtool records dependencies recursively and
   print a warning if it does
       currently it prints the warning on every system which has libtool
       installed from ports (only my local version doesn't do this, the
       version in the ports is not correctly patched for this, a patch
       similar in complexity (= simple) like the ltdl.m4 one in the
       libtool-port-patch-directory is needed)
 - enhance the regex which is responsible to not print a dependency to
   the port we are just checking
 - add a work in progress (not executed) to collapse the USE_* which
   can have more than one value

neededlibs.sh:
 - we also care about shared libs

resolveportsfromlibs.sh:
 - take care about USE_OPENSSL, USE_EFL, USE_GL, USE_FAM, USE_OPENLDAP,
   USE_SDL
 - search in the "ldconfig -r" output if we can not find the lib ourself
 - a better way of getting the first part of the LIB_DEPENDS stuff
   (lib/libXYZ.so can be specified now too)
 - some line wrapping + whitespace
 - print the origin for the USE_* too (except USE_OPENSSL), so an user
   can make some sanity checks and the explicit_lib_depends.sh can DTRT
   if we check the USE_* port itself
 - warn if we can not determine the right component (can happen for XORG)
   unambiguously.
2007-09-04 19:32:36 +00:00
Alexander Leidinger
5c2c9f05ad One more sanity check, else /+CONTENS will be searched. 2007-08-30 16:48:33 +00:00
Alexander Leidinger
142947601f USE_FREETYPE is for v1, not for v2. Use a perfect match in the case statement. 2007-08-30 16:47:11 +00:00
Alexander Leidinger
bd34c2cc9c Do not list libs which come with the port itself. 2007-08-26 15:03:55 +00:00
Alexander Leidinger
1fc8d4fdd0 Add explicit_lib_depends.sh. 2007-08-26 15:02:24 +00:00
Alexander Leidinger
dd65a20d30 - Add USE_FREETYPE.
- Print the portsdir variable in front of the origin for easy copy und paste
  of the output.
2007-08-26 14:43:52 +00:00
Alexander Leidinger
63e512d248 We should also look into sbin/ (not only bin/ and lib/). 2007-08-24 15:30:31 +00:00
Alexander Leidinger
2733307251 explicit_lib_depends.sh shows you the current explicit dependency list of
libs for a given installed port, e.g.:
	# ./explicit_lib_depends.sh gnome-vfs-2.18.1_2
	USE_GETTEXT=yes
	USE_GNOME+=gconf2
	USE_GNOME+=glib20
	USE_GNOME+=gnomevfs2
	USE_GNOME+=libxml2
	USE_GNOME+=orbit2
	USE_ICONV=yes
	avahi-client:net/avahi
	avahi-common:net/avahi
	avahi-glib:net/avahi
	dbus-1:devel/dbus
	dbus-glib-1:devel/dbus-glib
2007-08-24 14:29:42 +00:00
Alexander Leidinger
011560cf37 Teach about USE_ICONV, USE_GETTEXT, USE_GNOME and USE_XORG. 2007-08-24 14:29:03 +00:00
Alexander Leidinger
43b63893be Add 2 scripts which help in port maintenance to list all direct library
dependencies of a port:
	neededlibs.sh
		Extract direct library dependencies (filenames) from binaries.
	resolveportsfromlibs.sh
		Prints the name(s) of ports(s) given a library filename,
		suitable for direct use (copy&paste) in LIB_DEPENDS.

Example usage is included in the scripts. The following combined usage may
be helpful for further porting/testing automation:
	resolveportsfromlibs.sh -b /usr/local $(neededlibs.sh /test/bin/*)

Requested by:	kris, lofi (sort of)
2007-08-12 12:49:23 +00:00
Gabor Kovesdan
4c9cd257fd - Extract command variable definitions from bsd.port.mk, they are now stored in
bsd.commands.mk and can be easily reused within the infrastructure.
- Revert old DESTDIR implementation.
- Add a new, fully chrooted DESTDIR implementation as bsd.destdir.mk.

Sponsored by:	Google Summer of Code 2007
Approved by:	portmgr (pav)
2007-08-04 11:37:24 +00:00
Kris Kennaway
687f043633 Support builds using zfs and tmpfs backing store
zfs:

* Enabled by use_zfs=1 in portbuild.conf

* Populate build chroots by cloning a zfs snapshot instead of maintaining
  many duplicate copies.  In principle this is very efficient since
  everything is copy-on-write and zfs snapshot creation is almost
  instantaneous.  There might be additional overheads from building on zfs
  though.  Currently the snapshot base is hard-wired to y/${branch}@base
  but should be parametrized.  This also must be populated beforehand, e.g.
  during machine startup

* Clean build chroots by just destroying the snapshot.

tmpfs:

* Enabled by use_tmpfs=1 and tmpfs_size in portbuild.conf

* The previous md strategy of mounting in used/, populating and then
  remounting (to avoid possible races from multiple builds claiming the
  same chroot) doesn't work here because tmpfs instances are destroyed at
  umount.  I am not entirely sure the simpler approach is free from races.
2007-07-29 20:27:54 +00:00
Kris Kennaway
8d84574a4a Refresh old sample portbuild.conf with the one currently used for i386
builds.  Probably this should be better documented and commented ;)
2007-07-29 20:06:05 +00:00