1
0
mirror of https://git.FreeBSD.org/ports.git synced 2024-12-12 03:00:28 +00:00
Commit Graph

1104 Commits

Author SHA1 Message Date
Mark Linimon
f149057e10 Restore the display of statistics for non-mainline runs. 2010-08-16 09:17:05 +00:00
Mark Linimon
8153ebd48c Add some missing error handling. 2010-08-16 06:22:37 +00:00
Mark Linimon
856a21b160 Handle yet another error case. 2010-08-15 09:16:51 +00:00
Mark Linimon
ff7f1257bf Correct some misleading documentation. 2010-08-10 05:39:57 +00:00
Mark Linimon
3a3eb397af Fix braino from production. 2010-08-02 07:11:11 +00:00
Mark Linimon
2e418d3548 Fix mishandling of root/all case. 2010-07-30 03:18:21 +00:00
Mark Linimon
2c4d0a67a8 Push PACKAGE_BUILDING and MALLOC_OPTIONS up a level.
Feature safe:	yes
2010-06-30 14:51:47 +00:00
Mark Linimon
aadc9bb5d1 Remove now-redundant setting of PACKAGE_BUILDING.
Feature safe:	yes
2010-06-30 14:50:06 +00:00
Mark Linimon
bcb18c7e0b Fix up usage of buildenv.client to deal with overenthusiastic
refactoring.  Pick up new CLIENT_MALLOC_OPTIONS parameter.  Clean
up comments.

Feature safe:	yes
2010-06-30 14:48:57 +00:00
Mark Linimon
937750f8bc Add CLIENT_MALLOC_OPTIONS.
Feature safe:	yes
2010-06-30 14:46:27 +00:00
Mark Linimon
bc2d32d595 The previous refactoring was too agressive. SRCBASE and friends are used
on both the client side and server side.

Pointy hat to:	self
Feature safe:	yes
2010-06-30 14:43:12 +00:00
Mark Linimon
090587b616 Correct the URLs; 'http://' is not included in MASTER_URL.
Feature safe:	yes
2010-06-28 01:59:25 +00:00
Mark Linimon
08115cf45a Document some hard-coded stuff in the scripts.
Feature safe:	yes
2010-06-25 23:45:38 +00:00
Mark Linimon
15fff60e91 A script that can be remotely run on package build clients to attempt to
clean up problems with squid.

Feature safe:	yes
2010-06-25 23:44:39 +00:00
Mark Linimon
2ca917890d Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

NOTE: these scripts are not yet parameterized, so this is WIP.

Feature safe:	yes
2010-06-25 23:43:24 +00:00
Mark Linimon
90089c9f17 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Several bugs are fixed and improvements are made:

 - parameterize everything.
 - switch to csup.  (This necessitates pushing the repos down one
   directory level, e.g. ports-head/ports and src/src-8.  However,
   this will give us the flexibility to have e.g. src/src-8.1 in
   the future.)
 - note that the misnomer filename 'cvsdone', which was created
   here, is now changed to '.updated'.

Feature safe:	yes
2010-06-25 23:42:51 +00:00
Mark Linimon
2fe59a537a Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

While here, do some refactoring.

Feature safe:	yes
2010-06-25 23:39:54 +00:00
Mark Linimon
a588885b10 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Note the change of the misnomer filename 'cvsdone' to '.updated'.

Feature safe:	yes
2010-06-25 23:38:01 +00:00
Mark Linimon
18d0ec2618 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Several bugs are fixed and improvements are made:

 - suppress error message if there are no logs yet.
 - change the misnomer filename 'cvsdone' to '.updated'.
 - fix a bug by un-escaping '_' in the timestamp.

Feature safe:	yes
2010-06-25 23:37:19 +00:00
Mark Linimon
5b3b7d003e Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

While here, fix error messages displayed on new runs.

Feature safe:	yes
2010-06-25 23:31:11 +00:00
Mark Linimon
b88965abda Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

*very* carefully note where the exported variables are consumed.

Several bugs are fixed and improvements are made:

 - the file UNAME_VERSION appears unused, so it is removed.  The
   information is passed in overrides to uname(1).
 - parameterize arch-specific setup code.

Feature safe:	yes
2010-06-25 23:30:20 +00:00
Mark Linimon
2e131db722 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Feature safe:	yes
2010-06-25 23:27:16 +00:00
Mark Linimon
2cff86f1e9 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Note the removal of the hard-coding of INDEX.N, where N has to be
a single digit, and also have no '.' in it.

Feature safe:	yes
2010-06-25 23:25:36 +00:00
Mark Linimon
338c392806 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Note the very special handling of cvs tag.

Feature safe:	yes
2010-06-25 23:24:23 +00:00
Mark Linimon
8784a39301 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Fix the sed script to correctly handle '.' in branch directory names.

Feature safe:	yes
2010-06-25 23:23:17 +00:00
Mark Linimon
3d5c2cff29 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Gracefully handle the case of INDEX failing to build.  This helps the
calling script to exit instead of spewing errors.

Feature safe:	yes
2010-06-25 23:22:45 +00:00
Mark Linimon
f1c9dc80e7 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Add the copying over of client.conf to the clients, and take care
to force the file copy regardless of whether it is a symlink.

Feature safe:	yes
2010-06-25 23:21:46 +00:00
Mark Linimon
6cbd9ecd57 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Several bugs are fixed and improvements are made:

 - the name of the journal is changed from 'make' (ew) to 'journal'.
 - 'cvsdate' is replaced by the more accurate name '.updated' and
   documentation adjusted to match.
 - make it more clear that '.updated' means 'ports tree updated'
   instead of 'src tree updated' (although the same filename is
   used for both).
 - correctly handle the general case of non-mainstream branches
   (e.g. "7-exp") without hardcoding "-exp".

Feature safe:	yes
2010-06-25 23:20:14 +00:00
Mark Linimon
31f34112f9 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Note: the major change is to require branch as an argument.  The
silly code that expected the branch to be passed via a symlinked
command such as 'dopackages.7' is removed.

Several bugs are fixed and improvements are made:

 - make the separation of 'date' and 'datestamp' clearer.
 - make handling of error conditions (e.g. for a first-time run
   where 'build create' has been forgotten) more obvious.

Feature safe:	yes
2010-06-25 23:16:46 +00:00
Mark Linimon
80e1a07ccb Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Several bugs are fixed and improvements are made:

 - the name of the journal file is changed from 'make' (ew) to
   'journal'.
 - the 'date' parameter becomes 'datestamp' to delinate the script's
   uses of the date command vs. the parameter.  This is mostly for
   readability.
 - add the -nochecksubdirs option.  This is only useful for restarted
   or incremental runs.
 - refactor the error returns a bit.
 - exit gracefully if no INDEXFILE.
 - fix a bug in the 'packages built' logic that would return -2 if no
   packages were actually built.
 - change the misnamed 'cvsdone' file to '.updated'.
 - clean up the 'task waiting' logic.

Feature safe:	yes
2010-06-25 23:14:13 +00:00
Mark Linimon
f06078517f Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Note that this script will now figure out 'branches' by itself, thus
removing the special-casing for '-exp'.

Feature safe:	yes
2010-06-25 23:08:14 +00:00
Mark Linimon
21fd819842 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Note: it is still not clear to me if this script is currently used.

Feature safe:	yes
2010-06-25 23:05:49 +00:00
Mark Linimon
ecc847e0d6 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

While here, rework the code a bit to parameterize arch-specific tasks.

Feature safe:	yes
2010-06-25 23:04:53 +00:00
Mark Linimon
1629562d00 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

While here, carefully document the origin of the values of the
environment variables that are passed in.

Feature safe:	yes
2010-06-25 23:03:11 +00:00
Mark Linimon
381f8cb94b Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

While here, do some cleanup.

Feature safe:	yes
2010-06-25 23:02:09 +00:00
Mark Linimon
75fda7ac40 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Several bugs are fixed and improvements are made:

 - the definitions for valid 'arch' and 'branch' are moved to
   server.conf.
 - the script is broken up into two pieces; the old 'buildenv' name
   becomes the server side, and 'buildenv.client' is add for the
   client side.  'buildenv.common' is what you would expect.  This
   makes the separation of what controls what more clear.
 - the concept of 'branch base' is generalized to match any pattern
   postpended with '-', thus removing the specialness of '-exp'.  More
   work remains on the other scripts to best take advantage of this.
 - as a corollary, 'branch' can also have '.' in it, e.g., 6.4.
 - the obsolete variables FTP_PASSIVE_MODE, PKGZIPCMD, and X_WINDOW_SYSTEM
   are removed.

Feature safe:	yes
2010-06-25 23:01:05 +00:00
Mark Linimon
2173088340 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

Several bugs are fixed and improvements are made:

 - archive directories are created  if they do not exist.
 - the location of ZFS-based directories is generalized.
 - the special string "-exp" disappears.
 - handling of buildid is made more robust.

Feature safe:	yes
2010-06-25 22:55:00 +00:00
Mark Linimon
11ad804e71 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

While here, fix a but in the handling of the 'all' case.

Feature safe:	yes
2010-06-25 22:51:05 +00:00
Mark Linimon
ae733423e0 Generalize the packge building scripts to be able to be run on more than
one 'head' node, rather than just pointyhat itself.

Constants are factored out into installation-specific files known as
portbuild/conf/server.conf and portbuild/conf/client.conf.  There is
only one server.conf file.  Individual <arch> directories may have
their own client.conf files, or may symlink to ../conf/client.conf.

This first set of files only gets the *.conf files and some cleanup.

Feature safe:	yes
2010-06-25 22:49:56 +00:00
Mark Linimon
73d56dd444 Variant of the allgohans script that does locking. This has been in
use on pointyhat for a while.

Feature safe:	yes
2010-06-25 22:31:00 +00:00
Mark Linimon
7f02f37b5d Note the files that shouldn't be checked into the repository (they
may contain private information).

Feature safe:	yes
2010-06-25 21:42:24 +00:00
Mark Linimon
0f99ddce2e Remove obsolete cruft that has not been used in several years.
Feature safe:	yes
2010-06-22 23:55:31 +00:00
Mark Linimon
e8f9e57eca A new file to document some generalization of scripts on a package
building server.  For security reasons, the scripts themselves will
not be checked in to this repository.

Please do not commit to this file without the approval of portmgr.

Feature safe:	yes
2010-06-22 23:48:07 +00:00
Mark Linimon
408042ec04 Some configuration information that is shipped to the clients of a
package building cluster.

Please do not commit to this file without the approval of portmgr.

Feature safe:	yes
2010-06-22 23:46:55 +00:00
Mark Linimon
8a1c53277f A new file to centralize definitions that apply to the the package nodes
used from a package building cluster.  This is part of a rewrite to remove
a great deal of hardcoding.

Please do not commit to this file without the approval of portmgr.

Feature safe:	yes
2010-06-22 23:46:06 +00:00
Mark Linimon
536fd89955 A new file to centralize definitions that apply to the server-side of
a package building cluster.  This is part of a rewrite to remove a great
deal of hardcoding.

Please do not commit to this file without the approval of portmgr.

Feature safe:	yes
2010-06-22 23:45:27 +00:00
Mark Linimon
aa7717e661 Catch up with master copy on pointyhat.
Note: all 3 of these files will be changing radically in the near future.
This is the last checkin from the "classic" codebase installation.

Feature safe:	yes
2010-06-22 02:51:54 +00:00
Mark Linimon
7ce566ebb0 These appear to no longer be used.
Feature safe:	yes
2010-06-22 02:45:41 +00:00
Romain Tartière
4a54a88a4f - Unbreak the -n flag;
- Remove invalid reference to freefall.

Approved by:	garga
2010-06-06 09:30:54 +00:00
Mark Linimon
666d694375 Restore line dropped in revision 1.21.
Pointyhat to:		linimon
2010-06-03 07:01:35 +00:00
Mark Linimon
fa2be9ee8b Sigh. This had already been done, just not checked out on production.
Pointy hat:	self
2010-05-31 20:06:08 +00:00
Mark Linimon
dcabe96b55 Get rid of the remnants of X11BASE. 2010-05-31 20:01:31 +00:00
Mark Linimon
69105de3df Remove some hardcoding. 2010-05-31 19:33:22 +00:00
Mark Linimon
74affe36b5 Add a comment. 2010-05-31 19:29:25 +00:00
Mark Linimon
05e815f4bd Eliminate some more hardcoding. Expand the pattern for "non-mainstream"
builds to also catch \., not just \-.
2010-05-25 20:08:56 +00:00
Mark Linimon
ed6716f76f Remove the hard-coding of '-exp'. While here, factor out some other
hardcoding to defines.
2010-05-25 07:37:48 +00:00
Mark Linimon
9672ea0555 Remove obsolete cruft. 2010-05-23 03:09:09 +00:00
Mark Linimon
a9c84913e9 Cronjob that runs on pointyhat to cleanup stale chroots on clients. 2010-05-22 00:58:11 +00:00
Mark Linimon
4a1e5c660e Trim backup files a bit; make the emails a bit more verbose. 2010-05-20 04:07:13 +00:00
Mark Linimon
b5b4dafb6f Refactor things a bit. 2010-05-20 04:04:24 +00:00
Mark Linimon
44bd147989 Improve error handling a bit. 2010-05-20 03:58:50 +00:00
Erwin Lansing
ad1b924b23 Revert the clang part of rev 1.12, which is now default in clangbsd. 2010-05-10 15:05:59 +00:00
Mark Linimon
7fc55a84af Add a case to detect a transient problem that shows up on all package
nodes, but most regularly on sparc64.  (Occasionally, on amd64 and ia64).

For reasons I haven't been quite able to track down, on some occasions
a pkg_add command is unable to extract a dependency; the tarfile shows
up as being truncated.  This does not seem to be due to disk-low or
memory-low conditions, nor is it a problem with scp; the md5 on the file
is fine when examined afterwards.

The only clue so far is that it seems to happen on systems with the most
package builds running simultaneously -- and thus, possibly more than one
pkg_add running in parallel.
2010-03-30 21:19:36 +00:00
Mark Linimon
5c7fe20b5f Add an entry for the "truncated distfiles" problem; improve one other entry. 2010-03-30 21:14:16 +00:00
Dirk Meyer
5e55949069 - allow addport to work in ports-mgnt again
Approved by:	garga, drso, flz
2010-03-25 15:44:17 +00:00
Mark Linimon
81ee5011c8 Catch up with production.
Feature safe:	yes
2010-02-19 16:27:24 +00:00
Mark Linimon
2492e191ee Bring this up to date with what is in production: add 'queue length'
column from pav; fix first-column formatting.

Feature safe:	yes
2010-02-19 16:15:58 +00:00
Mark Linimon
261bae1be6 Bah, another few references to "8-current", which no longer exists. 2010-02-05 18:09:52 +00:00
Jun Kuriyama
e2f30d68c5 - Add a script to check unneeded dependencies for perl modules. 2010-01-29 14:52:45 +00:00
Mark Linimon
d1c3f4ddd2 Add some new error cases to handle regressions in -current. (Not yet
analyzed for how much they will slow this script down; consider this a
rush-job.)

 - dirent denotes some change in the usage of dirent.h.
 - termios denotes the deprecation of <sys/termios.h>.
 - uname denotes the hiding of the uname symbol.  This has been backed
   out in src so let's hope this case can go away soon.
 - utmp_x denotes the replacement of utmp.h with utmpx.h.

Together these catch ~150 new errors on i386-9.  However, there are more
that are not caught (second-order effects.)
2010-01-20 09:08:55 +00:00
Mark Linimon
99e391bf77 Note that we are at 9-CURRENT now. Also, ia64 builds have resumed. 2009-12-19 14:05:51 +00:00
Pav Lucistnik
931cab63a3 - Remove use of X11BASE from pointyhat scripts 2009-12-18 14:25:58 +00:00
Maxim Sobolev
4ba8d15977 Correct depreciation warning with Python 2.6 and fix several warnings
from the pytchecker.
2009-11-21 00:40:59 +00:00
Mark Linimon
7676829042 A few more loose ends.
Testing by:	miwi
2009-11-19 06:47:05 +00:00
Mark Linimon
a7e34a9e41 Make the create command more robust by making even more directories
and checking more error results.
2009-11-19 03:25:47 +00:00
Mark Linimon
4bf33692a7 Properly supppress empty buildenvs. 2009-11-17 04:49:41 +00:00
Mark Linimon
1263d0c3a0 Refactor this to minimize hard-coding. When our support for branches
changes, now you only need to change 2 lines.
2009-11-16 00:17:05 +00:00
Mark Linimon
b5923ad846 Harden these a bit: make them identify on which host they were running;
skip unncessary steps.
2009-11-16 00:16:08 +00:00
Mark Linimon
8d20989476 Fix up the problems that can happen when 2 copies of makeworld are
running simultaneously for separate buildenvs.
2009-11-13 04:24:35 +00:00
Mark Linimon
8b562a8a84 Make more robust. 2009-11-12 23:48:36 +00:00
Mark Linimon
6b924aa4a6 Parameterize. 2009-11-12 23:48:22 +00:00
Mark Linimon
7c3e5b64a5 Harden this script so it will DTRT if any subdirectories of a new build,
or pieces of a pre-existing build, are missing.  These are both the case
when dopackages follows 'build create'.
2009-11-10 08:02:37 +00:00
Mark Linimon
debe14542f Remove a trailing slash on a directory name, that merely gets doubled
later in the code.
2009-11-10 07:54:50 +00:00
Mark Linimon
5b7f7f4aeb Finally fix the 'build create' command to DTRT by reusing some already-
working code.  While here, clean up a bit.
2009-11-10 00:18:04 +00:00
Mark Linimon
2d1dcdcd60 Reflect the fact that src-HEAD is now just a symlink to src-9. This
dance-step prevents having to introduce some special-case code into
the build script.
2009-11-10 00:17:20 +00:00
Mark Linimon
75ee9174fa Fix permissions and 'latest' link. 2009-11-09 09:09:19 +00:00
Mark Linimon
c01dff508b Remove a doubled slash. 2009-11-09 08:52:36 +00:00
Mark Linimon
9203344875 Implement the "build create" command. 2009-11-09 08:37:36 +00:00
Mark Linimon
99dc659674 Remove cruft. 2009-11-08 02:25:19 +00:00
Mark Linimon
661c4bdb3a Allow for the possibility of powerpc builds. Since these are not yet
set up, add error handling to gracefully handle missing directories.
2009-11-08 00:27:14 +00:00
Mark Linimon
551b5a7617 Allow for the possibility of powerpc builds. 2009-11-08 00:25:37 +00:00
Mark Linimon
4d6a33b6fa Remove dead code. 2009-11-08 00:25:03 +00:00
Mark Linimon
50b2db3470 Reflect reality on pointyhat. While here, clean up a bit. 2009-11-08 00:24:50 +00:00
Greg Larkin
e49ea39cc3 - Fixed "permission denied" error
- Prevented full ports tree build when operating with ~/.cvsrc

PR:		ports/139121
2009-10-06 00:48:13 +00:00
Gerald Pfeifer
413c1b6ad3 Correctly bump PORTREVISION?= instead of always resetting to 1 in this case.
PR:		138774
Approved by:	edwin
2009-09-15 09:34:08 +00:00
Mark Linimon
6700e1cbe7 Add links for 9-current. (note: 8-current is not 8-stable ... yet.)
While here, remove some staleness.
2009-08-24 02:32:52 +00:00
Mark Linimon
3fba981253 Generalize this script to remove the necessity of editing it every time
we create a new src branch.  While here, garbage collect some code that
hasn't been run for years.
2009-08-24 01:57:30 +00:00
Mark Linimon
99e14e14ec I believe that this script is obsolete. Retire it. 2009-08-23 15:30:24 +00:00
Mark Linimon
03def879af Add 9 and 9-exp to this sample file. Delete 6-exp2. 2009-08-23 14:38:17 +00:00
Erwin Lansing
6452647285 Add support for RELENG_8 2009-08-23 12:20:23 +00:00
Mark Linimon
86c19829e3 Reflect RELENG_9. 2009-08-23 11:01:42 +00:00
Erwin Lansing
ba594074e3 Add support for INDEX-9 2009-08-23 10:47:02 +00:00
Mark Linimon
94b770ad20 Reflect the fact that RELENG_9 has been created, and that there is some
interest in reviving ia64.  While here, generalize things a bit.
2009-08-23 10:32:15 +00:00
Mark Linimon
352d3da30f Turn off the 'show latest log' feature. While handy, this thrashes the
disks.

While here, note that there is some interest in reviving ia64 package
builds.
2009-08-23 10:08:15 +00:00
Erwin Lansing
cc977a1400 Add a timeout to the qmanager connection. 2009-08-13 15:21:20 +00:00
Erwin Lansing
c58c61be8d Make sure that some options to dopackages are passed to the environment
when invoking pdispatch.
2009-08-13 15:20:12 +00:00
Erwin Lansing
458ebe3860 Also put bzipped and checksums of INDEX, UPDATING and MOVED on the mirrors
PR:		131440
Requested by:	Hannes Hauswedell <freebsd@soulrebel.in-berlin.de>
2009-08-12 08:18:56 +00:00
Erwin Lansing
b38ab17a77 - Upload MOVED and UPDATING to the packages directory to make it
available on the ftp mirrors alongside the packages [1]
- While I'm here, remove a NOOP check for FreeBSD 4.x -exp

PR:		135024 [1]
Requested by:	Dominic Fandrey <kamikaze@bsdforen.de>
2009-08-11 07:51:37 +00:00
Mark Linimon
58d45fd566 5 and 5-exp are no longer supported. 2009-07-19 19:44:15 +00:00
Erwin Lansing
3accf1e172 Explicitly override LOCALBASE to make sure we don't get
confused by locally installed ports that might influence
PKGNAME or other variables.

Suggested by:	pav
2009-07-14 07:58:50 +00:00
Pav Lucistnik
0dbe63a36f - Seems magic slash stopped working on -CURRENT zfs, so do without it 2009-07-08 08:50:42 +00:00
Erwin Lansing
ca91e7339b Force cleaning of the environment from customized local settings. 2009-06-05 16:22:45 +00:00
Mark Linimon
6ded0e595c Switch the 'skipped' column to point to duds.verbose instead of duds,
now that it is available in all current builds.
2009-06-03 03:42:30 +00:00
Erwin Lansing
3f3cfcfe52 Use zfs snapshots that are periodically updated on pointyhat for the
normal builds instead of a local cvs update.  This speeds up the
script several times and easies I/O on pointyhat.
2009-05-28 15:54:38 +00:00
Pav Lucistnik
ce4e44feb4 - Update to reflect a reality on pointyhat 2009-05-21 16:17:15 +00:00
Pav Lucistnik
46e2ac6a6f - Separate ports tree updating to use cvsup from cvsup-master instead of cvs
update from NFS mount, which is just way too slow
2009-05-21 16:15:56 +00:00
Philip M. Gollucci
5d1e7f6179 note mark_safe.pl 2009-05-17 11:40:16 +00:00
Philip M. Gollucci
1e47318f7d o don't set anything in ports with NO_BUILD
o fix rubygem skipping
o next, not die
2009-05-17 11:12:24 +00:00
Philip M. Gollucci
0df0ef1432 Do not mark rubygem ports, its a NO-OP
Discussed with: pav on irc (#bsdports)
2009-05-16 01:51:40 +00:00
Ion-Mihai Tetcu
3fb7a3d1a2 Make previous commit work for both context and unified input patches. 2009-05-10 16:34:20 +00:00
Ion-Mihai Tetcu
ec5050b6b2 When naming the split patch files, replace the original path '/' with '__'
instead of a single '_' to follow the current patch naming guidelines.
2009-05-10 16:28:52 +00:00
Philip M. Gollucci
f700032948 - fix --port support
- don't change the Makefile if MAKE_JOBS_(UN)SAFE is already present
2009-04-29 21:16:43 +00:00
Philip M. Gollucci
4dfe2a9982 - Start some docs
- Add --ports -p to specify a list on the cli
2009-04-29 21:08:53 +00:00
Philip M. Gollucci
4fa6707cf8 Script to set MAKE_JOBS_(UN)SAFE= yes for a MAINTAINERs ports.
Please be sure to completely test the results of running this _BEFORE_ committing
several times and in a Tinderbox

Feel free to commit to this file
2009-04-22 00:33:44 +00:00
Mark Linimon
dbee5e6a5e Fix bug where number of error logs was incorrect. 2009-03-26 10:38:44 +00:00
Mark Linimon
4b88f15929 Rewrite of processlog by kris. 2009-03-26 10:33:51 +00:00
Mark Linimon
cceb2bfce4 - Add a new case 'nested-declaration' to catch errors on -current.
- Fix false positives for 'cluster'.
 - Add 'error: invalid pure specifier' case to badc++.
2009-03-25 10:18:41 +00:00
Christian Weisgerber
1895bc9e69 Use diff -p both when first creating and when updating a patch.
Previously a patch created by update-patches would be changed again
by the next update-patches run.
2009-02-15 16:16:33 +00:00
Pav Lucistnik
93139e9aed Remove 6-exp2 builds, they haven't been used in a while 2009-01-21 16:11:24 +00:00
Gerald Pfeifer
a351950aad Factor out the code that performs the PORTREVISION bump for one port.
Approved by:	edwin
2009-01-07 21:33:17 +00:00
Erwin Lansing
9f53e58103 Reduce per branch code duplication
Submitted by:	pgollucci
2008-12-27 21:24:30 +00:00
Maxim Sobolev
3fe7e87724 Make sure we don't add ${WRKDIR}. 2008-10-21 21:32:08 +00:00
Mark Linimon
c9551596f6 Handle the rest of the edge cases; drop 5 as obsolete; cleanup.
Now re-enabled on production.

Approved by:	portmgr (self)
2008-09-18 06:01:57 +00:00
Mark Linimon
eed6489990 Bring this into the new portbuild world order. While here, try to
suppress hrefs for non-existant files (incomplete).

Approved by:	portmgr (self)
2008-09-17 18:01:07 +00:00
Mark Linimon
6b397acc55 RELENG_5 is desupported.
Approved by:	portmgr (self)
2008-09-12 03:25:47 +00:00
Mark Linimon
1eec24b50c Major overhaul:
- bring this closer to the default FreeBSD page style
 - remove unsupported releases
 - remove the date stamps, which no longer work
 - remove obsolete commented-out junk

Discussed on:	portmgr, some time ago
2008-09-07 05:34:13 +00:00
Kris Kennaway
452822535e Add a new script for performing high-level administration of a build.
list : lists available builds
   clone : creates a new build by cloning a previous one
   portsupdate : update a ports tree to the latest ZFS snapshot
   srcupdate : update a src tree to the latest ZFS snapshot
   cleanup : clean up or remove a build on the clients
   destroy : remove a build on the server

There is some trickiness here in that various commands either expect
to run as root, or expect to run as a ports-* user.  For the latter
case we can easily use su to proxy as the ports user when running as
root; for the former we use the buildproxy to validate and re-execute
the command as root.
2008-08-08 16:24:05 +00:00
Kris Kennaway
5918a825ad Add an option to clean up old distfiles. We don't have enough space to
keep dozens of full distfile sets lying around.
2008-08-08 16:19:38 +00:00
Erwin Lansing
3a04ff0e8a Update the list of openoffice.org ports: remove 1.x and add 3.x 2008-07-29 18:06:33 +00:00
Kris Kennaway
54d4698a9f Oops, commit the right version of this (adds extra checks that only
expected commands will be proxied)
2008-07-26 15:34:49 +00:00
Kris Kennaway
6c3363b444 * Simple client that passes on a build request to the buildproxy 2008-07-26 15:25:21 +00:00
Kris Kennaway
88e9a32308 * Python daemon run as root that proxies privileged build commands for
the ports-* users.  Currently it is not possible to delegate
  management of ZFS filesystems to non-root users, so root privilege
  is required to manipulate them.  We validate the command passed on
  a local domain socket and re-execute the build script with the requested
  parameters.
2008-07-26 15:24:13 +00:00
Kris Kennaway
d1aea0930d Script run from cron to regularly update the master ZFS copies of the
ports and source trees.  Since we have >=1 consumer of these trees
that run frequently but do not insist on up-to-the-second trees, it
makes sense to "pre-update" them regularly and then then re-use in all
of the consumers, instead of potentially doing several updates
simultaneously or on demand.  Consumers can clone the ZFS snapshot
into their local filesystem which takes a couple of seconds instead of
minutes or tens of minutes for the CVS update.

We update to a date stamp instead of "." because this avoids
ambiguity of commits that happen while the tree update is in progress
(unfortunately it's slower).
2008-07-26 15:16:16 +00:00
Kris Kennaway
9ed197c29c Script run from cron on the package clients to report metrics to ganglia.
Currently we collect:

* The current and maximum number of vnodes in use

* The number of packages built over the past hour
2008-07-26 15:09:00 +00:00
Kris Kennaway
4663e0b500 Simple script to expire ZFS snapshots older than a certain age 2008-07-26 15:06:41 +00:00
Kris Kennaway
17885ef52d Python script for backing up ZFS filesystems on pointyhat. For each
listed filesystem we take a new snapshot each time it is run and if
the last full backup was not too long ago, do a compressed incremental
backup from the previous backup.
2008-07-26 15:05:58 +00:00
Kris Kennaway
00cada47c5 * Add comment that this is unused 2008-07-26 15:01:50 +00:00
Kris Kennaway
b472fe55ac * Add comments 2008-07-26 15:01:30 +00:00
Kris Kennaway
e683ebb83c * Cleanup
* Catch up to build ID directory changes

* Support a meta-hostname of 'all' for setting up all clients at once.
  This is better than the old way of running one copy of the script
  for each client by hand, since it is easier and involves less
  duplicated work.

* We copy in the per-build ports, src, and bindist .tbz files and .md5
  checksums, as well as refreshing the build scripts and
  bindist-$(hostname).tar customization tarball.

* The -force switch forces copying of files and re-extraction of the
  tarballs on the client.  This is necessary in order to propagate
  local changes to the tarballs after the initial client setup
  (e.g. if you need to change a file in the ports tree, it must be
  recompressed, redistributed, and re-extracted on the client).

* The -queue switch will poll the client's job queue after completion
  of the setup.  This is racy and should only be used when the machine
  is not currently accepting jobs.

* For cleaning up a build the 'build cleanup' command should now be
  used instead.  It calls back into this command but also allows full
  clenaup of build-local files on the client.

TODO: "all" setups are hard on the server since they may spawn dozens
of rsyncs at once.  A better solution would be to have a worker pool
of setup tasks to limit the maximum load.
2008-07-26 15:00:37 +00:00
Kris Kennaway
89c8fd897f * Cleanup
* Catch up to build ID directory changes
* Make it easier to kill a build by not running dopackages in the background
  where it is detached from shell job control.  Now, sending a termination
  signal to this process (e.g. ^C) will also kill off the dopackages script
  and in turn the processes created by it.  Some background processes
  spawned by dopackages, pdispatch, etc, may still remain and need to be
  killed by hand.
2008-07-26 14:52:05 +00:00
Kris Kennaway
4a7f6d83cb * Cleanup
* Catch up to build ID directory changes

* Improve usage()

* Fix a variety of small bugs

* Remove support for -ftp builds: we have not supported direct
  uploading for many years due to the desire to manually inspect build
  output for quality

* All data associated to a build is now localized in its own directory
  named according to a build ID:
  /var/portbuild/${arch}/${branch}/builds/${buildid}, where ${buildid}
  is the creation time.  These are actually ZFS filesystems.

* Tasks such as cloning a new build, updating a ZFS snapshot, and
  cleaning up a build are exported to the "build" script, which can be
  used independently.

* Creating a new build is done by ZFS cloning and takes a couple of
  seconds since it is copy-on-write (i.e. no data needs to be copied).

* Ports and source trees are also cloned from pre-updated ZFS images
  (updated regularly from the "updatesnap" cron job).  In most cases
  we do not care if we are building a ports tree that is an hour or so
  old since it will become outdated almost immediately anyway, so no
  matter what we do there will be times when a port has been fixed by
  the time the build error is generated by a client.

* In case an up-to-the-second tree is desired, the -portscvs and
  -srccvs switches update the existing ports tree via CVS.

* -noports and -nosrc can be used to prevent any automatic changes to
   the ports tree.  This is useful for dealing with local
   modifications (e.g. for -exp builds), since the default when
   creating a new build is to replace the previous trees with fresh,
   pristine trees.  If you forget to use this then any local changes
   that are not also present in other trees will be lost.

* By default we keep two builds for each arch/branch pair.  These
  build IDs also may be referred to via "latest" and "previous"
  symlinks.  When creating a new build, the old "previous" build is
  destroyed by default, unless it was originally created using the
  -keep switch.  This prevents the build from being destroyed
  automatically.

* By default when a build finishes all of the clients are completely
  cleaned up (i.e. all build data such as ports trees, tarballs,
  client chroots, etc are deleted).  This is needed to save space on
  the clients.  If you expect to *immediately* perform further builds
  after this one completes, the -nocleanup switch prevents this step.
  Otherwise they will just be set up again if further builds are
  scheduled.

* Try to parallelize build pre-processing as much as possible, by
  running jobs in the background wherever possible.  In several places
  we operate on the same parts of the filesystem from multiple jobs,
  so we can make good use of caching to improve performance

* Clients no longer need to be set up explicitly at the start of the
  build, they will be set up on-demand when the first job is
  dispatched to them.  This allows fast clients or those that already
  have been set up to begin building ports as soon as possible, while
  slow clients are set up in the background.  It also improves
  robustness of client recovery, e.g. if the client was offline at the
  time of build startup but later brought back online.

* Optimize copying back in the previous set of restricted packages by
  hardlinking instead of copying.

TODO: The record of failed ports is arch/branch-global still.  This is
the only thing preventing us from running concurrent builds of the
same arch/branch (e.g. while one is stuck building openoffice, the
next build can start to keep the cluster busy).  The difficulty is
that one build from a later ports tree may signal that a build was
successful, then a phase 2 build from an earlier ports tree may
indicate that it was broken.  The solution is probably to migrate this
to a real database instead of a flat file, and query it for the set of
broken ports as of a certain ports tree date.
2008-07-26 14:49:26 +00:00
Kris Kennaway
efe865a26c * Catch up to build ID directory changes
* Clients no longer mount ports/src trees via NFS (even the FreeBSD.org
  local clients).  This was putting too much load on the server and
  slowing down builds.

* Instead ports and src .tbz files are pushed to the clients and
  unpacked.  MD5 checksums are used to verify correctness

* -force forces re-extraction of the tarballs even if they exist and
  appear to be checked out

* Also unpack the compressed bindist

TODO: When we are not using md or ZFS builds it would be even faster
to keep an unpacked copy of the bindist on the scratch filesystem and
hardlink the files into the target directory
2008-07-26 14:19:31 +00:00
Kris Kennaway
b9dde2b9f8 * Catch up to build ID directory changes
* Optimize by copying old packages using cpio -dumpl (i.e. create hardlink
  instead of copying the file).
2008-07-26 14:14:35 +00:00
Kris Kennaway
07e904cab8 * Catch up to build ID directory changes 2008-07-26 14:13:35 +00:00
Kris Kennaway
1d5ba88d7a * Cleanup
* Catch up to build ID directory changes
* Remove need for /etc/arch file
2008-07-26 14:12:53 +00:00
Kris Kennaway
9f29c725dd * Cleanup
* Catch up to build ID directory changes
* Improved support for ZFS
* Desupport X11BASE
2008-07-26 14:12:28 +00:00
Kris Kennaway
f8a634d336 * Cleanup
* Catch up to build ID directory changes
* Improved support for ZFS builds
* Improved robustness
* Report status verbosely to the caller; whether we succeeded in claiming
  a chroot, whether the caller needs to first set up the client, or
  whether a setup is in progress.
* If we discover that the client has not been set up either because it
  freshly booted and newfs'ed its filesystem, or because a particular
  build has not yet been encountered, atomically claim a cookie and
  report this to the caller to act on
2008-07-26 14:11:26 +00:00
Kris Kennaway
316ad2a0a7 * Cleanup
* Catch up to build ID directory changes
2008-07-26 14:07:49 +00:00
Kris Kennaway
1dc6876bab * Cleanup
* Catch up to build ID directory changes
* Add helper functions for resolving a build ID symlink and
  validating an arch/branch combination (centralize instead of doing it
  in many scripts)
2008-07-26 14:06:30 +00:00
Kris Kennaway
5acb87ae92 * Desupport alpha and ia64
* Catch up to build ID directory changes
2008-07-26 14:05:01 +00:00
Kris Kennaway
46356ad8f8 * Add comments 2008-07-26 14:04:23 +00:00
Kris Kennaway
0b457b9cf0 * Implement basename and dirname using shell builtins 2008-07-26 14:02:55 +00:00
Kris Kennaway
f204e78013 * Cleanup
* Catch up to build ID directory changes
* Record package build completion for reporting to ganglia
2008-07-26 14:02:38 +00:00
Kris Kennaway
90e209c3d9 * Cleanup
* Catch up to build ID directory changes
* Add support for ssh_cmd and scp_cmd to allow using HPN-SSH with the
  none cipher where possible (for performance)
* Lazy client setup; claim-chroot will report if the client needs to be
  set up with this buildid, and we initiate the setup and poll until
  it is complete.  This allows fast clients to begin building before
  slow ones have finished setting up.

TODO: a better solution would be to avoid trying to dispatch jobs onto
clients that are in the process of setting up, since they often have low
loads and are picked preferentially by the job scheduler.
2008-07-26 14:01:07 +00:00
Kris Kennaway
a52cf32275 * Cleanup
* Remove vestiges of archaic support for building bindists from FTP
  snapshots; we haven't used this for years and building a world is no
  longer a challenge
* Revert half-baked bindist generation number and make it per-buildid
  instead.  Compress and md5 it for distribution to the clients.

TODO: Merge with makeworld?
2008-07-26 13:54:59 +00:00
Kris Kennaway
46a114508f * Cleanup
* Catch up to build ID directory changes
* Optimize by using ECHO_MSG=true instead of /usr/bin/true
* Try harder to avoid pollution from local host
2008-07-26 13:52:32 +00:00
Kris Kennaway
18cafe9ff8 * Cleanup
* Catch up to build ID directory layout
2008-07-26 13:51:30 +00:00
Kris Kennaway
1ba1b7f79e * Cleanup
* Catch up to build ID directory changes
* Export the INDEX_PRISTINE and INDEX_QUIET variables (old bug)
* Desupport X11BASE
2008-07-26 13:50:15 +00:00
Kris Kennaway
4bcc698d1c * Cleanup
* Catch up to build ID directory changes
* Desupport 5.x
2008-07-26 13:47:45 +00:00
Kris Kennaway
e9fe4c9896 * Cleanup
* Catch up to build ID directory changes
* Optimize by using __MAKE_SHELL=/rescue/sh
2008-07-26 13:47:03 +00:00
Kris Kennaway
23fa193076 Rewrite in python and combine the functions of the former
checkmachines script.  Polls build machines for their status either
once-off or regularly as a daemon.  Optionally it will update the
queue entries but this remains subject to race conditions.

TODO: Integrate with queue manager and forward machine status changes
to it
2008-07-26 13:45:19 +00:00
Kris Kennaway
335c9a9ec3 More verbose status reporting using key=value format. We now also
report error status, architecture and OS version, and available build
environments, as well as load and number of running jobs
2008-07-26 13:42:14 +00:00
Kris Kennaway
54e565eedc Major optimizations for 'make index' and other recursive traversal
targets.

* Use /rescue/sh for index builds instead of /bin/sh, when it exists.
  The former is statically linked and faster to execute, which becomes
  significant when executing it tens of thousands of times.  This
  trick can be used with other recursive targets by passing in
  __MAKE_SHELL.

* Get rid of make variable assignments that use != command invocations
  in the critical path, using several methods:

  - rewriting logic to use shell or make builtins instead of external command executions
  - macroizing commands and executing them in the targets where they
    are needed instead of with every invocation of make
  - precomputing the results of invariant commands in
    bsd.port.subdir.mk and passing them in explicitly to child makes,
    and using this to avoid recalculation in all the children. NB: the
    commands are still run one per top-level subdirectory but this
    does not currently seem to be a major issue.  They could be moved
    further up into the top-level Makefile at the cost of some
    cleanliness.
  - Committers are strongly discouraged from adding further "bare" !=
    assignments to the ports tree, even in their own ports.  One of
    the above strategies should be used to avoid future bloat.

* Rewrite the core 'describe' target to work entirely within a single
  shell process using only builtin commands.  The old version is
  retained as a backup for use on systems older than 603104, which
  does not have the make :u modifier.  This cuts down the number of
  processes executed during the course of a 'make index' by an order
  of magnitude, and we are essentially now amortized to the minimum of
  a single make + sh instance per port, plus whatever commands the
  port makefile itself executes (which are usually unnecessary and
  bogus).

* Less validation of the WWW: target is performed; this can become
  policed at a port level by portlint.  Specifically we look at the
  second word of the first line beginning with "WWW:" in pkg-descr,
  and append "http://" to it unless it already begins with "http://",
  "https://" or "ftp://".  Thanks to dougb for the idea of how to
  extract WWW: using shell builtins.

* Use the "true" shell builtin instead of echo > /dev/null for a
  measurable decrease in CPU use.

* Add a note about dubious escaping strategy in bsd.port.subdir.mk

* Minor change in output of 'make describe': it no longer strips
  trailing CR characters from pkg-descr files with MSDOS CR/LF
  termination.  Instead the makeindex perl script that post-processes
  make describe into the INDEX is tweaked to strip on input.

The bottom line is that on my test hardware INDEX builds are now
faster by more than a factor of 2 and with a reduction in system time
by a factor of 4-8 depending on configuration.
2008-07-19 17:59:41 +00:00
Edwin Groothuis
17e6ef1c06 Have a different approach on how to check if the modules file has changed.
Because the $FreeBSD$ keyword isn't expanded in the new version, we can't
just do a diff, check the return value and ignore the output.

Every new modules file, changed or not with regarding to the contents,
has at least four lines in the diff output (line number, old line,
seperator, new line). Only commit it if there are more than four
lines difference between it.
2008-07-14 03:56:23 +00:00
Edwin Groothuis
00809d33a0 Keep the sources of the CVSROOT-ports/modules update script safely
in the ports CVS repository.
2008-07-08 21:03:42 +00:00
Edwin Groothuis
de831ac6da [PATCH] Tools/scripts: Some cleanups, additions to bump_version.pl
- added -c, doesn't change anything
	- added -n, deal with an already checkouted tree
	- removed negative logic in favor of 'unless'
	- switch to 3 arg form of open()
	- don't use globs for filehandles, this is been obsolete
	  since at least 5.6.1
	- handle possible errors in close()
	- allow CVSROOT to be overriden in the ENV

PR:		ports/125025
Submitted by:	"Philip M. Gollucci" <pgollucci@p6m7g8.com>
2008-07-06 02:00:52 +00:00
Mark Linimon
29b9f9c62d Reflect latest changes from production:
- no more 5-exp
 - add 8, 8-exp
 - fix two error-name hrefs
2008-07-02 08:44:20 +00:00
Kris Kennaway
a530ab018b This conversion script is no longer useful 2008-06-25 22:27:17 +00:00
Erwin Lansing
e4bbe0c85e CVSROOT/modules no longer contains an entry per port, remove support
for this from these scripts.
2008-06-19 22:09:01 +00:00
Kris Kennaway
cabd6f0d4a Modernize this script a bit.
* Remove 5.x support
* Leave the archaic ftp snapshot support for now, it is not hurting anything
  but will not work
* Be more careful when removing files (use absolute paths)
* Switch to bindist/tmp for the tmp dir
* Fix the recording of the bindist.tar generation number
* Get rid of redundant or useless processing of the world image
2008-06-11 13:30:35 +00:00
Kris Kennaway
8a8d78247c * Distfile collection is now the default; replace -distfiles with -nodistfiles
* Record the CVS update stamp in some extra places and make sure to remove it
  if the build is started with -noportscvs (since this probably means the
  ports tree was updated by hand at some random time)
2008-06-11 13:28:30 +00:00
Kris Kennaway
fce1fcb22a Add some test -d's to avoid cd'ing into directories that do not exist 2008-06-11 13:25:49 +00:00
Kris Kennaway
fdbc5869f0 Major optimizations. Instead of copying the distfiles around, mv
them in batches according to their target directory.
2008-06-11 13:25:13 +00:00
Kris Kennaway
efaa197bfb Revive this script and make it useful. Transfer the distfiles using rsync
and make sure they have been post-processed first.
2008-06-11 13:24:17 +00:00
Kris Kennaway
271351e954 * Catch up to X11R6 removal
* Keep RESTRICTED distfiles in a separate DISTDIR so we can easily
  avoid accidentally publishing them to the FTP site (idea from des@)
2008-06-11 13:22:58 +00:00
Edwin Groothuis
4937bd57bd Add support for "PORTREVISION?=" and "PORTEPOCH"
Tested on: devel/gettext bump
2008-06-06 05:52:57 +00:00
Kris Kennaway
5953f77e24 Rewrite this to make it more efficient (fewer external command
invocations).  It also fixes some edge cases that were not handled in
the previous version.

TODO: Correctly report IPv6 sockets (already in use by the sparc64 build)
2008-06-02 19:46:03 +00:00
Erwin Lansing
cdb5013c48 Remove support for FreeBSD 5.x 2008-06-01 09:26:12 +00:00
Mark Linimon
11ea3eeed1 Remove the force file, if it was used.
Forgotten by:	linimon
2008-05-29 14:14:54 +00:00
Mark Linimon
ed5147b294 Add a force function here, just like processlogs. 2008-05-29 02:34:41 +00:00
Mark Linimon
4cc67bd216 Add a note that processonelog and processlogs2 are finicky about the
header format of the log files.
2008-05-29 01:46:08 +00:00
Mark Linimon
f5c9292932 Fix these after the 1.31 update to buildscript.
Forgotten by:	pav
2008-05-29 01:45:39 +00:00
Kris Kennaway
9100b4ee0d Rewrite this in python instead of shell. Because we can read the
INDEX once and process internally instead of invoking many external
utilities, runtime is improved from ~20 minutes to <10 seconds.
2008-05-25 18:07:49 +00:00
Kris Kennaway
778518d745 NNew build scheduler written in python to replace the make+sh job
ordering, which had become too limited.

We now build packages ordered by those that are part of the longest
dependency chains first.  This has the effect of building the deepest
parts of the tree first and levelling out the tree height, hopefully
avoiding the situation we currently face where there appear
bottlenecks late in the build where the cluster becomes mostly idle
while waiting for a few long dependency chains to finish building
before the cluster can become fully loaded again.

The algorithm is that we sort the list of remaining packages according
to height (longest dependency chain), then add leaf packages from each
in order until we have filled a queue of length between 100 and 200,
to amortise the cost of this queue rebalancing while not losing the
height averaging property.  Jobs are dispatched from this queue into
worker threads as machine slots become available.

Unlike the make-based solution that required a fixed -j concurrency
value and could not respond to addition/removal of build resources, we
now can dynamically add new machines as they become available to the
queue.

The other advantage of using python is that we have more
customisability and visibility into the build status, e.g. we
periodically report the number of remaining packages, as well as the
list of deepest packages that we are working on.

TODO:

* Implement mtime checking for parent package staleness, so that
  parents are rebuilt if the dependencies are touched more recently.
  Currently packages will not be rebuild if they exist, whether or not
  they are "stale" wrt their dependencies.

* Offload the machine selection into an external queue manager.
  Currently the queue manager used here doesn't interoperate with the
  old one (getmachine/releasemachine) because it's not possible to use
  the lockf()-based mutual exclusion within a multithreaded client.
  Doing that will also allow for a more flexible job placement
  algorithm as well as finer queue customization.
2008-05-10 13:22:51 +00:00
Andrew Pantyukhin
0b5ca793be Add a tiny sh hack - ardiff - compares two archives. 2008-05-06 16:17:01 +00:00
Pav Lucistnik
989ac675fc Remove XFree86-4 from quickports 2008-04-11 11:34:30 +00:00
Pav Lucistnik
b88fea571b Parallelize to 4 concurrent jobs 2008-04-11 11:33:38 +00:00
Pav Lucistnik
0c02d135ff Include per-machine configuration and respect use_zfs flag 2008-04-11 11:32:29 +00:00
Pav Lucistnik
2545b7c79e Sync with pointyhat (reorg) 2008-04-11 11:31:33 +00:00
Pav Lucistnik
7b00cf5640 - Add the port's Makefile ident string to the build log headers
PR:		113234
Submitted by:	Andrej Zverev <andrey.zverev@electro-com.ru>

- Sync with actual script on pointyhat (whitespace, mostly)
2008-03-13 09:54:03 +00:00