1
0
mirror of https://git.FreeBSD.org/ports.git synced 2024-12-05 01:55:52 +00:00
Commit Graph

841 Commits

Author SHA1 Message Date
Erwin Lansing
3f3cfcfe52 Use zfs snapshots that are periodically updated on pointyhat for the
normal builds instead of a local cvs update.  This speeds up the
script several times and easies I/O on pointyhat.
2009-05-28 15:54:38 +00:00
Pav Lucistnik
ce4e44feb4 - Update to reflect a reality on pointyhat 2009-05-21 16:17:15 +00:00
Pav Lucistnik
46e2ac6a6f - Separate ports tree updating to use cvsup from cvsup-master instead of cvs
update from NFS mount, which is just way too slow
2009-05-21 16:15:56 +00:00
Philip M. Gollucci
5d1e7f6179 note mark_safe.pl 2009-05-17 11:40:16 +00:00
Philip M. Gollucci
1e47318f7d o don't set anything in ports with NO_BUILD
o fix rubygem skipping
o next, not die
2009-05-17 11:12:24 +00:00
Philip M. Gollucci
0df0ef1432 Do not mark rubygem ports, its a NO-OP
Discussed with: pav on irc (#bsdports)
2009-05-16 01:51:40 +00:00
Ion-Mihai Tetcu
3fb7a3d1a2 Make previous commit work for both context and unified input patches. 2009-05-10 16:34:20 +00:00
Ion-Mihai Tetcu
ec5050b6b2 When naming the split patch files, replace the original path '/' with '__'
instead of a single '_' to follow the current patch naming guidelines.
2009-05-10 16:28:52 +00:00
Philip M. Gollucci
f700032948 - fix --port support
- don't change the Makefile if MAKE_JOBS_(UN)SAFE is already present
2009-04-29 21:16:43 +00:00
Philip M. Gollucci
4dfe2a9982 - Start some docs
- Add --ports -p to specify a list on the cli
2009-04-29 21:08:53 +00:00
Philip M. Gollucci
4fa6707cf8 Script to set MAKE_JOBS_(UN)SAFE= yes for a MAINTAINERs ports.
Please be sure to completely test the results of running this _BEFORE_ committing
several times and in a Tinderbox

Feel free to commit to this file
2009-04-22 00:33:44 +00:00
Mark Linimon
dbee5e6a5e Fix bug where number of error logs was incorrect. 2009-03-26 10:38:44 +00:00
Mark Linimon
4b88f15929 Rewrite of processlog by kris. 2009-03-26 10:33:51 +00:00
Mark Linimon
cceb2bfce4 - Add a new case 'nested-declaration' to catch errors on -current.
- Fix false positives for 'cluster'.
 - Add 'error: invalid pure specifier' case to badc++.
2009-03-25 10:18:41 +00:00
Christian Weisgerber
1895bc9e69 Use diff -p both when first creating and when updating a patch.
Previously a patch created by update-patches would be changed again
by the next update-patches run.
2009-02-15 16:16:33 +00:00
Pav Lucistnik
93139e9aed Remove 6-exp2 builds, they haven't been used in a while 2009-01-21 16:11:24 +00:00
Gerald Pfeifer
a351950aad Factor out the code that performs the PORTREVISION bump for one port.
Approved by:	edwin
2009-01-07 21:33:17 +00:00
Erwin Lansing
9f53e58103 Reduce per branch code duplication
Submitted by:	pgollucci
2008-12-27 21:24:30 +00:00
Maxim Sobolev
3fe7e87724 Make sure we don't add ${WRKDIR}. 2008-10-21 21:32:08 +00:00
Mark Linimon
c9551596f6 Handle the rest of the edge cases; drop 5 as obsolete; cleanup.
Now re-enabled on production.

Approved by:	portmgr (self)
2008-09-18 06:01:57 +00:00
Mark Linimon
eed6489990 Bring this into the new portbuild world order. While here, try to
suppress hrefs for non-existant files (incomplete).

Approved by:	portmgr (self)
2008-09-17 18:01:07 +00:00
Mark Linimon
6b397acc55 RELENG_5 is desupported.
Approved by:	portmgr (self)
2008-09-12 03:25:47 +00:00
Mark Linimon
1eec24b50c Major overhaul:
- bring this closer to the default FreeBSD page style
 - remove unsupported releases
 - remove the date stamps, which no longer work
 - remove obsolete commented-out junk

Discussed on:	portmgr, some time ago
2008-09-07 05:34:13 +00:00
Kris Kennaway
452822535e Add a new script for performing high-level administration of a build.
list : lists available builds
   clone : creates a new build by cloning a previous one
   portsupdate : update a ports tree to the latest ZFS snapshot
   srcupdate : update a src tree to the latest ZFS snapshot
   cleanup : clean up or remove a build on the clients
   destroy : remove a build on the server

There is some trickiness here in that various commands either expect
to run as root, or expect to run as a ports-* user.  For the latter
case we can easily use su to proxy as the ports user when running as
root; for the former we use the buildproxy to validate and re-execute
the command as root.
2008-08-08 16:24:05 +00:00
Kris Kennaway
5918a825ad Add an option to clean up old distfiles. We don't have enough space to
keep dozens of full distfile sets lying around.
2008-08-08 16:19:38 +00:00
Erwin Lansing
3a04ff0e8a Update the list of openoffice.org ports: remove 1.x and add 3.x 2008-07-29 18:06:33 +00:00
Kris Kennaway
54d4698a9f Oops, commit the right version of this (adds extra checks that only
expected commands will be proxied)
2008-07-26 15:34:49 +00:00
Kris Kennaway
6c3363b444 * Simple client that passes on a build request to the buildproxy 2008-07-26 15:25:21 +00:00
Kris Kennaway
88e9a32308 * Python daemon run as root that proxies privileged build commands for
the ports-* users.  Currently it is not possible to delegate
  management of ZFS filesystems to non-root users, so root privilege
  is required to manipulate them.  We validate the command passed on
  a local domain socket and re-execute the build script with the requested
  parameters.
2008-07-26 15:24:13 +00:00
Kris Kennaway
d1aea0930d Script run from cron to regularly update the master ZFS copies of the
ports and source trees.  Since we have >=1 consumer of these trees
that run frequently but do not insist on up-to-the-second trees, it
makes sense to "pre-update" them regularly and then then re-use in all
of the consumers, instead of potentially doing several updates
simultaneously or on demand.  Consumers can clone the ZFS snapshot
into their local filesystem which takes a couple of seconds instead of
minutes or tens of minutes for the CVS update.

We update to a date stamp instead of "." because this avoids
ambiguity of commits that happen while the tree update is in progress
(unfortunately it's slower).
2008-07-26 15:16:16 +00:00
Kris Kennaway
9ed197c29c Script run from cron on the package clients to report metrics to ganglia.
Currently we collect:

* The current and maximum number of vnodes in use

* The number of packages built over the past hour
2008-07-26 15:09:00 +00:00
Kris Kennaway
4663e0b500 Simple script to expire ZFS snapshots older than a certain age 2008-07-26 15:06:41 +00:00
Kris Kennaway
17885ef52d Python script for backing up ZFS filesystems on pointyhat. For each
listed filesystem we take a new snapshot each time it is run and if
the last full backup was not too long ago, do a compressed incremental
backup from the previous backup.
2008-07-26 15:05:58 +00:00
Kris Kennaway
00cada47c5 * Add comment that this is unused 2008-07-26 15:01:50 +00:00
Kris Kennaway
b472fe55ac * Add comments 2008-07-26 15:01:30 +00:00
Kris Kennaway
e683ebb83c * Cleanup
* Catch up to build ID directory changes

* Support a meta-hostname of 'all' for setting up all clients at once.
  This is better than the old way of running one copy of the script
  for each client by hand, since it is easier and involves less
  duplicated work.

* We copy in the per-build ports, src, and bindist .tbz files and .md5
  checksums, as well as refreshing the build scripts and
  bindist-$(hostname).tar customization tarball.

* The -force switch forces copying of files and re-extraction of the
  tarballs on the client.  This is necessary in order to propagate
  local changes to the tarballs after the initial client setup
  (e.g. if you need to change a file in the ports tree, it must be
  recompressed, redistributed, and re-extracted on the client).

* The -queue switch will poll the client's job queue after completion
  of the setup.  This is racy and should only be used when the machine
  is not currently accepting jobs.

* For cleaning up a build the 'build cleanup' command should now be
  used instead.  It calls back into this command but also allows full
  clenaup of build-local files on the client.

TODO: "all" setups are hard on the server since they may spawn dozens
of rsyncs at once.  A better solution would be to have a worker pool
of setup tasks to limit the maximum load.
2008-07-26 15:00:37 +00:00
Kris Kennaway
89c8fd897f * Cleanup
* Catch up to build ID directory changes
* Make it easier to kill a build by not running dopackages in the background
  where it is detached from shell job control.  Now, sending a termination
  signal to this process (e.g. ^C) will also kill off the dopackages script
  and in turn the processes created by it.  Some background processes
  spawned by dopackages, pdispatch, etc, may still remain and need to be
  killed by hand.
2008-07-26 14:52:05 +00:00
Kris Kennaway
4a7f6d83cb * Cleanup
* Catch up to build ID directory changes

* Improve usage()

* Fix a variety of small bugs

* Remove support for -ftp builds: we have not supported direct
  uploading for many years due to the desire to manually inspect build
  output for quality

* All data associated to a build is now localized in its own directory
  named according to a build ID:
  /var/portbuild/${arch}/${branch}/builds/${buildid}, where ${buildid}
  is the creation time.  These are actually ZFS filesystems.

* Tasks such as cloning a new build, updating a ZFS snapshot, and
  cleaning up a build are exported to the "build" script, which can be
  used independently.

* Creating a new build is done by ZFS cloning and takes a couple of
  seconds since it is copy-on-write (i.e. no data needs to be copied).

* Ports and source trees are also cloned from pre-updated ZFS images
  (updated regularly from the "updatesnap" cron job).  In most cases
  we do not care if we are building a ports tree that is an hour or so
  old since it will become outdated almost immediately anyway, so no
  matter what we do there will be times when a port has been fixed by
  the time the build error is generated by a client.

* In case an up-to-the-second tree is desired, the -portscvs and
  -srccvs switches update the existing ports tree via CVS.

* -noports and -nosrc can be used to prevent any automatic changes to
   the ports tree.  This is useful for dealing with local
   modifications (e.g. for -exp builds), since the default when
   creating a new build is to replace the previous trees with fresh,
   pristine trees.  If you forget to use this then any local changes
   that are not also present in other trees will be lost.

* By default we keep two builds for each arch/branch pair.  These
  build IDs also may be referred to via "latest" and "previous"
  symlinks.  When creating a new build, the old "previous" build is
  destroyed by default, unless it was originally created using the
  -keep switch.  This prevents the build from being destroyed
  automatically.

* By default when a build finishes all of the clients are completely
  cleaned up (i.e. all build data such as ports trees, tarballs,
  client chroots, etc are deleted).  This is needed to save space on
  the clients.  If you expect to *immediately* perform further builds
  after this one completes, the -nocleanup switch prevents this step.
  Otherwise they will just be set up again if further builds are
  scheduled.

* Try to parallelize build pre-processing as much as possible, by
  running jobs in the background wherever possible.  In several places
  we operate on the same parts of the filesystem from multiple jobs,
  so we can make good use of caching to improve performance

* Clients no longer need to be set up explicitly at the start of the
  build, they will be set up on-demand when the first job is
  dispatched to them.  This allows fast clients or those that already
  have been set up to begin building ports as soon as possible, while
  slow clients are set up in the background.  It also improves
  robustness of client recovery, e.g. if the client was offline at the
  time of build startup but later brought back online.

* Optimize copying back in the previous set of restricted packages by
  hardlinking instead of copying.

TODO: The record of failed ports is arch/branch-global still.  This is
the only thing preventing us from running concurrent builds of the
same arch/branch (e.g. while one is stuck building openoffice, the
next build can start to keep the cluster busy).  The difficulty is
that one build from a later ports tree may signal that a build was
successful, then a phase 2 build from an earlier ports tree may
indicate that it was broken.  The solution is probably to migrate this
to a real database instead of a flat file, and query it for the set of
broken ports as of a certain ports tree date.
2008-07-26 14:49:26 +00:00
Kris Kennaway
efe865a26c * Catch up to build ID directory changes
* Clients no longer mount ports/src trees via NFS (even the FreeBSD.org
  local clients).  This was putting too much load on the server and
  slowing down builds.

* Instead ports and src .tbz files are pushed to the clients and
  unpacked.  MD5 checksums are used to verify correctness

* -force forces re-extraction of the tarballs even if they exist and
  appear to be checked out

* Also unpack the compressed bindist

TODO: When we are not using md or ZFS builds it would be even faster
to keep an unpacked copy of the bindist on the scratch filesystem and
hardlink the files into the target directory
2008-07-26 14:19:31 +00:00
Kris Kennaway
b9dde2b9f8 * Catch up to build ID directory changes
* Optimize by copying old packages using cpio -dumpl (i.e. create hardlink
  instead of copying the file).
2008-07-26 14:14:35 +00:00
Kris Kennaway
07e904cab8 * Catch up to build ID directory changes 2008-07-26 14:13:35 +00:00
Kris Kennaway
1d5ba88d7a * Cleanup
* Catch up to build ID directory changes
* Remove need for /etc/arch file
2008-07-26 14:12:53 +00:00
Kris Kennaway
9f29c725dd * Cleanup
* Catch up to build ID directory changes
* Improved support for ZFS
* Desupport X11BASE
2008-07-26 14:12:28 +00:00
Kris Kennaway
f8a634d336 * Cleanup
* Catch up to build ID directory changes
* Improved support for ZFS builds
* Improved robustness
* Report status verbosely to the caller; whether we succeeded in claiming
  a chroot, whether the caller needs to first set up the client, or
  whether a setup is in progress.
* If we discover that the client has not been set up either because it
  freshly booted and newfs'ed its filesystem, or because a particular
  build has not yet been encountered, atomically claim a cookie and
  report this to the caller to act on
2008-07-26 14:11:26 +00:00
Kris Kennaway
316ad2a0a7 * Cleanup
* Catch up to build ID directory changes
2008-07-26 14:07:49 +00:00
Kris Kennaway
1dc6876bab * Cleanup
* Catch up to build ID directory changes
* Add helper functions for resolving a build ID symlink and
  validating an arch/branch combination (centralize instead of doing it
  in many scripts)
2008-07-26 14:06:30 +00:00
Kris Kennaway
5acb87ae92 * Desupport alpha and ia64
* Catch up to build ID directory changes
2008-07-26 14:05:01 +00:00
Kris Kennaway
46356ad8f8 * Add comments 2008-07-26 14:04:23 +00:00
Kris Kennaway
0b457b9cf0 * Implement basename and dirname using shell builtins 2008-07-26 14:02:55 +00:00
Kris Kennaway
f204e78013 * Cleanup
* Catch up to build ID directory changes
* Record package build completion for reporting to ganglia
2008-07-26 14:02:38 +00:00