mirror of
https://git.FreeBSD.org/ports.git
synced 2024-10-25 21:07:40 +00:00
31fb70c70d
Make INDEX build failure a fatal error.
Copy INDEX to INDEX-${branch} for later perusal.
There is no -a flag in OpenSSH's scp.
Use sed to remove /usr/ports and stuff from restricted.sh and
cdrom.sh, so they can be run without further modification using
relative directories.
Take more options, namely -norestr (don't build RESTRICTED ports),
-noplistcheck (turn off "extra files check"), and -nodummy (use
open-motif instead of Motif-dummy and XFree86-4-* instead of
XFree86-3). -nodummy will become the default soon. Also a new flag
-nocvs will prevent cvs update on ${branch}/src only (while -nocvsup
will prevent cvsup and cvs update on usr/ports). -nocvs is to be used
when builds for more than one branch shares a same cvsup session.
Write out time cvsup is done so it can be used by log processing
scripts.
Handle dummy ports by using a more generic approach -- list the
directories they reside in (e.g., x11-toolkits/Motif-dummy) and
generate the package name from there, instead of hardcoding them
package names everywhere. Hopefully this will go away when we get rid
of the special case handling for Motif and XFree86.
Add something called "quickports" -- these are packages with very long
dependency lists that always seem to throttle the parallel build near
the end. By spelling them out explicitly, try to make make build
dependencies for them earlier while there are a lot of other stuff to
do. (Our make doesn't do anything with the target list, but maybe I
can fix that someday.)
Copy bindist.tar to clients before the second run of package build
too. It can be useful to "sneak" in fixes if something minorly wrong
is discovered in the bindist during the first run. Since there is a
md5 checking, this copying will not be done unless the files are
actually different.
When -norestr is set, delete restricted packages and distfiles using
restricted.sh immediately after completion of package build.
Generate CHECKSUM.MD5 after packages are built.
Copy distfiles over to the ftp site after everything is done. This is
run in the background on bak/distfiles so the next package build can
immediately start.
|
||
|---|---|---|
| .. | ||
| 2.2 | ||
| 3 | ||
| 4 | ||
| scripts | ||
| mlist | ||
| README | ||
This is the parallelized package building system. Many thanks to
Steve Price for helping me put this together.
- Satoshi
In the following, ${branch} is either "2.2" or "3.0" depending on
which packages you intend to build.
Note that this system is still under development. It will require a
substantial amount of effort to set up initially, and the following is
probably missing a lot of stuff. Please let me know of any warts you
find.
(1) One of the machines are to be designated as the "master". It is
defined in the portbuild script. There is also a "buildroot"
directory where everything should reside on the master machine.
It is defined in ports/Makefile and the portbuild script. Copy
everything under this directory there and make necessary changes.
(2) On the master, check out the ports tree under
${buildroot}/usr/ports, the appropriate source tree under
${buildroot}/${branch}/src and the doc tree under
${buildroot}/usr/opt/doc. These directories have to be NFS
exported to the build machines. (Hint: you should probably add
"-alldirs" to /etc/exports.)
(3) Setup ssh for root between the build machines and the master. It
has to work in both directions.
(4) Put the list of machines you have in "mlist". There is a sample
provided in this directory. Each line should have two entries,
"hostname" and "power". The "hostname" is self-explanatory; the
"power" entry generally discribes how many compilations that
machine can run in parallel. It is only relative, so designating
two machines 1 and 2 is the same as having them as 10 and 20
(modulo any rounding errors).
Note that you can put the master here as well, but our experience
is that our NFS is likely to act up when the master is too busy.
Even with a two-processor P6-200, we had to refrain from running
any compilations on the master to get NFS to work reliably.
(5) Go to ${branch}/bindist and follow the instructions in the README
files there to create the bindist tarball.
(6) Generate an XFree86 tarball in ports/x11/XFree86. Use
PLIST.stripped to create a smaller version. (You can use it by
setting 'PLIST=${PKGDIR}/PLIST.stripped'.) Put it in
${branch}/tarballs.
(7) If you have Motif, generate a Motif tarball by using
ports/x11-toolkits/Motif-dummy. Put it in ${branch}/tarballs.
(8) Copy ${branch}/tarballs and ${buildroot}/portbuild to same
directories on the build machines. Create a directory
${branch}/chroot on the build machines.
(9) Create the directory ${buildroot}/distfiles, ${branch}/{errors,logs}
and ${branch}/packages/All on the master. Copy the XFree86 and
Motif tarballs to the latter directory.
(10) Run the "reportload" script on the build machines in the
background (you can put reportload.sh in /usr/local/etc/rc.d to
run it automatically upon reboot), and then run the
"checkmachines" script on the master, again in the background.
This will check all the machines listed in "mlist" periodically
and print the list of least-loaded machines to "ulist".
(11) Run the "makeduds" script at ${buildroot}/usr/ports. The one
supplied is for building 2.2 packages on a 3.0 machines. For
3.0, you can replace it with just two lines:
unset DISPLAY
PARALLEL_PACKAGE_BUILD=t make ignorelist ECHO_MSG=true > ../../3.0/duds
(12) Run the "makeindex" script at ${buildroot}/usr/ports. The one
supplied is for building 2.2 packages on a 3.0 machines. For
3.0, you can replace it with just one line:
PORTSDIR=$(pwd) PARALLEL_PACKAGE_BUILD=t make index
(13) Run "make parallel > ../../${branch}/Makefile" in
${buildroot}/usr/ports to generate the Makefile.
(14) Go to ${branch}/packages/All on the master and type "make -k
-j<whatever> -f ../../Makefile &". Then wait. :)
(15) Note that the new scheme will not clean up the chroot dirs under
${buildroot}/${branch}/chroot on the build machines, but will
instead reuse them during the course of the build. Make sure you
delete them all when your bindist.tar changes.