2005-11-19 20:26:52 +00:00
|
|
|
.\" Copyright (c) 2005 Chris Jones
|
|
|
|
.\" All rights reserved.
|
2005-11-20 10:35:46 +00:00
|
|
|
.\"
|
2005-11-19 20:26:52 +00:00
|
|
|
.\" This software was developed for the FreeBSD Project by Chris Jones
|
|
|
|
.\" thanks to the support of Google's Summer of Code program and
|
|
|
|
.\" mentoring by Lukas Ertl.
|
2005-11-20 10:35:46 +00:00
|
|
|
.\"
|
|
|
|
.\" Redistribution and use in source and binary forms, with or without
|
|
|
|
.\" modification, are permitted provided that the following conditions
|
|
|
|
.\" are met:
|
2005-11-19 20:26:52 +00:00
|
|
|
.\" 1. Redistributions of source code must retain the above copyright
|
|
|
|
.\" notice, this list of conditions and the following disclaimer.
|
|
|
|
.\" 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
.\" notice, this list of conditions and the following disclaimer in the
|
|
|
|
.\" documentation and/or other materials provided with the distribution.
|
|
|
|
.\"
|
|
|
|
.\" THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
|
|
|
|
.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
.\" ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
|
|
|
|
.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
.\" SUCH DAMAGE.
|
2005-11-20 10:35:46 +00:00
|
|
|
.\"
|
2005-11-19 20:26:52 +00:00
|
|
|
.\" $FreeBSD$
|
|
|
|
.\"
|
2006-09-30 11:02:17 +00:00
|
|
|
.Dd March 23, 2006
|
2005-11-19 20:26:52 +00:00
|
|
|
.Dt GVINUM 8
|
|
|
|
.Os
|
|
|
|
.Sh NAME
|
|
|
|
.Nm gvinum
|
|
|
|
.Nd Logical Volume Manager control program
|
|
|
|
.Sh SYNOPSIS
|
|
|
|
.Nm
|
|
|
|
.Op Ar command
|
|
|
|
.Op Fl options
|
|
|
|
.Sh COMMANDS
|
|
|
|
.Bl -tag -width indent
|
Import the gvinum work that have been done during and after Summer of Code 2007.
The work have been under testing and fixing since then, and it is mature enough
to be put into HEAD for further testing.
A lot have changed in this time, and here are the most important:
- Gvinum now uses one single workerthread instead of one thread for each
volume and each plex. The reason for this is that the previous scheme was
very complex, and was the cause of many of the bugs discovered in gvinum.
Instead, gvinum now uses one worker thread with an event queue, quite
similar to what used in gmirror.
- The rebuild/grow/initialize/parity check routines no longer runs in
separate threads, but are run as regular I/O requests with special flags.
This made it easier to support mounted growing and parity rebuild.
- Support for growing striped and raid5-plexes, meaning that one can extend the
volumes for these plex types in addition to the concat type. Also works while
the volume is mounted.
- Implementation of many of the missing commands from the old vinum:
attach/detach, start (was partially implemented), stop (was partially
implemented), concat, mirror, stripe, raid5 (shortcuts for creating volumes
with one plex of these organizations).
- The parity check and rebuild no longer goes between userland/kernel, meaning
that the gvinum command will not stay and wait forever for the rebuild to
finish. You can instead watch the status with the list command.
- Many problems with gvinum have been reported since 5.x, and some has been hard
to fix due to the complicated architecture. Hopefully, it should be more
stable and better handle edge cases that previously made gvinum crash.
- Failed drives no longer disappears entirely, but now leave behind a dummy
drive that makes sure the original state is not forgotten in case the system
is rebooted between drive failures/swaps.
- Update manpage to reflect new commands and extend it with some examples.
Sponsored by: Google Summer of Code 2007
Mentored by: le
Tested by: Rick C. Petty <rick-freebsd2008 -at- kiwi-computer.com>
2009-03-28 17:20:08 +00:00
|
|
|
.It Ic attach Ar plex volume Op Cm rename
|
|
|
|
.It Ic attach Ar subdisk plex Oo Ar offset Oc Op Cm rename
|
|
|
|
Attach a plex to a volume, or a subdisk to a plex.
|
|
|
|
If offset is specified, the subdisk will be attached to the given offset within
|
|
|
|
the plex.
|
|
|
|
If rename is specified, the subdisk or plex will change name according to the
|
|
|
|
object it attaches to.
|
2006-09-30 11:02:17 +00:00
|
|
|
.It Ic checkparity Oo Fl f Oc Ar plex
|
|
|
|
Check the parity blocks of a RAID-5 plex.
|
|
|
|
The parity check will start at the
|
2005-11-20 10:35:46 +00:00
|
|
|
beginning of the plex if the
|
2005-11-19 20:26:52 +00:00
|
|
|
.Fl f
|
|
|
|
flag is specified, or otherwise at the location of the parity check pointer,
|
2006-09-30 11:02:17 +00:00
|
|
|
the first location at which plex's parity is incorrect.
|
|
|
|
All subdisks in the
|
2005-11-19 20:26:52 +00:00
|
|
|
plex must be up for a parity check.
|
Import the gvinum work that have been done during and after Summer of Code 2007.
The work have been under testing and fixing since then, and it is mature enough
to be put into HEAD for further testing.
A lot have changed in this time, and here are the most important:
- Gvinum now uses one single workerthread instead of one thread for each
volume and each plex. The reason for this is that the previous scheme was
very complex, and was the cause of many of the bugs discovered in gvinum.
Instead, gvinum now uses one worker thread with an event queue, quite
similar to what used in gmirror.
- The rebuild/grow/initialize/parity check routines no longer runs in
separate threads, but are run as regular I/O requests with special flags.
This made it easier to support mounted growing and parity rebuild.
- Support for growing striped and raid5-plexes, meaning that one can extend the
volumes for these plex types in addition to the concat type. Also works while
the volume is mounted.
- Implementation of many of the missing commands from the old vinum:
attach/detach, start (was partially implemented), stop (was partially
implemented), concat, mirror, stripe, raid5 (shortcuts for creating volumes
with one plex of these organizations).
- The parity check and rebuild no longer goes between userland/kernel, meaning
that the gvinum command will not stay and wait forever for the rebuild to
finish. You can instead watch the status with the list command.
- Many problems with gvinum have been reported since 5.x, and some has been hard
to fix due to the complicated architecture. Hopefully, it should be more
stable and better handle edge cases that previously made gvinum crash.
- Failed drives no longer disappears entirely, but now leave behind a dummy
drive that makes sure the original state is not forgotten in case the system
is rebooted between drive failures/swaps.
- Update manpage to reflect new commands and extend it with some examples.
Sponsored by: Google Summer of Code 2007
Mentored by: le
Tested by: Rick C. Petty <rick-freebsd2008 -at- kiwi-computer.com>
2009-03-28 17:20:08 +00:00
|
|
|
.It Ic concat Oo Fl fv Oc Oo Fl n Ar name Oc Ar drives
|
|
|
|
Create a concatenated volume from the specified drives.
|
|
|
|
If no name is specified, a unique name will be set by gvinum.
|
|
|
|
.It Ic create Oo Fl f Oc Op Ar description-file
|
2005-11-19 20:26:52 +00:00
|
|
|
Create a volume as described in
|
|
|
|
.Ar description-file .
|
|
|
|
If no
|
2005-11-20 10:35:46 +00:00
|
|
|
.Ar description-file
|
|
|
|
provided, opens an editor and provides the current
|
2005-11-19 20:26:52 +00:00
|
|
|
.Nm
|
|
|
|
configuration for editing.
|
Import the gvinum work that have been done during and after Summer of Code 2007.
The work have been under testing and fixing since then, and it is mature enough
to be put into HEAD for further testing.
A lot have changed in this time, and here are the most important:
- Gvinum now uses one single workerthread instead of one thread for each
volume and each plex. The reason for this is that the previous scheme was
very complex, and was the cause of many of the bugs discovered in gvinum.
Instead, gvinum now uses one worker thread with an event queue, quite
similar to what used in gmirror.
- The rebuild/grow/initialize/parity check routines no longer runs in
separate threads, but are run as regular I/O requests with special flags.
This made it easier to support mounted growing and parity rebuild.
- Support for growing striped and raid5-plexes, meaning that one can extend the
volumes for these plex types in addition to the concat type. Also works while
the volume is mounted.
- Implementation of many of the missing commands from the old vinum:
attach/detach, start (was partially implemented), stop (was partially
implemented), concat, mirror, stripe, raid5 (shortcuts for creating volumes
with one plex of these organizations).
- The parity check and rebuild no longer goes between userland/kernel, meaning
that the gvinum command will not stay and wait forever for the rebuild to
finish. You can instead watch the status with the list command.
- Many problems with gvinum have been reported since 5.x, and some has been hard
to fix due to the complicated architecture. Hopefully, it should be more
stable and better handle edge cases that previously made gvinum crash.
- Failed drives no longer disappears entirely, but now leave behind a dummy
drive that makes sure the original state is not forgotten in case the system
is rebooted between drive failures/swaps.
- Update manpage to reflect new commands and extend it with some examples.
Sponsored by: Google Summer of Code 2007
Mentored by: le
Tested by: Rick C. Petty <rick-freebsd2008 -at- kiwi-computer.com>
2009-03-28 17:20:08 +00:00
|
|
|
The
|
|
|
|
.Fl f
|
|
|
|
flag will make gvinum ignore any errors regarding creating objects that already
|
|
|
|
exists.
|
|
|
|
However, in contrast to vinum, objects that are not properly named in the
|
|
|
|
.Ar description-file
|
|
|
|
will not be created when the
|
|
|
|
.Fl f
|
|
|
|
flag is given.
|
|
|
|
.It Ic detach Oo Fl f Oc Op Ar plex | subdisk
|
|
|
|
Detach a plex or subdisk from the volume or plex to which it is
|
|
|
|
attached.
|
2006-09-30 11:02:17 +00:00
|
|
|
.It Ic help
|
2005-11-20 10:35:46 +00:00
|
|
|
Provides a synopsis of
|
|
|
|
.Nm
|
2005-11-19 20:26:52 +00:00
|
|
|
commands and arguments.
|
2006-09-30 11:02:17 +00:00
|
|
|
.It Ic l | list Oo Fl rvV Oc Op Ar volume | plex | subdisk
|
|
|
|
.It Ic ld Oo Fl rvV Oc Op Ar drive ...
|
|
|
|
.It Ic ls Oo Fl rvV Oc Op Ar subdisk ...
|
|
|
|
.It Ic lp Oo Fl rvV Oc Op Ar plex ...
|
|
|
|
.It Ic lv Oo Fl rvV Oc Op Ar volume ...
|
|
|
|
List information about the relevant object(s).
|
|
|
|
The
|
2005-11-19 20:26:52 +00:00
|
|
|
.Fl r
|
|
|
|
flag provides recursive display, showing each object's subordinate objects in
|
2006-09-30 11:02:17 +00:00
|
|
|
proper relation.
|
|
|
|
The
|
2005-11-19 20:26:52 +00:00
|
|
|
.Fl v
|
2005-11-20 10:35:46 +00:00
|
|
|
and
|
2005-11-19 20:26:52 +00:00
|
|
|
.Fl V
|
|
|
|
flags provide progressively more detailed output.
|
Import the gvinum work that have been done during and after Summer of Code 2007.
The work have been under testing and fixing since then, and it is mature enough
to be put into HEAD for further testing.
A lot have changed in this time, and here are the most important:
- Gvinum now uses one single workerthread instead of one thread for each
volume and each plex. The reason for this is that the previous scheme was
very complex, and was the cause of many of the bugs discovered in gvinum.
Instead, gvinum now uses one worker thread with an event queue, quite
similar to what used in gmirror.
- The rebuild/grow/initialize/parity check routines no longer runs in
separate threads, but are run as regular I/O requests with special flags.
This made it easier to support mounted growing and parity rebuild.
- Support for growing striped and raid5-plexes, meaning that one can extend the
volumes for these plex types in addition to the concat type. Also works while
the volume is mounted.
- Implementation of many of the missing commands from the old vinum:
attach/detach, start (was partially implemented), stop (was partially
implemented), concat, mirror, stripe, raid5 (shortcuts for creating volumes
with one plex of these organizations).
- The parity check and rebuild no longer goes between userland/kernel, meaning
that the gvinum command will not stay and wait forever for the rebuild to
finish. You can instead watch the status with the list command.
- Many problems with gvinum have been reported since 5.x, and some has been hard
to fix due to the complicated architecture. Hopefully, it should be more
stable and better handle edge cases that previously made gvinum crash.
- Failed drives no longer disappears entirely, but now leave behind a dummy
drive that makes sure the original state is not forgotten in case the system
is rebooted between drive failures/swaps.
- Update manpage to reflect new commands and extend it with some examples.
Sponsored by: Google Summer of Code 2007
Mentored by: le
Tested by: Rick C. Petty <rick-freebsd2008 -at- kiwi-computer.com>
2009-03-28 17:20:08 +00:00
|
|
|
.It Ic mirror Oo Fl fsv Oc Oo Fl n Ar name Oc Ar drives
|
|
|
|
Create a mirrored volume from the specified drives.
|
|
|
|
It requires at least a multiple of 2 drives.
|
|
|
|
If no name is specified, a unique name will be set by gvinum.
|
|
|
|
If the
|
|
|
|
.Fl s
|
|
|
|
flag is specified, a striped mirror will be created, and thus requires a
|
|
|
|
multiple of 4 drives.
|
2006-09-30 11:02:17 +00:00
|
|
|
.It Ic move | mv Fl f Ar drive subdisk Op Ar ...
|
|
|
|
Move the subdisk(s) to the specified drive.
|
|
|
|
The
|
2005-11-20 10:35:46 +00:00
|
|
|
.Fl f
|
2005-11-19 20:26:52 +00:00
|
|
|
flag is required, as all data on the indicated subdisk(s) will be destroyed as
|
2006-09-30 11:02:17 +00:00
|
|
|
part of the move.
|
|
|
|
This can currently only be done when the subdisk is
|
2005-11-19 20:26:52 +00:00
|
|
|
not being accessed.
|
|
|
|
.Pp
|
Import the gvinum work that have been done during and after Summer of Code 2007.
The work have been under testing and fixing since then, and it is mature enough
to be put into HEAD for further testing.
A lot have changed in this time, and here are the most important:
- Gvinum now uses one single workerthread instead of one thread for each
volume and each plex. The reason for this is that the previous scheme was
very complex, and was the cause of many of the bugs discovered in gvinum.
Instead, gvinum now uses one worker thread with an event queue, quite
similar to what used in gmirror.
- The rebuild/grow/initialize/parity check routines no longer runs in
separate threads, but are run as regular I/O requests with special flags.
This made it easier to support mounted growing and parity rebuild.
- Support for growing striped and raid5-plexes, meaning that one can extend the
volumes for these plex types in addition to the concat type. Also works while
the volume is mounted.
- Implementation of many of the missing commands from the old vinum:
attach/detach, start (was partially implemented), stop (was partially
implemented), concat, mirror, stripe, raid5 (shortcuts for creating volumes
with one plex of these organizations).
- The parity check and rebuild no longer goes between userland/kernel, meaning
that the gvinum command will not stay and wait forever for the rebuild to
finish. You can instead watch the status with the list command.
- Many problems with gvinum have been reported since 5.x, and some has been hard
to fix due to the complicated architecture. Hopefully, it should be more
stable and better handle edge cases that previously made gvinum crash.
- Failed drives no longer disappears entirely, but now leave behind a dummy
drive that makes sure the original state is not forgotten in case the system
is rebooted between drive failures/swaps.
- Update manpage to reflect new commands and extend it with some examples.
Sponsored by: Google Summer of Code 2007
Mentored by: le
Tested by: Rick C. Petty <rick-freebsd2008 -at- kiwi-computer.com>
2009-03-28 17:20:08 +00:00
|
|
|
If a single subdisk is moved, and it forms a part of a RAID-5 plex, the moved
|
|
|
|
subdisks will need to be set to the
|
|
|
|
.Dq stale
|
|
|
|
state, and the plex will require a
|
|
|
|
.Ic start
|
|
|
|
command.
|
|
|
|
If multiple subdisk(s) is moved, and form part of a RAID-5 plex, the
|
|
|
|
moved disk(s) will need to be set to the
|
2006-09-30 11:02:17 +00:00
|
|
|
.Dq up
|
|
|
|
state and the plex will require a
|
2005-11-19 20:26:52 +00:00
|
|
|
.Ic rebuildparity
|
Import the gvinum work that have been done during and after Summer of Code 2007.
The work have been under testing and fixing since then, and it is mature enough
to be put into HEAD for further testing.
A lot have changed in this time, and here are the most important:
- Gvinum now uses one single workerthread instead of one thread for each
volume and each plex. The reason for this is that the previous scheme was
very complex, and was the cause of many of the bugs discovered in gvinum.
Instead, gvinum now uses one worker thread with an event queue, quite
similar to what used in gmirror.
- The rebuild/grow/initialize/parity check routines no longer runs in
separate threads, but are run as regular I/O requests with special flags.
This made it easier to support mounted growing and parity rebuild.
- Support for growing striped and raid5-plexes, meaning that one can extend the
volumes for these plex types in addition to the concat type. Also works while
the volume is mounted.
- Implementation of many of the missing commands from the old vinum:
attach/detach, start (was partially implemented), stop (was partially
implemented), concat, mirror, stripe, raid5 (shortcuts for creating volumes
with one plex of these organizations).
- The parity check and rebuild no longer goes between userland/kernel, meaning
that the gvinum command will not stay and wait forever for the rebuild to
finish. You can instead watch the status with the list command.
- Many problems with gvinum have been reported since 5.x, and some has been hard
to fix due to the complicated architecture. Hopefully, it should be more
stable and better handle edge cases that previously made gvinum crash.
- Failed drives no longer disappears entirely, but now leave behind a dummy
drive that makes sure the original state is not forgotten in case the system
is rebooted between drive failures/swaps.
- Update manpage to reflect new commands and extend it with some examples.
Sponsored by: Google Summer of Code 2007
Mentored by: le
Tested by: Rick C. Petty <rick-freebsd2008 -at- kiwi-computer.com>
2009-03-28 17:20:08 +00:00
|
|
|
command.
|
|
|
|
If the subdisk(s) form part of a plex that is mirrored with other
|
2006-09-30 11:02:17 +00:00
|
|
|
plexes, the plex will require restarting and will sync once restarted.
|
|
|
|
Moving
|
2005-11-19 20:26:52 +00:00
|
|
|
more than one subdisk in a RAID-5 plex or subdisks from both sides of a
|
2006-09-30 11:02:17 +00:00
|
|
|
mirrored plex volume will destroy data.
|
|
|
|
Note that parity rebuilds and syncing
|
2005-11-19 20:26:52 +00:00
|
|
|
must be started manually after a move.
|
2006-09-30 11:02:17 +00:00
|
|
|
.It Ic printconfig
|
2005-11-19 20:26:52 +00:00
|
|
|
Write a copy of the current configuration to standard output.
|
2006-09-30 11:02:17 +00:00
|
|
|
.It Ic quit
|
2005-11-20 10:35:46 +00:00
|
|
|
Exit
|
2005-11-19 20:26:52 +00:00
|
|
|
.Nm
|
2006-09-30 11:02:17 +00:00
|
|
|
when running in interactive mode.
|
|
|
|
Normally this would be done by entering the
|
2005-11-19 20:26:52 +00:00
|
|
|
EOF character.
|
Import the gvinum work that have been done during and after Summer of Code 2007.
The work have been under testing and fixing since then, and it is mature enough
to be put into HEAD for further testing.
A lot have changed in this time, and here are the most important:
- Gvinum now uses one single workerthread instead of one thread for each
volume and each plex. The reason for this is that the previous scheme was
very complex, and was the cause of many of the bugs discovered in gvinum.
Instead, gvinum now uses one worker thread with an event queue, quite
similar to what used in gmirror.
- The rebuild/grow/initialize/parity check routines no longer runs in
separate threads, but are run as regular I/O requests with special flags.
This made it easier to support mounted growing and parity rebuild.
- Support for growing striped and raid5-plexes, meaning that one can extend the
volumes for these plex types in addition to the concat type. Also works while
the volume is mounted.
- Implementation of many of the missing commands from the old vinum:
attach/detach, start (was partially implemented), stop (was partially
implemented), concat, mirror, stripe, raid5 (shortcuts for creating volumes
with one plex of these organizations).
- The parity check and rebuild no longer goes between userland/kernel, meaning
that the gvinum command will not stay and wait forever for the rebuild to
finish. You can instead watch the status with the list command.
- Many problems with gvinum have been reported since 5.x, and some has been hard
to fix due to the complicated architecture. Hopefully, it should be more
stable and better handle edge cases that previously made gvinum crash.
- Failed drives no longer disappears entirely, but now leave behind a dummy
drive that makes sure the original state is not forgotten in case the system
is rebooted between drive failures/swaps.
- Update manpage to reflect new commands and extend it with some examples.
Sponsored by: Google Summer of Code 2007
Mentored by: le
Tested by: Rick C. Petty <rick-freebsd2008 -at- kiwi-computer.com>
2009-03-28 17:20:08 +00:00
|
|
|
.It Ic raid5 Oo Fl fv Oc Oo Fl s Ar stripesize Oc Oo Fl n Ar name Oc Ar drives
|
|
|
|
Create a RAID-5 volume from the specified drives.
|
|
|
|
If no name is specified,a unique name will be set by
|
|
|
|
.Ic gvinum.
|
|
|
|
This organization requires at least three drives.
|
2006-09-30 11:02:17 +00:00
|
|
|
.It Ic rename Oo Fl r Oc Ar drive | subdisk | plex | volume newname
|
|
|
|
Change the name of the specified object.
|
|
|
|
The
|
2005-11-19 20:26:52 +00:00
|
|
|
.Fl r
|
|
|
|
flag will recursively rename subordinate objects.
|
|
|
|
.Pp
|
2005-11-20 10:35:46 +00:00
|
|
|
Note that device nodes will not be renamed until
|
2005-11-19 20:26:52 +00:00
|
|
|
.Nm
|
|
|
|
is restarted.
|
2006-09-30 11:02:17 +00:00
|
|
|
.It Ic rebuildparity Oo Fl f Oc Ar plex
|
|
|
|
Rebuild the parity blocks of a RAID-5 plex.
|
|
|
|
The parity rebuild will start at
|
2005-11-20 10:35:46 +00:00
|
|
|
the beginning of the plex if the
|
2005-11-19 20:26:52 +00:00
|
|
|
.Fl f
|
|
|
|
flag is specified, or otherwise at the location of the parity check pointer.
|
|
|
|
All subdisks in the plex must be up for a parity check.
|
2006-03-23 19:58:43 +00:00
|
|
|
.It Ic resetconfig
|
|
|
|
Reset the complete
|
|
|
|
.Nm
|
|
|
|
configuration.
|
2006-09-30 11:02:17 +00:00
|
|
|
.It Ic rm Oo Fl r Oc Ar volume | plex | subdisk
|
2005-11-20 10:35:46 +00:00
|
|
|
Remove an object and, if
|
2005-11-19 20:26:52 +00:00
|
|
|
.Fl r
|
|
|
|
is specified, its subordinate objects.
|
2006-09-30 11:02:17 +00:00
|
|
|
.It Ic saveconfig
|
2005-11-20 10:35:46 +00:00
|
|
|
Save
|
2005-11-19 20:26:52 +00:00
|
|
|
.Nm
|
|
|
|
configuration to disk after configuration failures.
|
2006-09-30 11:02:17 +00:00
|
|
|
.It Ic setstate Oo Fl f Oc Ar state volume | plex | subdisk | drive
|
2005-11-20 10:35:46 +00:00
|
|
|
Set state without influencing other objects, for diagnostic purposes
|
2006-09-30 11:02:17 +00:00
|
|
|
only.
|
|
|
|
The
|
2005-11-19 20:26:52 +00:00
|
|
|
.Fl f
|
|
|
|
flag forces state changes regardless of whether they are legal.
|
2006-09-30 11:02:17 +00:00
|
|
|
.It Ic start
|
2005-11-19 20:26:52 +00:00
|
|
|
Read configuration from all vinum drives.
|
2006-09-30 11:02:17 +00:00
|
|
|
.It Ic start Oo Fl S Ar size Oc Ar volume | plex | subdisk
|
|
|
|
Allow the system to access the objects.
|
Import the gvinum work that have been done during and after Summer of Code 2007.
The work have been under testing and fixing since then, and it is mature enough
to be put into HEAD for further testing.
A lot have changed in this time, and here are the most important:
- Gvinum now uses one single workerthread instead of one thread for each
volume and each plex. The reason for this is that the previous scheme was
very complex, and was the cause of many of the bugs discovered in gvinum.
Instead, gvinum now uses one worker thread with an event queue, quite
similar to what used in gmirror.
- The rebuild/grow/initialize/parity check routines no longer runs in
separate threads, but are run as regular I/O requests with special flags.
This made it easier to support mounted growing and parity rebuild.
- Support for growing striped and raid5-plexes, meaning that one can extend the
volumes for these plex types in addition to the concat type. Also works while
the volume is mounted.
- Implementation of many of the missing commands from the old vinum:
attach/detach, start (was partially implemented), stop (was partially
implemented), concat, mirror, stripe, raid5 (shortcuts for creating volumes
with one plex of these organizations).
- The parity check and rebuild no longer goes between userland/kernel, meaning
that the gvinum command will not stay and wait forever for the rebuild to
finish. You can instead watch the status with the list command.
- Many problems with gvinum have been reported since 5.x, and some has been hard
to fix due to the complicated architecture. Hopefully, it should be more
stable and better handle edge cases that previously made gvinum crash.
- Failed drives no longer disappears entirely, but now leave behind a dummy
drive that makes sure the original state is not forgotten in case the system
is rebooted between drive failures/swaps.
- Update manpage to reflect new commands and extend it with some examples.
Sponsored by: Google Summer of Code 2007
Mentored by: le
Tested by: Rick C. Petty <rick-freebsd2008 -at- kiwi-computer.com>
2009-03-28 17:20:08 +00:00
|
|
|
If necessary, plexes will be synced and rebuilt.
|
|
|
|
If a subdisk was added to a running RAID-5 or striped plex, gvinum will
|
|
|
|
expand into this subdisk and grow the whole RAID-5 array.
|
|
|
|
This can be done without unmounting your filesystem.
|
2006-09-30 11:02:17 +00:00
|
|
|
The
|
2005-11-19 20:26:52 +00:00
|
|
|
.Fl S
|
|
|
|
flag is currently ignored.
|
Import the gvinum work that have been done during and after Summer of Code 2007.
The work have been under testing and fixing since then, and it is mature enough
to be put into HEAD for further testing.
A lot have changed in this time, and here are the most important:
- Gvinum now uses one single workerthread instead of one thread for each
volume and each plex. The reason for this is that the previous scheme was
very complex, and was the cause of many of the bugs discovered in gvinum.
Instead, gvinum now uses one worker thread with an event queue, quite
similar to what used in gmirror.
- The rebuild/grow/initialize/parity check routines no longer runs in
separate threads, but are run as regular I/O requests with special flags.
This made it easier to support mounted growing and parity rebuild.
- Support for growing striped and raid5-plexes, meaning that one can extend the
volumes for these plex types in addition to the concat type. Also works while
the volume is mounted.
- Implementation of many of the missing commands from the old vinum:
attach/detach, start (was partially implemented), stop (was partially
implemented), concat, mirror, stripe, raid5 (shortcuts for creating volumes
with one plex of these organizations).
- The parity check and rebuild no longer goes between userland/kernel, meaning
that the gvinum command will not stay and wait forever for the rebuild to
finish. You can instead watch the status with the list command.
- Many problems with gvinum have been reported since 5.x, and some has been hard
to fix due to the complicated architecture. Hopefully, it should be more
stable and better handle edge cases that previously made gvinum crash.
- Failed drives no longer disappears entirely, but now leave behind a dummy
drive that makes sure the original state is not forgotten in case the system
is rebooted between drive failures/swaps.
- Update manpage to reflect new commands and extend it with some examples.
Sponsored by: Google Summer of Code 2007
Mentored by: le
Tested by: Rick C. Petty <rick-freebsd2008 -at- kiwi-computer.com>
2009-03-28 17:20:08 +00:00
|
|
|
.It Ic stop Oo Fl f Oc Op Ar volume | plex | subdisk
|
|
|
|
Terminate access to the objects, or stop
|
|
|
|
.Nm
|
|
|
|
if no parameters are specified.
|
|
|
|
.It Ic stripe Oo Fl fv Oc Oo Fl n Ar name Oc Ar drives
|
|
|
|
Create a striped volume from the specified drives. If no name is specified,
|
|
|
|
a unique name will be set by Ic gvinum. This organization requires at least two
|
|
|
|
drives.
|
2005-11-19 20:26:52 +00:00
|
|
|
.El
|
|
|
|
.Sh DESCRIPTION
|
|
|
|
The
|
|
|
|
.Nm
|
|
|
|
utility communicates with the kernel component of the GVinum logical volume
|
2006-09-30 11:02:17 +00:00
|
|
|
manager.
|
|
|
|
It is designed either for interactive use, when started without
|
2005-11-19 20:26:52 +00:00
|
|
|
command line arguments, or to execute a single command if the command is
|
2006-09-30 11:02:17 +00:00
|
|
|
supplied on the command line.
|
|
|
|
In interactive mode,
|
2005-11-19 20:26:52 +00:00
|
|
|
.Nm
|
|
|
|
maintains a command line history.
|
|
|
|
.Sh OPTIONS
|
2006-09-30 11:02:17 +00:00
|
|
|
The
|
2005-11-19 20:26:52 +00:00
|
|
|
.Nm
|
|
|
|
commands may be followed by an option.
|
|
|
|
.Bl -tag -width indent
|
|
|
|
.It Fl f
|
|
|
|
The
|
|
|
|
.Fl f
|
|
|
|
.Pq Dq force
|
2006-09-30 11:02:17 +00:00
|
|
|
option overrides safety checks.
|
|
|
|
It should be used with extreme caution.
|
|
|
|
This
|
2005-11-20 10:35:46 +00:00
|
|
|
option is required in order to use the
|
2005-11-19 20:26:52 +00:00
|
|
|
.Ic move
|
|
|
|
command.
|
|
|
|
.It Fl r
|
2005-11-20 10:35:46 +00:00
|
|
|
The
|
2005-11-19 20:26:52 +00:00
|
|
|
.Fl r
|
|
|
|
.Pq Dq recursive
|
2006-09-30 11:02:17 +00:00
|
|
|
option applies the command recursively to subordinate objects.
|
|
|
|
For example, in
|
2005-11-19 20:26:52 +00:00
|
|
|
conjunction with the
|
|
|
|
.Ic lv
|
2005-11-20 10:35:46 +00:00
|
|
|
command, the
|
2005-11-19 20:26:52 +00:00
|
|
|
.Fl r
|
|
|
|
option will also show information about the plexes and subdisks belonging to
|
2006-09-30 11:02:17 +00:00
|
|
|
the volume.
|
|
|
|
It is also used by the
|
2005-11-19 20:26:52 +00:00
|
|
|
.Ic rename
|
|
|
|
command to indicate that subordinate objects such as subdisks should be renamed
|
2005-11-20 10:35:46 +00:00
|
|
|
to match the object(s) specified and by the
|
2005-11-19 20:26:52 +00:00
|
|
|
.Ic rm
|
|
|
|
command to delete plexes belonging to a volume and so on.
|
|
|
|
.It Fl v
|
|
|
|
The
|
|
|
|
.Fl v
|
|
|
|
.Pq Dq verbose
|
|
|
|
option provides more detailed output.
|
|
|
|
.It Fl V
|
|
|
|
The
|
|
|
|
.Fl V
|
2006-09-30 11:02:17 +00:00
|
|
|
.Pq Dq "very verbose"
|
2005-11-19 20:26:52 +00:00
|
|
|
option provides even more detailed output than
|
|
|
|
.Fl v .
|
|
|
|
.El
|
2006-09-30 11:02:17 +00:00
|
|
|
.Sh ENVIRONMENT
|
|
|
|
.Bl -tag -width ".Ev EDITOR"
|
|
|
|
.It Ev EDITOR
|
|
|
|
The name of the editor to use for editing configuration files, by
|
|
|
|
default
|
|
|
|
.Xr vi 1
|
|
|
|
is invoked.
|
|
|
|
.El
|
2005-11-19 20:26:52 +00:00
|
|
|
.Sh FILES
|
2006-09-30 11:02:17 +00:00
|
|
|
.Bl -tag -width ".Pa /dev/gvinum/plex"
|
2005-11-19 20:26:52 +00:00
|
|
|
.It Pa /dev/gvinum
|
|
|
|
directory with device nodes for
|
|
|
|
.Nm
|
|
|
|
objects
|
|
|
|
.El
|
Import the gvinum work that have been done during and after Summer of Code 2007.
The work have been under testing and fixing since then, and it is mature enough
to be put into HEAD for further testing.
A lot have changed in this time, and here are the most important:
- Gvinum now uses one single workerthread instead of one thread for each
volume and each plex. The reason for this is that the previous scheme was
very complex, and was the cause of many of the bugs discovered in gvinum.
Instead, gvinum now uses one worker thread with an event queue, quite
similar to what used in gmirror.
- The rebuild/grow/initialize/parity check routines no longer runs in
separate threads, but are run as regular I/O requests with special flags.
This made it easier to support mounted growing and parity rebuild.
- Support for growing striped and raid5-plexes, meaning that one can extend the
volumes for these plex types in addition to the concat type. Also works while
the volume is mounted.
- Implementation of many of the missing commands from the old vinum:
attach/detach, start (was partially implemented), stop (was partially
implemented), concat, mirror, stripe, raid5 (shortcuts for creating volumes
with one plex of these organizations).
- The parity check and rebuild no longer goes between userland/kernel, meaning
that the gvinum command will not stay and wait forever for the rebuild to
finish. You can instead watch the status with the list command.
- Many problems with gvinum have been reported since 5.x, and some has been hard
to fix due to the complicated architecture. Hopefully, it should be more
stable and better handle edge cases that previously made gvinum crash.
- Failed drives no longer disappears entirely, but now leave behind a dummy
drive that makes sure the original state is not forgotten in case the system
is rebooted between drive failures/swaps.
- Update manpage to reflect new commands and extend it with some examples.
Sponsored by: Google Summer of Code 2007
Mentored by: le
Tested by: Rick C. Petty <rick-freebsd2008 -at- kiwi-computer.com>
2009-03-28 17:20:08 +00:00
|
|
|
.Sh EXAMPLES
|
|
|
|
To create a mirror on disks /dev/ad1 and /dev/ad2, create a filesystem, mount,
|
|
|
|
unmount and then stop Ic gvinum:
|
|
|
|
.Pp
|
|
|
|
.Dl "gvinum mirror /dev/ad1 /dev/ad2"
|
|
|
|
.Dl "newfs /dev/gvinum/gvinumvolume0"
|
|
|
|
.Dl "mount /dev/gvinum/gvinumvolume0 /mnt"
|
|
|
|
.Dl "..."
|
|
|
|
.Dl "unmount /mnt"
|
|
|
|
.Dl "gvinum stop"
|
|
|
|
.Pp
|
|
|
|
To create a striped mirror on disks /dev/ad1 /dev/ad2 /dev/ad3 and /dev/ad4
|
|
|
|
named "data" and create a filesystem:
|
|
|
|
.Pp
|
|
|
|
.Dl "gvinum mirror -s -n data /dev/ad1 /dev/ad2 /dev/ad3 /dev/ad4"
|
|
|
|
.Dl "newfs /dev/gvinum/data"
|
|
|
|
.Pp
|
|
|
|
To create a raid5 array on disks /dev/ad1 /dev/ad2 and /dev/ad3, with stripesize
|
|
|
|
493k you can use the raid5 command:
|
|
|
|
.Pp
|
|
|
|
.Dl "gvinum raid5 -s 493k /dev/ad1 /dev/ad2 /dev/ad3"
|
|
|
|
.Pp
|
|
|
|
Then the volume will be created automatically.
|
|
|
|
Afterwards, you have to initialize the volume:
|
|
|
|
.Pp
|
|
|
|
.Dl "gvinum start myraid5vol"
|
|
|
|
.Pp
|
|
|
|
The initialization will start, and the states will be updated when it's
|
|
|
|
finished.
|
|
|
|
The list command will give you information about its progress.
|
|
|
|
.Pp
|
|
|
|
Imagine that one of the drives fails, and the output of 'printconfig' looks
|
|
|
|
something like this:
|
|
|
|
.Pp
|
|
|
|
.Dl "drive gvinumdrive1 device /dev/ad2"
|
|
|
|
.Dl "drive gvinumdrive2 device /dev/???"
|
|
|
|
.Dl "drive gvinumdrive0 device /dev/ad1"
|
|
|
|
.Dl "volume myraid5vol"
|
|
|
|
.Dl "plex name myraid5vol.p0 org raid5 986s vol myraid5vol"
|
|
|
|
.Dl "sd name myraid5vol.p0.s2 drive gvinumdrive2 len 32538s driveoffset 265s"
|
|
|
|
.Dl "plex myraid5vol.p0 plexoffset 1972s"
|
|
|
|
.Dl "sd name myraid5vol.p0.s1 drive gvinumdrive1 len 32538s driveoffset 265s"
|
|
|
|
.Dl "plex myraid5vol.p0 plexoffset 986s"
|
|
|
|
.Dl "sd name myraid5vol.p0.s0 drive gvinumdrive0 len 32538s driveoffset 265s"
|
|
|
|
.Dl "plex myraid5vol.p0 plexoffset 0s"
|
|
|
|
.Pp
|
|
|
|
Create a new drive with this configuration:
|
|
|
|
.Pp
|
|
|
|
.Dl "drive gdrive4 device /dev/ad4"
|
|
|
|
.Pp
|
|
|
|
Then move the stale subdisk to the new drive:
|
|
|
|
.Pp
|
|
|
|
.Dl "gvinum move gdrive4 myraid5vol.p0.s2"
|
|
|
|
.Pp
|
|
|
|
Then, initiate the rebuild:
|
|
|
|
.Pp
|
|
|
|
.Dl "gvinum start myraid5vol.p0"
|
|
|
|
.Pp
|
|
|
|
The plex will go up form degraded mode after the rebuild is finished.
|
|
|
|
The plex can still be used while the rebuild is in progress, although requests
|
|
|
|
might be delayed.
|
|
|
|
For a more advanced usage and detailed explanation of gvinum, the
|
|
|
|
handbook is recommended.
|
|
|
|
.Pp
|
|
|
|
Given the configuration as in the previous example, growing a RAID-5 or STRIPED
|
|
|
|
array is accomplished by adding a new subdisk to the plex with a
|
|
|
|
.Ar description-file
|
|
|
|
similar to this:
|
|
|
|
.Pp
|
|
|
|
.Dl "drive newdrive device /dev/ad4"
|
|
|
|
.Dl "sd drive newdrive plex myraid5vol.p0"
|
|
|
|
.Pp
|
|
|
|
If everything went ok, the plex state should now be set to growable.
|
|
|
|
You can then start the growing with the
|
|
|
|
.Ic start
|
|
|
|
command:
|
|
|
|
.Pp
|
|
|
|
.Dl "gvinum start myraid5vol.p0"
|
|
|
|
.Pp
|
|
|
|
As with rebuilding, you can watch the progress using the
|
|
|
|
.Ic list
|
|
|
|
command.
|
|
|
|
.Pp
|
2005-11-19 20:26:52 +00:00
|
|
|
.Sh SEE ALSO
|
|
|
|
.Xr geom 4 ,
|
|
|
|
.Xr geom 8
|
|
|
|
.Sh HISTORY
|
|
|
|
The
|
|
|
|
.Nm
|
2005-11-20 10:35:46 +00:00
|
|
|
utility first appeared in
|
2006-09-30 11:02:17 +00:00
|
|
|
.Fx 5.3 .
|
|
|
|
The
|
2005-11-20 10:35:46 +00:00
|
|
|
.Nm vinum
|
2005-11-19 20:26:52 +00:00
|
|
|
utility, on which
|
2005-11-20 10:35:46 +00:00
|
|
|
.Nm
|
2006-09-30 11:02:17 +00:00
|
|
|
is based, was written by
|
|
|
|
.An "Greg Lehey" .
|
2005-11-19 20:26:52 +00:00
|
|
|
.Pp
|
2006-09-30 11:02:17 +00:00
|
|
|
The
|
2005-11-20 10:35:46 +00:00
|
|
|
.Nm
|
2006-09-30 11:02:17 +00:00
|
|
|
utility
|
|
|
|
was written by
|
|
|
|
.An "Lukas Ertl" .
|
|
|
|
The
|
|
|
|
.Ic move
|
|
|
|
and
|
|
|
|
.Ic rename
|
|
|
|
commands and
|
|
|
|
documentation were added by
|
|
|
|
.An "Chris Jones"
|
|
|
|
through the 2005 Google Summer
|
2005-11-19 20:26:52 +00:00
|
|
|
of Code program.
|
Import the gvinum work that have been done during and after Summer of Code 2007.
The work have been under testing and fixing since then, and it is mature enough
to be put into HEAD for further testing.
A lot have changed in this time, and here are the most important:
- Gvinum now uses one single workerthread instead of one thread for each
volume and each plex. The reason for this is that the previous scheme was
very complex, and was the cause of many of the bugs discovered in gvinum.
Instead, gvinum now uses one worker thread with an event queue, quite
similar to what used in gmirror.
- The rebuild/grow/initialize/parity check routines no longer runs in
separate threads, but are run as regular I/O requests with special flags.
This made it easier to support mounted growing and parity rebuild.
- Support for growing striped and raid5-plexes, meaning that one can extend the
volumes for these plex types in addition to the concat type. Also works while
the volume is mounted.
- Implementation of many of the missing commands from the old vinum:
attach/detach, start (was partially implemented), stop (was partially
implemented), concat, mirror, stripe, raid5 (shortcuts for creating volumes
with one plex of these organizations).
- The parity check and rebuild no longer goes between userland/kernel, meaning
that the gvinum command will not stay and wait forever for the rebuild to
finish. You can instead watch the status with the list command.
- Many problems with gvinum have been reported since 5.x, and some has been hard
to fix due to the complicated architecture. Hopefully, it should be more
stable and better handle edge cases that previously made gvinum crash.
- Failed drives no longer disappears entirely, but now leave behind a dummy
drive that makes sure the original state is not forgotten in case the system
is rebooted between drive failures/swaps.
- Update manpage to reflect new commands and extend it with some examples.
Sponsored by: Google Summer of Code 2007
Mentored by: le
Tested by: Rick C. Petty <rick-freebsd2008 -at- kiwi-computer.com>
2009-03-28 17:20:08 +00:00
|
|
|
.Ic a partial rewrite of gvinum was done by "Lukas Ertl" and "Ulf Lilleengen"
|
|
|
|
through the 2007 Google Summer of Code program.
|
|
|
|
The documentation have been updated to reflect the new functionality.
|
2006-09-30 11:02:17 +00:00
|
|
|
.Sh AUTHORS
|
|
|
|
.An Lukas Ertl Aq le@FreeBSD.org
|
|
|
|
.An Chris Jones Aq soc-cjones@FreeBSD.org
|
Import the gvinum work that have been done during and after Summer of Code 2007.
The work have been under testing and fixing since then, and it is mature enough
to be put into HEAD for further testing.
A lot have changed in this time, and here are the most important:
- Gvinum now uses one single workerthread instead of one thread for each
volume and each plex. The reason for this is that the previous scheme was
very complex, and was the cause of many of the bugs discovered in gvinum.
Instead, gvinum now uses one worker thread with an event queue, quite
similar to what used in gmirror.
- The rebuild/grow/initialize/parity check routines no longer runs in
separate threads, but are run as regular I/O requests with special flags.
This made it easier to support mounted growing and parity rebuild.
- Support for growing striped and raid5-plexes, meaning that one can extend the
volumes for these plex types in addition to the concat type. Also works while
the volume is mounted.
- Implementation of many of the missing commands from the old vinum:
attach/detach, start (was partially implemented), stop (was partially
implemented), concat, mirror, stripe, raid5 (shortcuts for creating volumes
with one plex of these organizations).
- The parity check and rebuild no longer goes between userland/kernel, meaning
that the gvinum command will not stay and wait forever for the rebuild to
finish. You can instead watch the status with the list command.
- Many problems with gvinum have been reported since 5.x, and some has been hard
to fix due to the complicated architecture. Hopefully, it should be more
stable and better handle edge cases that previously made gvinum crash.
- Failed drives no longer disappears entirely, but now leave behind a dummy
drive that makes sure the original state is not forgotten in case the system
is rebooted between drive failures/swaps.
- Update manpage to reflect new commands and extend it with some examples.
Sponsored by: Google Summer of Code 2007
Mentored by: le
Tested by: Rick C. Petty <rick-freebsd2008 -at- kiwi-computer.com>
2009-03-28 17:20:08 +00:00
|
|
|
.An Ulf Lilleengen Aq lulf@FreeBSD.org
|
2005-11-19 20:26:52 +00:00
|
|
|
.Sh BUGS
|
2006-09-30 11:02:17 +00:00
|
|
|
Currently,
|
|
|
|
.Nm
|
|
|
|
does not rename devices in
|
2005-11-20 10:35:46 +00:00
|
|
|
.Pa /dev/gvinum
|
2005-11-19 20:26:52 +00:00
|
|
|
until reloaded.
|
|
|
|
.Pp
|
2005-11-20 10:35:46 +00:00
|
|
|
The
|
2005-11-19 20:26:52 +00:00
|
|
|
.Fl S
|
2005-11-20 10:35:46 +00:00
|
|
|
initsize flag to
|
2005-11-19 20:26:52 +00:00
|
|
|
.Ic start
|
|
|
|
is ignored.
|
|
|
|
.Pp
|
2005-11-20 10:35:46 +00:00
|
|
|
Moving subdisks that are not part of a mirrored or RAID-5 volume will
|
2006-09-30 11:02:17 +00:00
|
|
|
destroy data.
|
|
|
|
It is perhaps a bug to permit this.
|
2005-11-19 20:26:52 +00:00
|
|
|
.Pp
|
2005-11-20 10:35:46 +00:00
|
|
|
Plexes in which subdisks have been moved do not automatically sync or
|
2006-09-30 11:02:17 +00:00
|
|
|
rebuild parity.
|
|
|
|
This may leave data unprotected and is perhaps unwise.
|
2005-11-19 20:26:52 +00:00
|
|
|
.Pp
|
2006-09-30 11:02:17 +00:00
|
|
|
Currently,
|
|
|
|
.Nm
|
|
|
|
does not yet fully implement all of the functions found in
|
|
|
|
.Xr vinum 4 .
|
|
|
|
Specifically, the following commands from
|
|
|
|
.Xr vinum 4
|
|
|
|
are not supported:
|
2005-11-19 20:26:52 +00:00
|
|
|
.Bl -tag -width indent
|
|
|
|
.It Ic debug
|
|
|
|
Cause the volume manager to enter the kernel debugger.
|
|
|
|
.It Ic debug Ar flags
|
|
|
|
Set debugging flags.
|
|
|
|
.It Ic dumpconfig Op Ar drive ...
|
|
|
|
List the configuration information stored on the specified drives, or all
|
|
|
|
drives in the system if no drive names are specified.
|
2006-09-30 11:02:17 +00:00
|
|
|
.It Ic info Op Fl vV
|
2005-11-19 20:26:52 +00:00
|
|
|
List information about volume manager state.
|
|
|
|
.It Ic label Ar volume
|
|
|
|
Create a volume label.
|
2006-09-30 11:02:17 +00:00
|
|
|
.It Ic resetstats Oo Fl r Oc Op Ar volume | plex | subdisk
|
2005-11-19 20:26:52 +00:00
|
|
|
Reset statistics counters for the specified objects, or for all objects if none
|
|
|
|
are specified.
|
|
|
|
.It Ic setdaemon Op Ar value
|
|
|
|
Set daemon configuration.
|
|
|
|
.El
|