566 lines
16 KiB
Plaintext
566 lines
16 KiB
Plaintext
Diagnostic Tools
|
|
shaharf@voltaire.com, halr@voltaire.com
|
|
|
|
General:
|
|
|
|
Model of operation: All utilities use direct MAD access to perform their
|
|
operations. Operations that require QP 0 mads only, may use direct routed
|
|
mads, and therefore may work even in unconfigured subnets. Almost all
|
|
utilities can operate without accessing the SM, unless GUID to lid translation
|
|
is required.
|
|
|
|
Dependencies: Most utilities depend on libibmad and libibumad.
|
|
All utilities depend on the ib_umad kernel module.
|
|
|
|
Multiple port/Multiple CA support: when no IB device or port is specified
|
|
(see the "local umad parameters" below), the libibumad library
|
|
selects the port to use by the following criteria:
|
|
1. the first port that is ACTIVE.
|
|
2. if not found, the first port that is UP (physical link up).
|
|
|
|
If a port and/or CA name is specified, the libibumad library
|
|
attempts to fulfill the user request, and will fail if it is not
|
|
possible.
|
|
For example:
|
|
ibaddr # use the 'best port'
|
|
ibaddr -C mthca1 # pick the best port from mthca1 only.
|
|
ibaddr -P 2 # use the second (active/up) port from the
|
|
first available IB device.
|
|
ibaddr -C mthca0 -P 2 # use the specified port only.
|
|
|
|
Common options & flags:
|
|
Most diagnostics take the following flags. The exact list of supported
|
|
flags per utility can be found in the usage message and can be shown
|
|
using util_name -h syntax.
|
|
|
|
# Debugging flags
|
|
-d raise the IB debugging level. May be used
|
|
several times (-ddd or -d -d -d).
|
|
-e show umad send receive errors (timeouts and others)
|
|
-h show the usage message
|
|
-v increase the application verbosity level.
|
|
May be used several times (-vv or -v -v -v)
|
|
-V show the internal version info.
|
|
|
|
# Addressing flags
|
|
-D use directed path address arguments. The path
|
|
is a comma separated list of out ports.
|
|
Examples:
|
|
"0" # self port
|
|
"0,1,2,1,4" # out via port 1, then 2, ...
|
|
-G use GUID address arguments. In most cases, it is the Port GUID.
|
|
Examples:
|
|
"0x08f1040023"
|
|
-s <smlid> use 'smlid' as the target lid for SA queries.
|
|
|
|
# Local umad parameters:
|
|
-C <ca_name> use the specified ca_name.
|
|
-P <ca_port> use the specified ca_port.
|
|
-t <timeout_ms> override the default timeout for the solicited mads.
|
|
|
|
CLI notation: all utilities use the POSIX style notation,
|
|
meaning that all options (flags) must precede all arguments
|
|
(parameters).
|
|
|
|
|
|
Utilities descriptions:
|
|
|
|
1. ibstatus
|
|
|
|
Description:
|
|
ibstatus is a script which displays basic information obtained from the local
|
|
IB driver. Output includes LID, SMLID, port state, link width active, and port
|
|
physical state.
|
|
|
|
Syntax:
|
|
ibstatus [-h] [devname[:port]]...
|
|
|
|
Examples:
|
|
ibstatus # display status of all IB ports
|
|
ibstatus mthca1 # status of mthca1 ports
|
|
ibstatus mthca1:1 mthca0:2 # show status of specified ports
|
|
|
|
See also:
|
|
ibstat
|
|
|
|
2. ibstat
|
|
|
|
Description:
|
|
Similar to the ibstatus utility but implemented as a binary and not a script.
|
|
It has options to list CAs and/or ports.
|
|
|
|
Syntax:
|
|
ibstat [-d(ebug) -l(ist_of_cas) -p(ort_list) -s(hort)] <ca_name> [portnum]
|
|
|
|
Examples:
|
|
ibstat # display status of all IB ports
|
|
ibstat mthca1 # status of mthca1 ports
|
|
ibstat mthca1 2 # show status of specified ports
|
|
ibstat -p mthca0 # list the port guids of mthca0
|
|
ibstat -l # list all CA names
|
|
|
|
See also:
|
|
ibstatus
|
|
|
|
3. ibroute
|
|
|
|
Description:
|
|
ibroute uses SMPs to display the forwarding tables (unicast
|
|
(LinearForwardingTable or LFT) or multicast (MulticastForwardingTable or MFT))
|
|
for the specified switch LID and the optional lid (mlid) range.
|
|
The default range is all valid entries in the range 1...FDBTop.
|
|
|
|
Syntax:
|
|
ibroute [options] <switch_addr> [<startlid> [<endlid>]]]
|
|
|
|
Non standard flags:
|
|
-a show all lids in range, even invalid entries.
|
|
-n do not try to resolve destinations.
|
|
-M show multicast forwarding tables. In this case the range
|
|
parameters are specifying mlid range.
|
|
|
|
Examples:
|
|
ibroute 2 # dump all valid entries of switch lid 2
|
|
ibroute 2 15 # dump entries in the range 15...FDBTop.
|
|
ibroute -a 2 10 20 # dump all entries in the range 10..20
|
|
ibroute -n 2 # simple format
|
|
ibroute -M 2 # show multicast tables
|
|
|
|
See also:
|
|
ibtracert
|
|
|
|
4. ibtracert
|
|
|
|
Description:
|
|
ibtracert uses SMPs to trace the path from a source GID/LID to a
|
|
destination GID/LID. Each hop along the path is displayed until the destination
|
|
is reached or a hop does not respond. By using the -m option, multicast path
|
|
tracing can be performed between source and destination nodes.
|
|
|
|
Syntax:
|
|
ibtracert [options] <src-addr> <dest-addr>
|
|
|
|
Non standard flags:
|
|
-n simple format; don't show additional information.
|
|
-m <mlid> show the multicast trace of the specified mlid.
|
|
|
|
Examples:
|
|
ibtracert 2 23 # show trace between lid 2 and 23
|
|
ibtracert -m 0xc000 3 5 # show multicast trace between lid 3 and 5
|
|
for mcast lid 0xc000.
|
|
|
|
5. smpquery
|
|
|
|
Description:
|
|
smpquery allows a basic subset of standard SMP queries including the following:
|
|
node info, node description, switch info, port info. Fields are displayed in
|
|
human readable format.
|
|
|
|
Syntax:
|
|
smpquery [options] <op> <dest_addr> [op_params]
|
|
|
|
Current supported operations and their parameters:
|
|
nodeinfo <addr>
|
|
nodedesc <addr>
|
|
portinfo <addr> [<portnum>] # default port is zero
|
|
switchinfo <addr>
|
|
pkeys <addr> [<portnum>]
|
|
sl2vl <addr> [<portnum>]
|
|
vlarb <addr> [<portnum>]
|
|
|
|
Examples:
|
|
smpquery nodeinfo 2 # show nodeinfo for lid 2
|
|
smpquery portinfo 2 5 # show portinfo for lid 2 port 5
|
|
|
|
6. smpdump
|
|
|
|
Description:
|
|
smpdump is a general purpose SMP utility which gets SM attributes from a
|
|
specified SMA. The result is dumped in hex by default.
|
|
|
|
Syntax:
|
|
smpdump [options] <dest_addr> <attr> [mod]
|
|
|
|
Non standard flags:
|
|
-s show output as string
|
|
|
|
Examples:
|
|
smpdump -D 0,1,2 0x15 2 # port info, port 2
|
|
smpdump 3 0x15 2 # port info, lid 3 port 2
|
|
|
|
7. ibaddr
|
|
|
|
Description:
|
|
ibaddr can be used to show the lid and GID addresses of the specified port,
|
|
or the local port by default.
|
|
Note: this utility can be used as simple address resolver.
|
|
|
|
Syntax:
|
|
ibaddr [options] [<dest_addr>]
|
|
|
|
Examples:
|
|
ibaddr # show local address
|
|
ibaddr 2 # show address of the specified port lid
|
|
ibaddr -G 0x8f1040023 # show address of the specified port guid
|
|
|
|
8. sminfo
|
|
|
|
Description:
|
|
sminfo issue and dumps the output of a sminfo query in human readable format.
|
|
The target SM is the one listed in the local port info, or the SM specified
|
|
by the optional SM lid or by the SM direct routed path.
|
|
Note: using sminfo for any purposes other then simple query may be very
|
|
dangerous, and may result in a malfunction of the target SM.
|
|
|
|
Syntax:
|
|
sminfo [options] <sm_lid|sm_dr_path> [sminfo_modifier]
|
|
|
|
Non standard flags:
|
|
-s <state> # use the specified state in sminfo mad
|
|
-p <priority> # use the specified priority in sminfo mad
|
|
-a <activity> # use the specified activity in sminfo mad
|
|
|
|
Examples:
|
|
sminfo # show sminfo of SM listed in local portinfo
|
|
sminfo 2 # query SM on port lid 2
|
|
|
|
9. perfquery
|
|
|
|
Description:
|
|
perfquery uses PerfMgt GMPs to obtain the PortCounters (basic performance
|
|
and error counters) from the PMA at the node specified. Optionally
|
|
show aggregated counters for all ports of node. Also, optionally, reset
|
|
after read, or only reset counters.
|
|
|
|
Syntax:
|
|
perfquery [options] [<lid|guid> [[port] [reset_mask]]]
|
|
|
|
Non standard flags:
|
|
-a show aggregated counters for all ports of the destination lid.
|
|
-r reset counters after read.
|
|
-R only reset counters.
|
|
|
|
Examples:
|
|
perfquery # read local port's performance counters
|
|
perfquery 32 1 # read performance counters from lid 32, port 1
|
|
perfquery -a 32 # read node aggregated performance counters
|
|
perfquery -r 32 1 # read performance counters and reset
|
|
perfquery -R 32 1 # reset performance counters of port 1 only
|
|
perfquery -R -a 32 # reset performance counters of all ports
|
|
perfquery -R 32 2 0xf000 # reset only non-error counters of port 2
|
|
|
|
10. ibping
|
|
|
|
Description:
|
|
ibping uses vendor mads to validate connectivity between IB nodes.
|
|
On exit, (IP) ping like output is show. ibping is run as client/server.
|
|
Default is to run as client. Note also that a default ping server is
|
|
implemented within the kernel.
|
|
|
|
Syntax:
|
|
ibping [options] <dest lid|guid>
|
|
|
|
Non standard flags:
|
|
-c <count> stop after count packets
|
|
-f flood destination: send packets back to back w/o delay
|
|
-o <oui> use specified OUI number to multiplex vendor mads
|
|
-S start in server mode (do not return)
|
|
|
|
11. ibnetdiscover
|
|
|
|
Description:
|
|
ibnetdiscover performs IB subnet discovery and outputs a human readable
|
|
topology file. GUIDs, node types, and port numbers are displayed
|
|
as well as port LIDs and NodeDescriptions. All nodes (and links) are displayed
|
|
(full topology). Optionally this utility can be used to list the current
|
|
connected nodes. The output is printed to the standard output unless a
|
|
topology file is specified.
|
|
|
|
Syntax:
|
|
ibnetdiscover [options] [<topology-filename>]
|
|
|
|
Non standard flags:
|
|
-l List of connected nodes
|
|
-H List of connected HCAs
|
|
-S List of connected switches
|
|
-g Grouping
|
|
|
|
12. ibhosts
|
|
|
|
Description:
|
|
ibhosts either walks the IB subnet topology or uses an already saved topology
|
|
file and extracts the CA nodes.
|
|
|
|
Syntax:
|
|
ibhosts [-h] [<topology-file>]
|
|
|
|
Dependencies:
|
|
ibnetdiscover, ibnetdiscover format
|
|
|
|
13. ibswitches
|
|
|
|
Description:
|
|
ibswitches either walks the IB subnet topology or uses an already saved
|
|
topology file and extracts the IB switches.
|
|
|
|
Syntax:
|
|
ibswitches [-h] [<topology-file>]
|
|
|
|
Dependencies:
|
|
ibnetdiscover, ibnetdiscover format
|
|
|
|
14. ibchecknet
|
|
|
|
Description:
|
|
ibchecknet uses a full topology file that was created by ibnetdiscover,
|
|
scans the network to validate the connectivity and reports errors
|
|
(from port counters).
|
|
|
|
Syntax:
|
|
ibchecknet [-h] [<topology-file>]
|
|
|
|
Dependencies:
|
|
ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckport, ibcheckerrs
|
|
|
|
15. ibcheckport
|
|
|
|
Description:
|
|
Check connectivity and do some simple sanity checks for the specified port.
|
|
Port address is lid unless -G option is used to specify a GUID address.
|
|
|
|
Syntax:
|
|
ibcheckport [-h] [-G] <lid|guid> <port_number>
|
|
|
|
Example:
|
|
ibcheckport 2 3 # check lid 2 port 3
|
|
|
|
Dependencies:
|
|
smpquery, smpquery output format, ibaddr
|
|
|
|
16. ibchecknode
|
|
|
|
Description:
|
|
Check connectivity and do some simple sanity checks for the specified node.
|
|
Port address is lid unless -G option is used to specify a GUID address.
|
|
|
|
Syntax:
|
|
ibchecknode [-h] [-G] <lid|guid>
|
|
|
|
Example:
|
|
ibchecknode 2 # check node via lid 2
|
|
|
|
Dependencies:
|
|
smpquery, smpquery output format, ibaddr
|
|
|
|
Usage:
|
|
|
|
17. ibcheckerrs
|
|
|
|
Description:
|
|
Check specified port (or node) and report errors that surpassed their predefined
|
|
threshold. Port address is lid unless -G option is used to specify a GUID
|
|
address. The predefined thresholds can be dumped using the -s option, and a
|
|
user defined threshold_file (using the same format as the dump) can be
|
|
specified using the -t <file> option.
|
|
|
|
Syntax:
|
|
ibcheckerrs [-h] [-G] [-t <threshold_file>] [-s(how_thresholds)] <lid|guid> [<port>]
|
|
|
|
Examples:
|
|
ibcheckerrs 2 # check aggregated node counter for lid 2
|
|
ibcheckerrs 2 4 # check port counters for lid 2 port 4
|
|
ibcheckerrs -t xxx 2 # check node using xxx threshold file
|
|
|
|
Dependencies:
|
|
perfquery, perfquery output format, ibaddr
|
|
|
|
18. ibportstate
|
|
|
|
Description:
|
|
ibportstate allows the port state and port physical state of an IB port
|
|
to be queried or a switch port to be disabled or enabled.
|
|
|
|
Syntax:
|
|
ibportstate [-d(ebug) -e(rr_show) -v(erbose) -D(irect) -G(uid) -s smlid
|
|
-V(ersion) -C ca_name -P ca_port -t timeout_ms] <dest dr_path|lid|guid>
|
|
<portnum> [<op>]
|
|
supported ops: enable, disable, query
|
|
|
|
Examples:
|
|
ibportstate 3 1 disable # by lid
|
|
ibportstate -G 0x2C9000100D051 1 enable # by guid
|
|
ibportstate -D 0 1 # by direct route
|
|
|
|
19. ibcheckwidth
|
|
|
|
Description:
|
|
ibcheckwidth uses a full topology file that was created by ibnetdiscover,
|
|
scans the network to validate the active link widths and reports any 1x
|
|
links.
|
|
|
|
Syntax:
|
|
ibcheckwidth [-h] [<topology-file>]
|
|
|
|
Dependencies:
|
|
ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckportwidth
|
|
|
|
20. ibcheckportwidth
|
|
|
|
Description:
|
|
Check connectivity and check the specified port for 1x link width.
|
|
Port address is lid unless -G option is used to specify a GUID address.
|
|
|
|
Syntax:
|
|
ibcheckportwidth [-h] [-G] <lid|guid> <port>
|
|
|
|
Example:
|
|
ibcheckportwidth 2 3 # check lid 2 port 3
|
|
|
|
Dependencies:
|
|
smpquery, smpquery output format, ibaddr
|
|
|
|
21. ibcheckstate
|
|
|
|
Description:
|
|
ibcheckstate uses a full topology file that was created by ibnetdiscover,
|
|
scans the network to validate the port state and port physical state,
|
|
and reports any ports which have a port state other than Active or
|
|
a port physical state other than LinkUp.
|
|
|
|
Syntax:
|
|
ibcheckstate [-h] [<topology-file>]
|
|
|
|
Dependencies:
|
|
ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckportstate
|
|
|
|
22. ibcheckportstate
|
|
|
|
Description:
|
|
Check connectivity and check the specified port for proper port state
|
|
(Active) and port physical state (LinkUp).
|
|
Port address is lid unless -G option is used to specify a GUID address.
|
|
|
|
yntax:
|
|
ibcheckportstate [-h] [-G] <lid|guid> <port_number>
|
|
|
|
Example:
|
|
ibcheckportstate 2 3 # check lid 2 port 3
|
|
|
|
Dependencies:
|
|
smpquery, smpquery output format, ibaddr
|
|
|
|
23. ibcheckerrors
|
|
|
|
ibcheckerrors uses a full topology file that was created by ibnetdiscover,
|
|
scans the network to validate the connectivity and reports errors
|
|
(from port counters).
|
|
|
|
Syntax:
|
|
ibnetcheckerrors [-h] [<topology-file>]
|
|
|
|
Dependencies:
|
|
ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckport, ibcheckerrs
|
|
|
|
24. ibdiscover.pl
|
|
|
|
ibdiscover.pl uses a topology file create by ibnetdiscover and a discover.map
|
|
file which the network administrator creates which indicates the nodes
|
|
to be expected and a ibdiscover.topo file which is the expected connectivity
|
|
and produces a new connectivity file (discover.topo.new) and outputs
|
|
the changes to stdout. The network administrator can choose to replace
|
|
the "old" topo file with the new one or certain changes in.
|
|
|
|
The syntax of the ibdiscover.map file is:
|
|
<nodeGUID>|port|"Text for node"|<NodeDescription from ibnetdiscover format>
|
|
e.g.
|
|
8f10400410015|8|"ISR 6000"|# SW-6IB4 Voltaire port 0 lid 5
|
|
8f10403960558|2|"HCA 1"|# MT23108 InfiniHost Mellanox Technologies
|
|
|
|
The syntax of the old and new topo files (ibdiscover.topo and
|
|
ibdiscover.topo.new) are:
|
|
<LocalPort>|<LocalNodeGUID>|<RemotePort>|<RemoteNodeGUID>
|
|
e.g.
|
|
10|5442ba00003080|1|8f10400410015
|
|
|
|
These topo files are produced by the ibdiscover.pl tool.
|
|
|
|
Syntax:
|
|
ibnetdiscover | ibdiscover.pl
|
|
|
|
Dependencies:
|
|
ibnetdiscover, ibnetdiscover format
|
|
|
|
25. ibnodes
|
|
|
|
Description:
|
|
ibnodes either walks the IB subnet topology or uses an already saved topology
|
|
file and extracts the IB nodes (CAs and switches).
|
|
|
|
Syntax:
|
|
ibnodes [<topology-file>]
|
|
|
|
Dependencies:
|
|
ibnetdiscover, ibnetdiscover format
|
|
|
|
26. ibclearerrors
|
|
|
|
Description:
|
|
ibclearerrors clears the PMA error counters in PortCounters by either walking
|
|
the IB subnet topology or using an already saved topology file.
|
|
|
|
Syntax:
|
|
ibclearerrors [-h] [<topology-file>]
|
|
|
|
Dependencies:
|
|
ibnetdiscover, ibnetdiscover format, perfquery
|
|
|
|
27. ibclearcounters
|
|
|
|
Description:
|
|
ibclearcounters clears the PMA port counters by either walking
|
|
the IB subnet topology or using an already saved topology file.
|
|
|
|
Syntax:
|
|
ibclearcounters [-h] [<topology-file>]
|
|
|
|
Dependencies:
|
|
ibnetdiscover, ibnetdiscover format, perfquery
|
|
|
|
28. saquery
|
|
|
|
Description:
|
|
Issue some SA queries.
|
|
|
|
Syntax:
|
|
Usage: saquery [-h -d -P -N -L -G -s -g][<name>]
|
|
Queries node records by default
|
|
-d enable debugging
|
|
-P get PathRecord info
|
|
-N get NodeRecord info
|
|
-L Return just the Lid of the name specified
|
|
-G Return just the Guid of the name specified
|
|
-s Return the PortInfoRecords with isSM capability mask bit on
|
|
-g get multicast group info
|
|
|
|
Dependencies:
|
|
OpenSM libvendor, OpenSM libopensm, libibumad
|
|
|
|
29. ibsysstat
|
|
|
|
Description:
|
|
ibsysstat uses vendor mads to validate connectivity between IB nodes
|
|
and obtain other information about the IB node. ibsysstat is run as
|
|
client/server. Default is to run as client.
|
|
|
|
Syntax:
|
|
ibsysstat [options] <dest lid|guid> [<op>]
|
|
|
|
Non standard flags:
|
|
Current supported operations:
|
|
ping - verify connectivity to server (default)
|
|
host - obtain host information from server
|
|
cpu - obtain cpu information from server
|
|
-o <oui> use specified OUI number to multiplex vendor mads
|
|
-S start in server mode (do not return)
|
|
|