MAN page from OpenSuSE 13.X VMware-vSphere-CLI-6.7.0-8156551.8156551.1.x86_64.rpm
ESXTOP
Section: VMware ESXi Manual (1)
Updated: February 1, 2014
Index NAME
esxtop, resxtop - display ESXi resource utilization statistics
SYNOPSIS
esxtop[
-][
h][
v][
b][
l][
s][
a][
cconfigfile][
Rvm-support_dir_path][
ddelay][
niter]
resxtop[-][h][v][b][s][a][cconfigfile][ddelay][niter][-serverserver-name[-vihosthost-name]][-portnumbersocket-port][-usernameuser-name]
DESCRIPTION
esxtopallows monitoring and collection of data for all system resources:CPU, memory, disk, network, etc. When used interactively, this datacan be viewed on different types of screens; one each for CPUstatistics, memory statistics, network statistics, disk adapterstatistics, disk device statistics, disk VM statistics, interruptstatistics, power statistics and vSAN (VSAN) statistics.In the batch mode, data can be redirected to a file for offline uses.
Many esxtop statistics are computed as rates, e.g. CPU statistics%USED. A rate is computed based on the refresh interval, the timebetween successive snapshots. For example,%USED = ( CPU used time at snapshot 2 - CPU used time at snapshot 1 )/ time elapsed between snapshots. The default refresh interval can bechanged by the command line option "-d", or the interactive command 's'.The return key can be pressed to force a refresh.
In each screen, data is presented at different levels of aggregation.It is possible to drill down to expanded views of this data. Eachscreen provides different expansion options.
It is possible to select all or some fields for which data collectionis done. In the case of interactive use of esxtop, the order in whichthe selected fields are displayed can be selected.
For more information, please refer to the official vSphere Monitoringand Performance documentation. You can run esxtop using the ESXi shell.It can only be run by userroot.It can be run in three differentmodes:interactive (default),batchandreplay.Worldsin this document refer to ESXi VMkernel schedulable entities,similar to processes in other operating systems.
resxtopprovides a fine-grain look at resource utilization of a specifiedserver in real time. Resxtop connects to the server using a user name andpassword. Options --server, --vihost, --portnumber and --username canbe used to specify the connection parameters. The resxtop user will beprompted to input the password. If resxtop connects to ESXidirectly, --server specifies the ESXi system name. If resxtop connectsto a VC server, --server specifies the VC server name and --vihost mustbe used to specify the ESXi host name registered in the VC server.Resxtop can be run in two differentmodesin the same way as esxtop:interactive (default)andbatch.
Interactive Mode
This is the default mode for esxtop. This mode displaysresource usage statistics in the following screens:
CPU screen,memory screen,storage adapter screen,storage device screen,storage VM screen,network screen,interrupt screen,power screen,and
VSAN screen.A
helpmenu is available for each screen.
In this mode there are several command line options available.
Command-line Options
hPrints
helpfor esxtop command-line options.vPrints esxtop
versionnumber.sTells esxtop to run in
securemode. This disables the interactive command to change
delaybetween screen updates.aShow all statistics. This option overides ~/.esxtop60rc configurationfile setups and shows all statistics.cLoad user-defined
configurationfile. Default configuration file is ~/.esxtop60rc.User can specify the filename to be used as configuration file.User-defined configuration file will be default file namefor saving configuration with
Wcommand.dSpecifies the
delaybetween screen updates. Default is
5seconds. Minimum is
2seconds.You can change this with the interactive command 's'. If a delay ofless than 2 seconds is specified then the delay is set to 2 seconds.nNumber of
iterations.Update the display 'n' number of times and then exit. It is set to10000 by default.
Common Statistics Description
Several statistics are displayed on the different screens whileesxtop is running in interactive mode. Statistics listed in thissection are
commonacross all screens."uptime"This first line found at the top of each of the esxtopscreens displays the
current time,time since last reboot andnumber of currently running Worlds. On this first line
CPU load averagesare also displayed for the CPU, storage and network screens.The load averages over the past
1,5and
15minutes are displayed. Load averages take into account both running andready-to-run Worlds. A load average of
1.00implies that all the physical CPUs are fully accounted for. A load average of
2.00implies that ESXi may be in need of twice as many physical CPUsas currently available.
For the memory screen
memory overcommit averagesare also displayed on the first line.The memory overcommit averages over the past
1,5and
15minutes are displayed. A memory overcommit average of
0.37implies that the memory is overcommit by
37%.Memory overcommitis the ratio of total requested memory and the "managed memory"minus 1. VMkernel computes the total requested memory as a sum of the followingcomponents: (a) VM configured memory (or memory limit setting if set), (b) theuser world memory, (c) the reserved overhead memory.
Common Interactive Commands
Several single-key commands are recognized while esxtop is runningin interactive mode. Commands listed in this section are
commonacross all screens. The command to specify the delay betweenupdates is disabled if the
soption has been given on the command line. All
sortinginteractive commands sort in
descendingorder."h or
?"Displays a help menu for the current screen giving a brief summaryof commands, and the status of secure mode.spaceImmediately updates the current screen.^LErases and redraws the current screen."f or
F"Displays a screen to add or remove statistics columns(fields)to or from the current screen. See below for more information."o or
O"Displays a screen to changeorderof statistics columns on the current screen.See below for more information.#Change the
numberof statistics rows to show. You will be prompted toenter the number. Any value greater than 0 overrides automaticdetermination of the number of rows to show, which is based on windowsize measurement. This number updated on any one of the esxtopscreens will be used for all esxtop screens.LChange the display length of the name fields, which are "NAME"for the CPU, memory and storage vm screens, "DEVICE" for thesotrage device screen, "DNAME" for the network screen, "DEVICES"for the interrupt screen. You will be prompted to enter the number.Any value greater than 0 overrides the default number of charactersof the name field to show. A value equals to 0 means the defaultvalue applies.sChange the delay between updates. You will be prompted to enterthe delay time, in seconds, between updates. Fractional valuesare recognized down to microseconds. The default value is 5seconds. The minimum value is 2 seconds. This command is notavailable in secure mode.WWrite current setup to
~/.esxtop60rc,or to user-defined configuration file specified by
-coption.This is the recommended way to write an esxtop configurationfile. See below for more information.kKill a world. (This command is only available in esxtop, not in resxtop.)qQuit.cSwitch to
CPUresource utilization screen.mSwitch to
memoryresource utilization screen.dSwitch to
storage (disk) adapterresource utilization screen.uSwitch to
storage (disk) deviceresource utilization screen.vSwitch to
storage (disk) VMresource utilization screen.nSwitch to
networkresource utilization screen.iSwitch to
interruptutilization screen.pSwitch to
powerutilization screen.xSwitch to
VSANutilization screen.
The Statistics Columns (fields) and Order Screens
After pressing
f,F,oor
Oyou will be shown a screen specifying the field order on the top lineand short descriptions of the field contents. The field order stringuses the following syntax: if the letter in the field stringcorresponding to a field is upper case, the field will be displayed.This is also indicated by an asterisk in front of the field description.The order of the fields corresponds to the order of the letters in thestring.From the field select screen you can toggle the display of a field bypressing the corresponding letter.From the order screen you may move a field to the left by pressingthe corresponding upper case letter and to the right by pressing thelower case one.
Configuration File
esxtop reads its default configuration from
~/.esxtop60rcor from specified file name with
-coption.This configuration file contains ten lines. The first nine linescontain lower and upper case letters to specify which fields in whatorder are to be displayed on the
CPU,memory,storage (disk) adapter,storage (disk) device,storage (disk) VM,network,interrupt,powerand
VSANscreens. The letters correspond to the letters in the Fields or Orderscreens for the respective esxtop screens. Editing this file by hand isnot recommended. Select fields and order in a running esxtop process andsave this file using the
Winteractive command.The last line contains information on the other options. Most important,if you have saved a configuration in secure mode, you will not get aninsecure esxtop without removing the lower 's' from this last line of your
~/.esxtop60rc.A digit specifies the delay time between updates.As in interactive mode, a lower 'c', 'm', 'd', 'u', 'v', 'n', 'i', 'p' and 'x'determines the screen esxtop starts on.
CPU Screen
This screen displays
server-wideand per
Resource Pool/WorldCPU utilization statistics. By default, World CPU utilization statistics are aggregatedinto Resource Pools of Worlds. For Worlds belonging to a Virtual Machine, these ResourcePools correspond to running Virtual Machines. All other Worlds are logically aggregatedinto Resource Pools to save vertical window space.
Statistics Descriptions
PCPU USED(%)This line displays the percentage CPU usage per
PCPU,and the percentage CPU usage averaged over all PCPUs. A
PCPUrefers to a physical hardware execution context -- a physicalCPU core if hyper-threading is unavailable or disabled, ora logical CPU (aka LCPU or SMT thread) if hyper-threading is enabled.
PCPU USED(%)is the percentage of PCPU nominal frequency that was usedsince the last screen update. PCPU USED(%) is the total sumof %USED for Worlds that ran on this PCPU. Note that ifPCPU is running at frequency that is higher than its nominal(rated) frequency, then PCPU USED(%) can be greater than 100%.Also, note that if a PCPU and its partner are busy whenhyper-threading is enabled, then each PCPU is accounted forhalf of the CPU usage.
PCPU UTIL(%)This line displays the percentage CPU utilization perPCPU,and the percentage CPU utilization averaged over all PCPUs. APCPUrefers to a physical hardware execution context -- a physicalCPU core if hyper-threading is unavailable or disabled, ora logical CPU (aka LCPU or SMT thread) if hyper-threading is enabled.UTIL(%)represents the raw PCPU utilization, that is the percentageof real time that PCPU was not idle.
USED(%) and UTIL(%) may be different due to power managementtechnologies or hyper-threading.
CORE UTIL(%)This line displays the percentage utilization of eachcoreand the average over all cores. It is displayed onlywhen hyper-threading is used.In interactive mode, the percentage utilization of a coreis aligned with the percentage utilization of its firstlogical CPU (aka LCPU or SMT thread).
A core is utilized, if either or both of its logical CPUs are utilized.The percentage utilization of a core is not the sum ofthe percentage utilization of both logical CPUs on this core.
CCPU(%)This line displays the percentages of total CPU time as reportedby the ESXi Service Console.usis for percentage user time,syis for percentage system time,idis for percentage idle time andwais for percentage wait time.cs/secis for the context switches per second recorded by the ESXiService Console.IDTheResource Pool IDof the running World's Resource Pool orWorld IDof running World.GIDTheResource Pool IDof running World's Resource Pool.LWIDTheLeader World IDof running World's Resource Pool.NAMEThenameof running World's Resource Pool ornameof running World.NWLDThenumber of worldsin running World's Resource Pool. If a Resource Pool is expanded using the interactivecommande(see interactive commands below) then NWLD for all the resulting Worldsbelonging to the Resource Pool will be 1 (some Resource Pool like theconsoleResource Pool have only 1 member).%STATE TIMESThis is a set of CPU statistics made up of the followingpercentages.Notethat for a World, the percentages are a percentage ofone physical CPU core.%USEDThe percentage of physical CPU core cycles used by the Resource Pool/World.
%USED may depend on the frequency with which CPU core is running. Whenrunning with lower CPU core frequency, %USED can be smaller than %RUN.On CPUs which support turbo mode, CPU frequency can also be higher thannominal (rated) frequency, and in that case %USED can be larger than %RUN.%SYSThe percentage of time spent in the ESXi VMkernel on behalfof the Resource Pool/World to process interrupts and to perform other systemactivities. This time is part ofused timeused to calculate%USEDabove.%OVRLPThe Percentage of system time that was spent on behalf of some other Resource Pool/Worldwhile Resource Pool/World was scheduled. This time is not included in the %SYS forResource Pool/World. For example, if Resource Pool A was currently scheduled and a networkpacket for Resource Pool B was being processed by the ESXi VMkernel, then the timespent doing so appears as %OVRLP for Resource Pool A and %SYS for Resource Pool B.%RUNPercentage of total time scheduled. This time does not account for hyper-threading andsystem time. Hence, on a hyper-threading enabled server, the %RUN can be twice as large as%USED.%WAITThe total percentage of time the Resource Pool/World spent inwaitstate. I.e., the world is waiting for some VMkernel resource.Notethat this percentage includes the percentage of time theResource Pool/World wasidle.%VMWAITThe total percentage of time the Resource Pool/World spent inblockedstate waiting for events.%CSTPThe percentage of time the Resource Pool/World spent inready, co-deschedulestate. This statistic is forinternal to VMware use only.%IDLEThe percentage of time the Resource Pool/World was idle. Subtracting thispercentage from%WAITabove gives you the percentage of time the Resource Pool/World was waitingon some event.%RDYThe percentage of time the Resource Pool/World was ready to run.%MLMTDPercentage of time the ESXi VMkernel deliberately did notrun the Resource Pool/World because that would violate the Resource Pool/World'slimitsetting. Since the Resource Pool/World is ready to run when it isprevented from running in this way, the %MLMTD (max limited) timeis included in %RDY time.%SWPWTPercentage of time the Resource Pool/World was waiting for the ESXi VMkernelswapping memory. The %SWPWT (swap wait) time is included in the %WAIT time.EVENT COUNTS/sThis is a set of CPU statistics made up of the following, per second,event rates. These statistics are forinternal to VMware use only.CPU ALLOCThis is a set of CPU statistics made up of the following CPU allocationconfiguration parameters.AMINResource Pool/World attributereservation.AMAXResource Pool/World attributelimit.ASHRSResource pool attributeshares.SUMMARY STATSThis is a set of CPU statistics made up of the following CPUconfiguration parameters and statistics.%LAT_CThe percentage of time the Resource Pool/World was ready to run but was notscheduled to run because of cpu resource contention.%LAT_MThe percentage of time the Resource Pool/World was ready to run but was notscheduled to run because of memory resource contention.%DMDThe CPU demand in percentage. It represents the average active CPU loadin the past 1 minute.EMINThe Effective Min in MHz for the Resource Pool/World. The amount ofCPU resources guaranteed to the world if all the worlds on the systemstart contending for CPU resources. ESXi VMkernel dynamically calculatesthe EMIN value for all worlds based on the resource settings(Reservations, Limits and Shares) of all the resource pools and VMson a system.This statistic is forinternal to VMware use only.TIMER/sThe timer rate the World is currently requesting for.AFFINITY BIT MASKBit mask showing the current scheduling affinity for the World.CPUThe physical or logical processor the World was found to be running on.HTSHARINGcurrent hyper-threading configuration.HTQWorld is currentlyquaratinedor not. 'N' implies no and 'Y' implies yes.POWERThe current CPU power consumption in Watts accounted for a Resource Pool.
Interactive Commands
eExpand/Rollup Resource Pool/Worlds CPU statistics. Allows viewing CPUresource utilization statistics broken down by individual Worldsbelonging to a Resource Pool.USort Resource Pools/Worlds by
Resource Pool %USEDcolumn. This is the default sort order.RSort Resource Pools/Worlds by
Resource Pool %RDYcolumn.NSort Resource Pools/Worlds by
GIDcolumn.VDisplay
VMinstances only.
Memory Screen
This screen displays server-wide and per Resource Pool memory utilizationstatistics. As on the
CPUscreen, Resource Pools correspond to running Virtual Machines or Worldsthat consume memory. Note the distinction between
machinememory and
physicalmemory in this section.
Physicalmemory is the virtual hardware physical memory presented to the VM.
Machinememory is actual physical RAM in the ESXi host.
Statistics Descriptions
PMEM (MB)This line displays the machine memory statistics for the server.All numbers are in
megabytes.totalis for the total amount of machine memory in the server,
cosis for the amount of machine memory allocated to the ESXiService Console,
vmkis for the amount of machine memory being used by theESXi VMkernel,
otheris for the amount of machine memory being used byeverything other than the ESXi Service Console andESXi VMkernel and
freeis for the amount of machine memory that is free.VMKMEM (MB)This line displays the machine memory statistics for the ESXiServer VMkernel.All numbers are in
megabytes.managedis for the total amount of machine memory managed by the ESXiServer VMkernel,
minfreeis for the minimum amount of machine memory that the ESXiVMkernel would like to keep free,
rsvdis for the amount of machine memory that is currently reserved,
ursvdis for the amount of machine memory that is currently unreservedand
stateis for the memory state as reported by the ESXi VMkernel.Possible values are
high, soft, hard and low.Here, high implies that the machine memory is not under any pressure andlow implies that the machine memory is under pressure. While the host'smemory state is not used to determine whether memory should be reclaimedfrom VMs (that decision is made at the resource pool level), it can affectwhat mechanisms are used to reclaim memory if necessary. In the high andsoft states, ballooning is favored over swapping. In the hard and lowstates, swapping is favored over ballooning.COSMEM (MB)This line displays the memory statistics as reportedby the ESXi Service Console.All numbers are in
megabytes.freeis for the amount of idle machine memory,
swap_tis for the total swap configured,
swap_fis for the amount of swap free,
r/sis for the rate at which memory is swapped in from disk and
w/sis for the rate at which memory is swapped to disk.NUMA (MB)This line displays the ESXi
NUMAstatistics. This line is only displayed if ESXiis running on a NUMA server. All numbers are in
megabytes.For each
NUMA nodein the server there are
twostatistics that are displayed. The first statistic is the
totalamount of machine memory in the NUMA node that is managed byESXi. The second statistic, that is displayed withinround brackets, is the amount of machine memory in the nodethat is currently
free.PSHARE (MB)This line displays the ESXi
page-sharingstatistics.All numbers are in
megabytes.sharedis for the amount of physical memory that is beingshared,
commonis for the amount of machine memory that iscommon across World(s) and
savingis for the amount of machine memory that is saveddue to page-sharing. The monitor maps guest physical memory to machine memory.VMkernel selects to map guest physical pages with the same content to the samemachine page. In other words, those guest physical pages are sharing the samemachine page. This kind of sharing can happen within the same VM or among the VMs.Since each VM's "shared" memory measures guest physical memory, the host's"shared" memory may be larger than the total amount of machine memory if memoryis overcommitted. "saving" illustrates the effectiveness of page sharing forsaving machine memory.SWAP (MB)This line displays the ESXi
swapusage statistics.All numbers are in
megabytes.curris for the current swap usage,
rclmtgtis for where ESXi expects the relaimed memory using swapping and compression to be,
r/sis for the rate at which memory is swapped in by ESXifrom disk and
w/sis for the rate at which memory is swapped to disk by the ESXiServer.
Note thatrclmtgtis the total target for both swapping and compression.We should compare it against the sum of thecurrswapped memory and thesavedmemory due to memory compression shown in theZIPline.
ZIP (MB)This line displays the ESXimemory compressionstatistics.All numbers are inmegabytes.zippedis for the total compressed physical memory,savedis for the saved memory by compression.MEMCTL (MB)This line displays thememory balloonstatistics.All numbers are inmegabytes.curris for the total amount of physical memory reclaimed using the vmmemctlmodules,targetis for the total amount of physical memory ESXi would like to reclaimusing the vmmemctl modules andmaxis for the maximum amount of physical memory ESXi can reclaim usingthe vmmemctl modules.GIDResource PoolIDLWIDTheLeader World IDof Resource Pool.NAMEResource PoolnameAMINResource Pool attributereservation.AMAXResource Pool attributelimit.ASHRSResource Pool attributeshares.NHNCurrentHome Nodefor Resource Pool. This statistic is only applicable on NUMA systems.NMIGNumber of NUMA migrations between two snapshots. It includes balancemigration, inter-node VM swaps performed for locality balancingand load balancing. This statistic is only applicable on NUMA systems.NRMEM (MB)Current amount ofremote memorybeing accessed by Resource Pool. This statistic is only applicable on NUMA systems.NLMEM (MB)Current amount oflocal memorybeing accessed by Resource Pool. This statistic is only applicable on NUMA systems.N%LCurrent percentage memory being accessed by Resource Pool that islocal.This statistic is only applicable on NUMA systems.GST_NDx (MB)The guest memory being allocated for Resource Pool on NUMA node x."x" is the node number.This statistic is only applicable on NUMA systems.OVD_NDx (MB)The VMM overhead memory being allocated for Resource Pool on NUMA node x."x" is the node number.This statistic is only applicable on NUMA systems.MEMSZ (MB)The amount of physical memoryallocatedto a Resource Pool.GRANT (MB)The amount of physical memorygrantedto a Resource Pool.It is the mapped guest physical memory. GRANT - SHRDSVD is theconsumed host machine memory. For a user world, it is the amount of virtual memorythat is backed by machine memory.SZTGT (MB)The amount of machine memory the ESXi VMkernel wants to allocate to Resource Pool.It includes the overhead memory for a VM.This is an internal counter, which is computed by ESXi memory scheduler. Usually,there is no need to worry about this. Roughly speaking,SZTGTof all the VMs iscomputed based on the resource usage, available memory, and the"limit/reservation/shares" settings. This computedSZTGTis compared against the current memory consumption plus overhead memory for a VM todetermine the swap and balloon target, so that VMkernel may balloon or swapappropriate amount of memory to meet its memory demand.TCHD (MB)Theworking setestimate for Resource Pool. VMkernel estimates active memory usage for a VM bysampling a random subset of the VM's memory resident in machine memory to detect thenumber of memory reads and writes. VMkernel then scales this number by the size ofVM's configured memory and averages it with previous samples. Over time, this averagewill approximate the amount of active memory for the VM.Notethat ballooned memory is considered inactive, so, it is excluded fromTCHD.TCHD_W (MB)Thewrite working setestimate for Resource Pool.%ACTVPercentage of guest physical memory that is being referencedby the guest. This is aninstantaneousvalue.%ACTVSPercentage of guest physical memory that is being referencedby the guest. This is aslowmoving average.%ACTVFPercentage of guest physical memory that is being referencedby the guest. This is afastmoving average.%ACTVNPercentage of guest physical memory that will be referencedby the guest. This is an estimation. This statistic is forinternal to VMware use only.MCTL?memory balloon driver is installed or not. 'N' impliesno and 'Y' imples yes.MCTLSZ (MB)The amount of physical memory reclaimed from Resource Poolby way ofballooning.This can be called "balloon size". A largeMCTLSZ
means lots of this VM's guest physical memory is "stolen" to decrease host memorypressure. This usually is not a problem, because balloon driver tends to smartlysteal guest physical memory that cause little performance problems.MCTLTGT (MB)The amount of physical memory ESXi would like to reclaim from Resource Poolby way of ballooning.This is an internal counter, which is computed by ESXi memory scheduler. Usually,there is no need to worry about this.MCTLMAX (MB)Themaximumamount of physical memory ESXi can reclaim from Resource Pool by way of ballooning.This maximum is dependent on guest operating system type.SWCUR (MB)Current swap usage by Resource Pool.SWTGT (MB)Where ESXi expectd the Swap usage by the Resource Pool to be.SWR/s (MB)Rate at which memory is being swapped in by ESXi from disk for the Resource Pool.When a VM is requesting machine memory to back its guest physical memory that wasswapped out to disk, VMkernel reads in the page. The swap-in operation is synchronous.SWW/s (MB)Rate at which Resource Pool memory is being swapped to disk by ESXi. This statsrefers to the VMkernel swapping, not the guest swapping.LLSWR/s (MB)Rate at which memory is faulted from host cache.LLSWW/s (MB)Rate at which memory is written to the host cache from various sources.CPTRD (MB)Amount of data read from checkpoint file.CPTTGT (MB)Size of checkpoint file.ZERO (MB)Resource Pool physical pages that are zeroed.SHRD (MB)Resource Pool physical pages that are shared. VMkernel page sharing module scansand finds guest physical pages with the same content and backs them with the samemachine page. It accounts the total guest physical pages that are shared bythe page sharing module.SHRDSVD (MB)Machine pages that are saved due to Resource Pool shared pages. Because a machinepage is shared by multiple guest physical pages, we only charge "1/ref" page asthe consumed machine memory for each of the guest physical pages, where "ref" isthe number of references. So, the saved machine memory will be "1 - 1/ref" page.COWH (MB)This statistic is forinternal to VMware use only.OVHDUW (MB)Current space overhead for the user world. This statistic is forinternal to VMware use only.OVHD (MB)Current space overhead for Resource Pool. It includes the overhead memory consumedby the monitor, the VMkernel and the vmx user world.OVHDMAX (MB)Maximum space overhead that may be incurred by Resource Pool. It is the overheadmemory a VM wants to consume in the future. This amount of reserved overhead memoryincludes the overhead memory reserved by the monitor, the VMkernel, and the vmxuser world.MEM Committed (MB)This is a set of memory statistics made up of minimum memorycommitment target, current memory commitment target, charged memorycommitment, and pages per share. These statistics are forinternal to VMware use only.CACHESZ (MB)Compression memory cache size.CACHEUSD (MB)Used compression memory cache.ZIP/s (MB/s)Compressed memory per second.UNZIP/s (MB/s)Decompressed memory per second.
Interactive Commands
VDisplay
VMinstances only.MSort Resource Pools by
Resource Pool MEMSZcolumn. This is the default sort order.BSort Resource Pools by
Resource Pool MCTLSZcolumn.NSort Resource Pools by
GIDcolumn.
Storage Adapter Screen
This screen displays server-wide storage utilizationstatistics. On this screen statistics are aggregated perstorage
adapterby default. Statistics can also be viewed per storage
path.See below for more information.
Statistics Descriptions
ADAPTRThe
nameof the storage adapter.PATHThe storage
pathname. This name is only visible if the corresponding adapteris expanded. See interactivecommand 'e' below.When the statistics are expanded to path level, esxtop showsthe path statistics.NPTHSThe
number of paths.AQLENThe
storage adapter queue depth.This is the maximum number of ESXi VMkernelactive commands that the adapter driver is configuredto support.CMDS/sThe number of
commandscompleted per second.READS/sThe number of
read commandscompleted per second.WRITES/sThe number of
write commandscompleted per second.MBREAD/sThe megabytes
readper second.MBWRTN/sThe megabytes
writtenper second.RESV/sThe number of SCSI reservations per second.CONS/sThe number of SCSI reservation conflicts per second.DAVG/cmdThe
average device latency (millisecs)per command. It is a good indicator of performance of the backend storage.If IO latencies are suspected to be causing performance problems,
DAVGshould be examined. Compare IO latencies with corresponding data from thestorage array. If they are close, check the array for misconfiguration orfaults. If not, compare DAVG with corresponding data from points inbetween the array and the ESXi Server, e.g., FC switches.If this intermediate data also matches
DAVGvalues, it is likely that the storage is under-configured for the application.Adding disk spindles or changing the RAID level may help in such cases.KAVG/cmdThe
average ESXi VMkernel latency (millisecs)per command. It's value should be very small in comparison to the
DAVGvalue and should be close to zero. When there is a lot of queuing in ESXi,
KAVG can be as high, or even higher thanDAVG.GAVG/cmdThe
average Guest OS latency (millisecs)per command.QAVG/cmdThe
average queue latency (millisecs)per command. Response time is the sum of the time spent in queues in the storagestack and the service time spent by each resource in servicing the request. Thelargest component of the service time is the time spent in retrieving data fromphysical storage. If
QAVGis high, another line of investigation is to examine the queue depths at eachlevel in the storage stack.DAVG/rdThe
average device read latency (millisecs)per read.KAVG/rdThe
average ESXi VMkernel read latency (millisecs)per read.GAVG/rdThe
average Guest OS read latency (millisecs)per read.QAVG/rdThe
average queue read latency (millisecs)per read.DAVG/wrThe
average device write latency (millisecs)per write.KAVG/wrThe
average ESXi VMkernel write latency (millisecs)per write.GAVG/wrThe
average Guest OS write latency (millisecs)per write.QAVG/wrThe
average queue write latency (millisecs)per write.FCMDS/sFailed commands per second.FREAD/sFailed
readcommands per second.FWRITE/sFailed
writecommands per second.FMBRD/sThe megabytes
readper second by failed commands.FMBWR/sThe megabytes
writtenper second by failed commands.FRESV/sThe number of
reservecommands failed per second.ABRTS/sThe number of commands
abortedper second. It can indicate that the storage system is unable to meet thedemands of the guest operating system. Abort commands are issued by theguest when the storage system has not responded within an acceptable amountof time, e.g. 60 seconds on some windows OS's. Also, resets issued by aguest OS on its virtual SCSI adapter will be translated to aborts of all thecommands outstanding on that virtual SCSI adapter.If multiple paths are connected to the same lun,this statistic is
per pathwhen it is expanded to luns.RESETS/sThe number of commands
resetper second.PAECMD/sThe number of
PAE commandsper second. It may point to hardware misconfiguration. When the guestallocates a buffer, the vmkernel assigns some machine memory, which mightcome from a .highmem. region. If you have a driver that is not PAE-aware,then this counter is updated if accesses to this memory region result incopies by the vmkernel into a lower memory location before issuing therequest to the adapter. This might happen if you do not populate the DIMMswith low memory first, then you may artificially cause .highmem. memoryaccesses.PAECP/sThe number of
PAE copiesper second.SPLTCMD/sThe number of
split commandsper second. Commands can be split when they reach the vmkernel. This mightimpact perceived latency to the guest. The guest may be issuing commands oflarge block sizes which have to be broken down by the vmkernel. Splittingcan also occur when IOs fall across partition boundaries but these areeasily differentiated from the splitting as a result of the IO size.SPLTCP/sThe number of
split copiesper second.
Interactive Commands
eExpand/Rollup storage
adapterstatistics.Allows viewing storage resource utilization statisticsbroken down by individual
pathsbelonging to an expanded storage adapter. You will beprompted to enter the adapter name.rSort by
READS/scolumn.wSort by
WRITES/scolumn.RSort by
MBREAD/scolumn.TSort by
MBWRTN/scolumn.NSort first by
ADAPTERcolumn, then by
PATHcolumn.This is the default sort order.
Storage Device Screen
This screen displays server-wide storage utilizationstatistics. On this screen statistics are aggregated perstorage
deviceby default. Statistics can also be viewed per
path,world,or
partition.See below for more information.
Statistics Descriptions
DEVICEThe
nameof the storage device.PATHThe path name. This name is only visible if the correspondingdevice is expanded to paths. See interactive command 'p' below.WORLDThe world id. This id is only visible if the corresponding deviceis expanded to worlds. See interactive command 'e' below.The world statistics are per world per device.PARTITIONThe partition id. This id is only visible if the correspondingdevice is expanded to partitions. See interactive command 't' below.NPHThe
number of paths.NWDThe
number of worlds.NPNThe
number of partitions.SHARESThe
number of shares.This statistic is only applicable to worlds.BLKSZThe
block sizein bytes.NUMBLKSThe
number of blocksof the device.DQLENThe
storage device queue depth.This is the maximum number of ESXi VMkernelactive commands that the device is configuredto support.WQLENThe
World queue depth.This is the maximum number of ESXi VMkernel activecommands that the World is allowed to have.
Notethat this is a per device maximum for the World.It is valid only if the corresponding device is expanded to worlds.ACTVThe number of commands in the ESXi VMkernel thatare currently
active.This statistic is only applicable to worlds and devices.QUEDThe number of commands in the ESXi VMkernel thatare currently
queued.This statistic is only applicable to worlds and devices. Queued commandsare commands waiting for an open slot in the queue. A large number ofqueued commands may be an indication that the storage system isoverloaded. A sustained high value for the QUED counter signals a storagebottleneck which may be alleviated by increasing the queue depth. Checkthat LOAD < 1 after increasing the queue depth. This should also beaccompanied by improved performance in terms of increased cmd/s.%USDThe
percentageof queue depth
usedby ESXi VMkernel active commands.This statistic is only applicable to worlds and devices.
%USDis a measure of how many of the available command queue "slots"are in use. Sustained high values indicate the potential for queueing;you may need to adjust the queue depths for system's HBAs if
QUEDis also found to be consistently > 1 at the same time. Queue sizes canbe adjusted in a few places in the IO path and can be used to alleviateperformance problems related to latency.LOADThe
ratioof ESXi VMkernel active commands plus ESXi VMkernel queued commands
toqueue depth.This statistic is only applicable to worlds and devices.CMDS/sThe number of
commandsissued per second.READS/sThe number of
read commandsissued per second.WRITES/sThe number of
write commandsissued per second.MBREAD/sThe megabytes
readper second.MBWRTN/sThe megabytes
writtenper second.RESV/sThe number of SCSI reservations per second.CONS/sThe number of SCSI reservation conflicts per second.DAVG/cmdThe
average device latency (millisecs)per command.KAVG/cmdThe
average ESXi VMkernel latency (millisecs)per command.GAVG/cmdThe
average Guest OS latency (millisecs)per command.QAVG/cmdThe
average queue latency (millisecs)per command.DAVG/rdThe
average device read latency (millisecs)per read.KAVG/rdThe
average ESXi VMkernel read latency (millisecs)per read.GAVG/rdThe
average Guest OS read latency (millisecs)per read.QAVG/rdThe
average queue read latency (millisecs)per read.DAVG/wrThe
average device write latency (millisecs)per write.KAVG/wrThe
average ESXi VMkernel write latency (millisecs)per write.GAVG/wrThe
average Guest OS write latency (millisecs)per write.QAVG/wrThe
average queue write latency (millisecs)per write.ABRTS/sThe number of commands
abortedper second.RESETS/sThe number of commands
resetper second.PAECMD/sThe number of
PAE commandsper second.This statistic is only applicable to paths.PAECP/sThe number of
PAE copiesper second.This statistic is only applicable to paths.SPLTCMD/sThe number of
split commandsper second.This statistic is only applicable to paths.SPLTCP/sThe number of
split copiesper second.This statistic is only applicable to paths.CLONE_RDThe number of
clone readcommands.CLONE_WRThe number of
clone writecommands.CLONE_FThe number of
failed clonecommands.MBC_RD/sThe megabytes
clone readper second.MBC_WR/sThe megabytes
clone writtenper second.ATSThe number of
atscommands.ATSFThe number of
failed atscommands.ZEROThe number of
zerocommands.ZERO_FThe number of
failed zerocommands.MBZERO/sThe megabytes
zeroedper second.CAVG/sucThe
average clone latency (millisecs)per successful command.CAVG/fThe
average clone latency (millisecs)per failed command.AAVG/sucThe
average ats latency (millisecs)per successful command.AAVG/fThe
average ats latency (millisecs)per failed command.ZAVG/sucThe
average zero latency (millisecs)per successful command.ZAVG/fThe
average zero latency (millisecs)per failed command.
Interactive Commands
eExpand/Rollup storage
worldstatistics.Allows viewing storage resource utilization statisticsbroken down by individual
worldsbelonging to an expanded storage device. You will beprompted to enter the device name. The statistics areper world per device.PExpand/Rollup storage
pathstatistics. Allows viewing storage resource utilizationstatistics broken down by individual
pathsbelonging to an expanded storage device. You will beprompted to enter the device name.tExpand/Rollup storage
partitionstatistics. Allows viewing storage resource utilizationstatistics broken down by individual
partitionsbelonging to an expanded storage device. You will beprompted to enter the device name.rSort by
READS/scolumn.wSort by
WRITES/scolumn.RSort by
MBREAD/scolumn.TSort by
MBWRTN/scolumn.NSort first by
DEVICEcolumn, then by
PATH/WORLD/PARTITIONcolumn.This is the default sort order.
Storage VM Screen
This screen displays VM-centric storage statistics. On thisscreen, statistics are aggregated on a
per-resource-poolbasis by default. One VM has one corresponding resource pool,so, they are equivalent to per-VM statistics.Statistics can also be viewed on a
per-vscsi-devicebasis.
Statistics Descriptions
IDThe
Resource Pool IDof the Resource Pool or
Vscsi IDof Vscsi Device.GIDThe
Resource Pool IDof Resource Pool.VMNAMEThe
nameof Resource Pool.VDEVNAMEThe
nameof the Virtual Device.NVDISKThe number of
vscsi devices.CMDS/sThe number of
commandsissued per second.READS/sThe number of
read commandsissued per second.WRITES/sThe number of
write commandsissued per second.MBREAD/sThe megabytes
readper second.MBWRTN/sThe megabytes
writtenper second.LAT/rdThe
average latency (millisecs)per read.LAT/wrThe
average latency (millisecs)per write.
Interactive Commands
eExpand/Rollup storage
vscsi devicestatistics.Allows viewing storage resource utilization statisticsbroken down by individual
vscsi devicesbelonging to a group. You will beprompted to enter the group id. The statistics areper
vscsi device.rSort by
READS/scolumn.wSort by
WRITES/scolumn.RSort by
MBREAD/scolumn.TSort by
MBWRTN/scolumn.NSort first by
VMcolumn, then by
VSCSIcolumn.This is the default sort order.
Network Screen
This screen displays server-wide network utilization statistics.On this screen statistics are arranged per
portper configured
virtual network device.For
physical NICstatistics, see the row corresponding to the port that the physicalNIC is connected to. For
virtual NICconfigured in a particular Virtual Machine statistics, see the rowcorresponding to the port that the virtual NIC is connected to.
Statistics Descriptions
PORT-IDThe virtual network device
port id.UPLINK
Yimplies the corresponding port is an uplink.
Nimplies it is not.UP
Yimplies the corresponding link is up.
Nimplies it is not.SPEEDThe
link speedin MegaBits per second.FDUPLX
Yimplies the corresponding link is operating at full duplex.
Nimplies it is not.USED-BYThe virtual network device port
user.TEAM-PNICThe physical NIC name for the team uplink.DNAMEThe virtual network device
name.PKTTX/sThe number of packets
transmittedper second.PKTRX/sThe number of packets
receivedper second.MbTX/sThe MegaBits
transmittedper second.MbRX/sThe MegaBits
receivedper second.PSZTXThe average packet size
transmittedin bytes.PSZRXThe average packet size
receivedin bytes.%DRPTXThe percentage of
transmitpackets
dropped.%DRPRXThe percentage of
receivepackets
dropped.ACTN/sNumber of
actionsper second. This statistic is for
internal to VMware use only.PKTTXMUL/sThe number of multicast packets
transmittedper second.PKTRXMUL/sThe number of multicast packets
receivedper second.PKTTXBRD/sThe number of broadcast packets
transmittedper second.PKTRXBRD/sThe number of broadcast packets
receivedper second.
Interactive Commands
TSort by
MbTX/scolumn.RSort by
MbRX/scolumn.tSort by
PKTTX/scolumn.rSort by
PKTRX/scolumn.NSort by
PORT IDcolumn. This is the default sort order.
Interrupt Screen
This screen displays interrupt utilization statistics.On this screen statistics are arranged per
interrupt cookie.Statistics Descriptions
COOKIEThe
interrupt cookie.COUNT/sThe
total number of interrupts per secondacross all the CPUs. It measures how often an interrupt is raised on the
device.E.g., If you have 2 CPUs, COUNT/s = COUNT_0 + COUNT_1.COUNT_0 and COUNT_1 are described below.COUNT_xThe
number of interrupts per secondon CPU 'x'. This is a per CPU counter. Comparing "COUNT_x" for the sameinterrupt cookie on different CPUs can tell us how balanced theinterrupts are scheduled across all the CPUs.TIME/intThe
average processing timein microseconds per interrupt. It is the average for all the interruptsof the same cookie.TIME_xThe
average processing timein microseconds per interrupt on CPU 'x'. It averages only theinterrupts raised on CPU 'x'.DEVICESThe devices that use the interrupt cookie.If the interrupt cookie is not enabled for the device, its nameis enclosed in "<>", e.g. "<VMK device>".
Power Screen
This screen displays CPU Power utilization statistics. On thisscreen statistics are arranged per
PCPU.A
PCPUrefers to a physical hardware execution context -- a physicalCPU core if hyper-threading is unavailable or disabled, ora logical CPU (aka LCPU or SMT thread) if hyper-threading is enabled.
Statistics Descriptions
Power UsageThe current total power usage in Watts.Power CapThe total power cap in Watts.%USEDThe percentage of PCPU nominal frequency that was usedsince the last screen update. It is the same as
PCPU USED(%)shown in the CPU Screen.%UTILThe raw PCPU utilization, that is the percentage of realtime that PCPU was not idle. It is the same as
PCPU UTIL(%)shown in the CPU Screen.%CxThe percentage of time the PCPU spent in C-State 'x'.%PxThe percentage of time the PCPU spent in P-State 'x'.%TxThe percentage of time the PCPU spent in T-State 'x'.
VSAN Screen
This screen displays VSAN statistics.
Statistics Descriptions
VSAN EnabledWhether VSAN is currently enabled on the host.ROLEThe
nameof VSAN DOM Role.READS/sNumber of read operations completed per second.MBREAD/sMegabytes read per second.WRITES/sNumber of write operations completed per second.MBWRITE/sMegabytes written per second.RECOWR/sNumber of recovery write operations completed per second.MBRECOWR/sMegabytes written for recovery per secondSDLATStandard deviation of latency in millisecs for read, write and recovery write.AVGLATAverage latency in millisecs for read, write and recovery write.
Batch Mode
Allows collecting and saving resource utilization statisticsin a file. Running in this mode is a two step process.
Step
onerequires running esxtop in interactive mode, switching to each ofthe nine available screens, selecting the columns on each screenyou are interested in and saving this configuration in the
~/.esxtop60rcfile using the
Winteractive command. See
Interactive Modeabove.
Notethat the global stats are always available in batch mode.
Step
tworequires running esxtop in batch mode and redirecting the output toa file. This may be done as follows:
esxtop -b > foo.csv.Notethat the file name needs to have a
.csvextension. Esxtop does not enforce this, but post-processing toolsmentioned next require this.
Statistics collected in batch mode may be post-processed usingsuch
MS Windowsapplications like
Exceland
Perfmon. In this mode, esxtop will not acceptinteractive commands. Esxtop in batch mode runs until it produces thenumber of iterations requested (seecommand-line option
nbelow for more details) or until killed using
CTRL c. In this mode there are several command line options available.
Command-line Options
bRun esxtop in Batch mode.dSpecifies the
delaybetween statistics snapshots. Default is
5seconds. Minimum is
2seconds. If a delay of less than 2 seconds is specified then thedelay is set to 2 seconds.nNumber of
iterations.Collect and save statistics this number of times and then exit.
Replay Mode
Replays resource utilization statistics collected using
vm-support(see vm-support man page for more information), resxtop does notsupport replay mode. Running in this mode is a two step process.
Step
onerequires running vm-support in
snapshot modeon the ESXi Service Console. This may be done as follows:
vm-support -S -d duration -i interval.The resulting zipped tar file needs to be un-zipped and un-tar'edbefore use by esxtop replay mode.
Step
tworequires running esxtop in replay mode. This may be done as follows:
esxtop -Rvm-support_dir_path.Notethat it is not required to run esxtop replay mode on the ESXiService Console.
In replay mode, esxtop needs to be of the same verison asthat of ESXi to avoid compatibility problem.
Replay mode can be run to produce
Batch Modestyle output (see command-line option
bbelow for more information).
In replay mode, esxtop accepts the same set of interactive commandsas in
Interactive Mode.Esxtop in replay mode runs until there are no more vm-supportcollected snapshots to be read or esxtop produces the number ofiterations requested (see command-line option
nbelow for more details).
In this mode there are several command line options available.
Command-line Options
RSpecifies the path to the vm-support collected snapshot's directory.bRun esxtop in Batch mode.dSpecifies the
delaybetween screen updates. Default is
5seconds. Minimum is
2seconds. If a delay of less than 2 seconds is specified then thedelay is set to 2 seconds.nNumber of
iterations.Update the display this number of times and then exit.
EXPERIMENTAL FEATURES
Export and Import Entities
This feature allows users show only the entities that they areinterested in, such as groups, devices, adapters, ports,interrupt cookies, etc. The ability of selecting interestingentities can also reduce the CPU load of esxtop itself.Two command line options are introduced for this purpose:[
export-entityentity-file]and[
import-entityentity-file].
Users can do three steps to select entities which they wantto display.
(1) Users first export the list of current entities usingthe export-entity option, by runningesxtop -export-entity entity-file.The entity-file is generated, including the ids of all theexisting entities at that time.
(2) Then, they can edit the generatedentity-fileto delete the entities by inserting#before their names.
(3) At last step, users can importentity-fileusing the import-entity option, by runningesxtop -import-entity entity-file.Therefore, esxtop will only show the stats for the entitiesselected in the entity-file.
Note that these two options are only available in esxtop,not in resxtop.
Entity Highlight/Expand/Deletion Interactive Operations
Under interactive mode, users can use keys in numpad to highlight,expand and delete entities.
User can press'8'to move up the highlight cursor, and'2'to move down the highlight cursor.
Users can press'6'to can expand/unroll the selected entity when appropriate.In CPU Screen, it expands/rollups ResourcePool/WorldsCPU statistics.In Storage Device Screen, pressing '6' can rotate the threedifferent expanding modes and rollup modes. Pressing '6' thefirst time expands storageworldstatistics; pressing '6' the second time expands storagepathstatistics; pressing '6' the third time expands storagepartitionstatistics; pressing '6' the fourth time rollups to thedefault view.In Disk VM Screen, pressing '6' can expand/rollup ResourcePool/WorldsDisk statistics.
Users can press'4'to remove the highlighted entity from display, which willdecrease the amount of stats we try to collect.(Note that this delete operation is only available in esxtop,not in resxtop.)
Users can press'.'to restore all the entities and we will collect data for allof them again.
NOTES:
Note that the disk VM screen shows the storage stats for a VM.The selection of group entities and disk device entities haveeffect to this screen. If the group is not chosen, it will not showup. If the device is not chosen, all operations on this devicewill be counted as zeros.
FILES
~/.esxtop60rcThe personal configuration file.
SEE ALSO
vm-support(1),
cpu(8),
mem(8),
diskbw(8),
numa(8)
COPYRIGHT
VMware ESXi is Copyright 1998-2014 VMware, Inc. All rights reserved.
DISCLAIMER
Based on top source code from the OpenBSD distribution.
Copyright (c) 1997, Jason Downs. All rights reserved.
Copyright (c) 1984, 1989, William LeFebvre, Rice University
Copyright (c) 1989, 1990, 1992, William LeFebvre,
Northwestern University
Redistribution and use in source and binary forms, with or withoutmodification, are permitted provided that the following conditionsare met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESSOR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIEDWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE AREDISCLAIMED. IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT,INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS ORSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVERCAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICTLIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAYOUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OFSUCH DAMAGE.
Index
- NAME
- SYNOPSIS
- DESCRIPTION
- Interactive Mode
- Command-line Options
- Common Statistics Description
- Common Interactive Commands
- The Statistics Columns (fields) and Order Screens
- Configuration File
- CPU Screen
- Memory Screen
- Storage Adapter Screen
- Storage Device Screen
- Storage VM Screen
- Network Screen
- Interrupt Screen
- Power Screen
- VSAN Screen
- Batch Mode
- Command-line Options
- Replay Mode
- Command-line Options
- EXPERIMENTAL FEATURES
- Export and Import Entities
- Entity Highlight/Expand/Deletion Interactive Operations
- NOTES:
- FILES
- SEE ALSO
- COPYRIGHT
- DISCLAIMER
This document was created byman2html,using the manual pages.
