LILO is the only supported boot loader for ESX, so don't replace it with any other Linux loader, e.g. GRUB.

The lilo.conf file is the configuration text file that defines how the Linux OS will boot. If you are familiar with Windows, then this file is similar to BOOT.INI. However, in contrast to the Windows file, the lilo.conf text file is compiled into a binary file, and it is that binary file which is actually used by LILO at boot time.

Here is a sample section of a lilo.conf file. You can see the initrd line which specifies the ramdisk image that the boot loader uses to load the Linux service console kernel. The Linux kernel image name is vmnix and many VMware administrators use the term vmnix when referring to the service console.

image=/boot/vmlinuz-2.4.9-vmnix2
     label=esx
     root=/dev/sda2
     initrd=/boot/initrd-2.4.9-vmnix2.img
     read-only
     append="mem=272M cpci=0:*;1:*;2:*;4:*;12:;16:*;"

  If you are troubleshooting the APPEND line, then use vmkpcidivy tool. You should not have to revert to manually editing this file. If you ever do edit this file, then you need to write those changes into the boot sector by running /sbin/lilo . If you are unsure the right changes will be made, you can do a trial run with the command /sbin/lilo -t .

The pci device mask specified in the append line of lilo.conf is actually an include, not a mask out. The important thing to note is that the append line defines the physical PCI bus hardware that is visible to the service console.

ESX manages allocation of PCI devices between service console and VMkernel with the expectation of the boot loader being LILO.

You can also view PCI device allocation using the MUI, whilst logged in as root. This is found under Startup Options in the Options tab of the MUI as shown below.

Alternatively, you could use the legacy MUI web interface using the URL

http://esxserver/pcidivy

Another alternative is to used the command vmkchdev -L.

The LILO boot loader has a boot prompt as well, displayed rather inconspicuously below the red text menu. It is at this boot prompt that you can supply additional boot parameters. You may wish to restrict LILO from accepting such user-entered boot parameters unless a password is entered.

password=<password>
restricted

If you only enter the password line to the file, then a password would be required to boot the system, if you also have the restricted option then you would only need the password for making boot modifications. In the LILO boot menu, any option that requires a password has a "P" next to the image name and any option with the restricted option has an "R" next to the image name.

If it is a concern that the /etc/lilo.conf file contains a password stored in clear text, the file should be secured using permissions that only allow root access, i.e. rwx------. You can implement this with the chmod command and the 600 numeric to represent rw.

id:3:initdefault:

The run level that the service console uses is run level 3, which specifies full multi-user mode. The init process then works through the start up scripts in the appropriate directory. For run level 3, this directory would be 

/etc/rc.d/rc3.d

The file also starts up the virtual terminals on the service console, mingetty tty2 through mingetty tty5.

The mingetty process is a manager of virtual terminals for Linux; it is a minimal version of universal getty found in UNIX. It does not support to connections of serial port connected terminals and is therefore "lighter" than getty and performs the majority of most terminal needs. In the past, when UNIX was deployed on large machines and dumb terminals were connected using serial connections, the getty service was used. Nowadays, almost nobody connects to a Linux machine by the serial port, and for that reason it was decided to lighten getty, adopting a "minimum getty" in many distributions of Linux.

This may be required when a server has only 2 physical NICs, but we really want 3, so we can dedicate 1 NIC to VMotion. To do this we add the following lines to the end of the rc.local file.

insmod vmxnet_console devName=vmnic0
ifup eth0

You can use the insmod utility to load driver modules either by explicitly stating the path and module file or by just the module name and insmod will locate the correct one. In the example above, the actual driver file is

/lib/modules/2.4.9-vmnix2/misc/vmxnet_console.o

If we do need to do this, then we need to decide which network functions should share a physical NIC (pNIC), for example

So, how you share your pNICs will depend on how much management traffic there is in relation to VM traffic as well as how often VMotion operations are likely to occur.

If you need to VLAN tag the service console traffic when using the vmxnet_console module, then you just add the VLAN ID number after the device name in rc.local. For example, to place the service console on VLAN number 105, we would modify the insmod line to read

insmod vmxnet_console devName=vmnic0.105

S00vmkstart
S10network
S11vmware
S12syslog
S55sshd
S56xinetd
S91httpd.vmware

By looking at the script titles we can guess what some of them do, e.g. S55 starts the secure shell daemon (putty in now!), S56 starts xinetd which amongst other things handles remote console sessions and then S91 starts, which gives us an Apache web server, known to us as simply as the MUI. If you would like to add your own scripts, you can place them anywhere in this start-up order. For example, if you wanted a script to start after xinetd but before the MUI, you could label it something like "S60custom".

A neat trick if you are looking to temporarily disable a start up script is to rename the file from capital "S" to lowercase "s".

ntpd   0:off 1:off 2:on  3:on  4:on  5:off 6:off
syslog 0:off 1:off 2:on  3:on  4:on  5:off 6:off
snmpd  0:off 1:off 2:off 3:off 4:off 5:off 6:off

If we wanted to change a service so that it is enabled for a particular run level, then we can use chkconfig –level.

chkconfig --level 1 ntpd on

The above command would turn on ntpd for run level 1, this would not affect the run levels that ntpd was already set for. So in this example, the ntpd run levels would be

ntpd 0:off 1:on 2:on 3:on 4:on 5:off 6:off

If we just want to turn on a daemon for the current run level we can just type the name of the service we want to enable/disable with on or off as a parameter. So to turn on nfs daemon for the current run level (whatever that may be) you would type:

chkconfig nfs on

If you are not sure what runlevel you are currently in, just use the command runlevel and the current runlevel will be displayed.

service –-status-all |grep running

would produce an output similar to the following:

crond (pid 1423) is running
httpd (pid 1486 1482 1479) is running
syslogd (pid 1136) is running
sshd (pid 1208) is running

  To avoid unnecessarily rebooting an ESX server after making certain configuration changes, we can frequently just restart the appropriate daemon. For example we could restart the Apache web server for the MUI with the command:

service httpd.vmware restart

and we can also check a named service running status with

service httpd.vmware status

The /etc/rc3.d/S12syslogd file is actually a logical link to the executable file in /etc/rc.d/init.d/syslog

logger -i -t username "This test message will appear in the service console log file!"

So now we could examine the last few lines of the service console log file to see our new entry:

tail /var/log/messages

If you were setting up logging from the service console of one ESX server to a centralised log server, then this would be a great way of testing that the centralised logging was working as expected.

A quick way to restart the SSH daemon is to enter:

# /etc/init.d/sshd restart

It is important to use the full path to the ssh daemon to do this. An easier way to do this is by using the service command

service sshd restart

The configuration of the SSH server daemon is stored in the text file /etc/ssh/sshd_config. An important setting in this file is PermitRootLogin=Yes/No. You can quickly check this with a grep on the file.

# grep Permit /etc/ssh/sshd_config

If you do edit the file, make sure you restart the service for the changes to take effect.
 

We can use it to gain a command line session with a remote host, typically the service console of another ESX Server. In the following example, we are logged on to the service console of ESX server "esx01" and we are opening a command line session with the service console of ESX server "esx02".

[root@esx01 root]# ssh esx02
The authenticity of host 'esx02 (192.168.22.32)' can't be established.
RSA key fingerprint is b0:d3:5f:87:65:6d:dd:29:be:49:e2:b5:1a:8e:db:37.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'esx02,192.168.22.32' (RSA) to the list of known hosts.
root@esx02's password:
Last login: Mon Apr 17 13:25:05 2006 from 172.16.110.204
[root@esx02 root]# exit
logout
[root@esx01 root]#

Once you have established an ssh session with another host, the known_hosts file on your server is populated.

The .ssh subdirectory is not created until you make an outbound ssh or scp connection to another host.

If you rebuild one of your ESX hosts, when you try to reconnect to it over ssh you may be prevented from connecting, if the known_hosts file has cached the old key. In the following command, we examine the contents of the known_hosts file (we've truncated the length of the key here!)

[root@esx1 root]# cat .ssh/known_hosts

esx02,192.168.22.32 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAocui7IApxnJevQgIPyIynde0SvVHRS02CM7ODFF7Mc/d <snip>

The -t switch specifies type

ssh-keygen -t dsa

Originally the inetd daemon helped in controlling network connections to a computer. When a request arrives at a TCP/UDP port that is  managed by inetd, the request is forwarded to a program called tcpd (/usr/sbin/tcpd). Then tcpd decides, in accordance with the rules contained in the hosts.{allow, deny} files whether or not to grant the request. If the request is allowed, then the the corresponding server process (e.g. ftp) can be started. This mechanism is also referred to as tcp_wrapper.

xinetd provides access control capabilities similar to the ones provided by tcp_wrapper.

The daemon itself is stored in /usr/sbin/xinetd This launches the daemons that are bound to it on demand. 

/etc/xinetd.d/vmware-authd

This text file contains the settings for the VMware remote access authentication daemon. This file specifies the TCP:902 port used by remote console.

   If this port was changed here, it must also be changed in the file /etc/vmware/config. Any changes must also be reflected in the remote console client settings and VirtualCenter.

If we wanted to add Kerberos off-box authentication for MUI access, then its in the pluggable authentication module configuration file that corresponds to this daemon that we would make a change. This file is found at

/etc/pam.d/vmware-authd

We would need to change the current "auth required" to "auth sufficient" and add a last line of "auth required" using the Kerberos authentication module. Modification may be required to the /etc/krb5.conf, /var/kerberos/krb5kdc/kdc.conf for server locations and /etc/hosts to resolve these server IP addresses.

vmnix driver   Loaded by modprobe vmnixmod.o
VMkernel       Loaded by vmkloader
Logger        
VMkdump        Any dump will be copied to /root
Starts VMs     Performed by vmware-serverd

This S11vmware file is actually a logical link file to the actual script which is stored in the file /etc/rc.d/init.d/vmware

If the automatic install of this component fails, it can be installed manually by copying the appropriate RPM package from the VirtualCenter server to the ESX host which is to be VC-managed.

Copy the RPM from C:\Program Files\VMware\VirtualCenter\ccagent\

to the ESX host and then from the command line run

rpm -Uav VMware-ccagent-esx-2.5.0

The most likely reason you would need to do this manual method is when the VC server is on a separate subnet from the ESX host and there is a firewall in-between. Even if TCP:902 is open between the subnets, some dynamic ports are temporarily required for this vmware-ccagent install.

If you are running ESX Server version 2.5.2 with VirtualCenter 1.3, you will no longer see the process vmware-ccagent. The original process name vmware-serverd remains even after adding the ESX host to a VirtualCenter farm.

If you are running ESX Server version 3 with VirtualCenter 2 (not released yet!) then you'll see something completely different.

/usr/lib/vmware-mui/apache/conf/httpd.conf

This process communicates with vmware-serverd for backend data. Remember a refresh in the browser is only a refresh to Apache, to get new data, click on the refresh button to get new kernel data. Remember if the httpd.vmware service starts and then stops immediately, check your service console disk space.

The S91httpd.vmware entry in /etc/rc3.d is a logical link to /etc/rc.d/init.d/httpd.vmware
 

The HTML files for the MUI can be found in the following path

/usr/lib/vmware-mui/apache/htdocs/vmware/en

The root of the VMware MIB is enterprises 6876

The Master SNMP agent (snmpd) can be replaced with the HP Insight Agent or Dell OpenManage as required.

The MIBs are stored on an ESX Server in the directory /usr/lib/vmware/snmp/mibs

# snmpsetup.sh default

This would produce an output similar to the following:

Stopping agents.
Stopping snmpd:                                            [FAILED]
Stopping vmware-snmpd:                                     [FAILED]

Checking for main agent.

Setting up basic config file.
Do you want to enable SNMP traps for virtual machine events? (y/n) y

Default trapsink is localhost.
You can modify /etc/snmp/snmpd.conf to set up a different trap destination.

Setup finished.
Restarting agents.
Starting snmpd:                                            [ OK ]
Waiting for master agent to start.
Starting vmware-snmpd as subagent:

You could then enable the Master SNMP Agent for required run-levels with

chkconfig snmpd on

Then enable the VMware SNMP SubAgent for required run-levels with

chkconfig vmware-snmpd on

Then we can start both SNMP daemons with

# service snmpd start
# service vmware-snmpd start

Also note, that if you are configuring snmp entirely from the command line, then you will also need to update the file /etc/vmware/config to include the text

serverd.snmpdconf.subagentenabled = "TRUE"

This is the configuration file for the Master SNMP Agent.

The following is the default contents of this file after ESX has been installed.

syscontact root@localhost (edit /etc/snmp/snmpd.conf)
syslocation room1 (edit /etc/snmp/snmpd.conf)
rocommunity public
trapcommunity public
trapsink localhost

Here is the output from lsmod

Module          Size   Used by Tainted: PF
vmnixmod        177056 121
e1000           68456  0 (unused)
usb-uhci        21220  0 (unused)
usbcore         50112  1 [usb-uhci]
megaraid2       32928  6

If a module has a tainted value of 1, this denotes the driver is not covered under the GNU license. The same information that lsmod produces can also be found by inspecting the file /proc/modules. We would do this with a tool such as cat. For example:

# cat /proc/modules

There is a different command which lists the driver modules that the VMkernel is using which is called vmkload_mod and can also be found in this guide.

shutdown -h now       Halt after shutdown
shutdown -r now       Restart after shutdown

esx            Normal ESX boot
linux          Linux SMP kernel, no VMkernel load
linux-up       Linux Uni-processor kernel, no VMkernel load

If we use the cursor key at the LILO screen to select one of the three default choices, the boot prompt (displayed below the menu) changes to reflect this. This allows us to augment the boot command with an option switch.

boot: linux –s

In this case, the –s instructs Linux to boot in single user mode. A critical security point here is that in single user mode, Linux automatically logs on as root! Once in single user mode if we wish to continue into multi-user mode then we type either exit or CTRL-D. To restrict access to single user mode, check the "restricted" parameter in the configuration file /etc/lilo.conf.

# rpm -qa
mailcap-2.1.6-1
setup-2.5.7-1
basesystem-7.0-2
bdflush-1.5-17
chkconfig-1.2.24-1
cracklib-2.7-12
db2-2.4.14-7
etc!.....

If we are only interested in the VMware rpms, then we can just pipe the output of rpm -qa command into the grep search tool.

rpm -qa |grep VMware

which should yield an output something like

VMware-mui-2.5.0-11548
VMware-esx-2.5.0-11548
VMware-perftools-2.5.0-11548
VMware-ccagent-esx-2.5.0-11343

If we then want to find out more information on an individual RPM package, we can use the rpm -qi option to query a package which reports the file version, vendor, license and description.

# rpm -qi VMware-ccagent-esx-2.5.0-11343

Name : VMware-ccagent-esx                Relocations: (not relocateable)
Version     : 2.5.0                           Vendor: VMware, Inc.
Release     : 11343                       Build Date: Tue Nov 30 05:52:16 2004
Install date: Tue Apr 4 17:48:07 2006     Build Host: pa-build11.vmware.com
Group       : Applications/Emulators      Source RPM: VMware-ccagent-esx-2.5.0-11343.src.rpm
Size        : 2360792                        License: commercial
Summary     : VMware CCagent package.
Description :

If we then want to know what files are included in the rpm package, we can use query with the list option to see the files inside. For example, to see the files

# rpm -ql VMware-perftools-2.5.0-11548
/usr/bin/esxtop
/usr/bin/rrdtool
/usr/bin/vmkusage
/usr/bin/vmkusagectl
/usr/lib/vmware/vmkusage-cron.sh
/usr/share/doc/vmware/README-perf
/usr/share/doc/vmware/open_source_licenses-perf.txt
/usr/share/man/man1/esxtop.1

# rpm2cpio VMware-perftools-2.5.0-11548.rpm | cpio -idmv

i = Restore archive
d = Create landing directories
m = Create previous file modification times
v = verbose

# ifup eth0

# ifdown eth0

would take Ethernet interface “eth0” down.

If we wish to take the interface down and then up again, we can separate these two commands with a semicolon to run the commands consecutively.

ifconfig eth0

ifconfig eth0 up
ifconfig lo down
 

Media Independent Interface tool. This tool can be used to force the service console network to a particular speed or duplex.

# mii-tool -F

Doesn’t work correctly with some network cards, including Intel 1000 Pro copper NICs.

ifdown eth0; ifup eth0

The semicolon separating the two commands in the above example can be used to separate any two command line entries when you wish the commands to be executed sequentially. In the Windows command line, the same thing can be achieved by the separator "&&".

Or instead of using the word service, some use

/etc/init.d/network restart

hosts: files nisplus dns

In the above example, the name service will use the /etc/hosts file, then NIS+ and then the DNS name server specified in the /etc/resolv.conf file.

If the application is using glibc library for resolver (gethostbybname) but the app could use its own resolver library. An example of this

XXX

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1          localhost.localdomain     localhost
192.168.1.10       esx1host.taupoconsulting.net    esx1host

Notice that each line has a FQDN in column 2 and an alias or shortname in column 3.

hostname -i displays the IP address

and

hostname -s displays the short hostname, i.e. without domain name

/etc/resolv.conf            - search domain.com, nameserver=w.x.y.z
/etc/hosts                  - a.b.c.d    esx1.domain.com
/etc/sysconfig/network      - HOSTNAME=esx1.domain.com

This tool does not appear to let you set the DNS domain name.

search taupoconsulting.net
nameserver 192.168.1.150

This text configuration file contains the service console hostname and default gateway IP address.

NETWORKING=yes
HOSTNAME=esx1
GATEWAY=192.168.1.1

DEVICE=eth0
BOOTPROTO=static
IPADDR=192.168.1.51
NETMASK=255.255.255.0
ONBOOT=yes
  

check_link_down () {
return 1;
}

Note this update only relates to Linux Guest operating systems inside a VM, this is not a setting required for the ifcfg-eth0 file in the service console.

route                               Prints routing table
route del –net default              Deletes the default gateway
route add –net default gw w.x.y.z   Adds a new default gateway
  

┌───────┤ Network configuration ├───────┐
│                                       │
│ Would you like to set up networking?  │
│                                       │
│ ┌─────┐ ┌────┐                        │
│ │ Yes │ │ No │                        │
│ └─────┘ └────┘                        │
│                                       │
│                                       │
└─────────────────────────────────────

This utility will update the following IP configuration files for you

/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/sysconfig/network

VMware ESX Server 2.1.2
Kernel 2.4.9-vmnix2 on an i686 

Linux esx1.taupoconsulting.net 2.4.9-vmnix2 #1 Fri Aug 6 04:38:44 PDT 2004 i686

This command displays the currently active network connections.

netstat --inet -n -p -e

id robin

would reveal something like:

uid=508(robin) gid=510(robin) groups=510(robin),506(techsupport)

This output tells us that the user robin has a UID of 508, a primary group membership of robin and secondary group membership of techsupport.

alias lsf='ls -F'

The above command alias will not however persist to another login session. To have that alias available to you on next login, you would need to add this text to your .bashrc file in your home directory.

To make the alias available to all users on the system, you could add the alias definition to the file /etc/bashrc, which is referenced by the users' /home/<user>/.bashrc file, like an include.

If you just type alias without parameters, you will see a list of the aliases you have defined.

passwd <user>

Remember that passwords are not stored in the /etc/passwd file, but in the file /etc/shadow 

If you are ever needing to reset an unknown root account password, then it is this utility you would run after booting into Linux single user mode.
 

useradd sally

would add a user called sally. We could equally have created a service console user by using "Users and Groups" in the Options tab of the MUI. We can set more than the basic properties of a user account with some additional switches. The following command

useradd robin -G techsupport -s /bin/bash -d /home/robin

would add a user called robin who is a member of the techsupportusers group and has a home directory /home/robin and will receive the Linux bash shell at login.

The service console is a modified version of Red Hat Linux (RHL), and by default in RHL, when a user account is added, a group is created of exactly the same name and has only the user account as a member. This feature is called User Private Groups (UPG) and is discussed in more detail on the RedHat documentation website found here.

So, now that we know about UPGs, looking again at the command above, the command adds a user called robin whose primary group (-g) is called robin and other group (-G) membership is techsupport

We can add additional parameters to the useradd command to more fully specify the account.

useradd alistair –g Finance –s /bin/false

In the above example the users’ primary group is Finance and the shell is specified. In this case the shell is /bin/false which is a bogus shell which would prevent interactive logon by this user. By default in the service console, the shell assigned to users is the BASH shell - specified as /bin/bash (BASH stands for Bourne-Again SHell). It appears the only other Linux shell that is shipped with the service console is csh (the C shell).

groupadd esxadmins

In the above example, a new group called esxadmins is created and therefore a new line appears in /etc/group.

gpasswd –a greg esxusers

Group removal is simple with the –d switch:

gpasswd –d tony esxusers

Be very careful with this command if you intend to use it to modify a users' group membership. When used with –G to set the users group membership, it is not adding the user to a group but is actually setting the list of secondary groups a user belongs to. Therefore in the following example if bill had secondary group list of esxusers and sqladmins, then after entering:

usermod –G techsupport bill

then bill would only have a secondary group of techsupport and nothing else! We would have overwritten the entry in the /etc/group file that listed bill as a member of esxusers and sqladmins. This is why the command gpasswd is so much clearer.

It is good to use the id command to check what groups a user is a member of, before and after the user modification operation to ensure you have got it right.

groupmod -n newgroupname oldgroupname

When it used without parameters, we are specifying to switch to the user root. However, we can use the su command to switch shell to any user account. In the first example, we are logged in as the user kevin and we are switching to user ali.

[kevin@esx1host kevin]$ su ali
Password:
[ali@esx1host kevin]

In this second example, we are switching from being logged on as a user called sara to being logged on as root. Notice to switch to root, we don't need to specify a username.

[sara@esx1host sara]$ su -
Password:
[root@esx1host root]#

If we restrict the built-in user account root from logging in over the SSH protocol, then we are forcing remote users to authenticate as themselves and then su to run privileged commands if need be, thus leaving a decent audit trail. The downside being that those users would still know the root account password.

There is a single line in this file that needs to be uncommented to limit the use of su. The line is shown below as it appears it that file, all that is required is the removal of the # symbol at the start of the line.

#auth required /lib/security/pam_wheel.so use_uid

[ali@esx1 ali]$ sudo vmkfstools

The vmkfstools command would then run under the security context of the root user. The superb feature of this tool is that the user ali does not need to know or supply the root password to be able to run the delegated command. Further, we can keep an audit trail of when sudo was invoked.

But a great benefit of using visudo over vi, is that it performs some basic syntax checking for us!

alistair ALL= /bin/kill, /usr/bin/, /usr/sbin/alistair/

The best source I've found so far on detailed use and background of sudo can be found at http://aplawrence.com/Basics/sudo.html

kirsten:x:505:kirsten
esxusers:x:507:kirsten,flagship
flagship:x:508:flagship
vpxuser:x:511:
adminaccount:x:512:
JohnSmith:x:513:

This may look like a list of users, but it is a list of groups. As the service console (vmnix) is a modified version of Red Hat Linux, the Linux security configuration is the same as Red Hat. One feature of Red Hat not found in all Linux distributions is that of the user private group (UPG). Whenever you create a user, a group of the same name is created also and the user is made a member. The format of the file is:

groupname:x:user1,user2

so when we see groups like JohnSmith:x:513 we can assume the 513 is the UID for the user JohnSmith and this is his UPG.

Here is a sample section of a passwd file:

ali:x:500:500:Alistair Sutherland:/home/ali:/bin/bash
sara:x:501:501:Sara Daniels:/home/sara:/bin/bash
janice:x:502:502::/home/janice:/bin/bash
andy:x:503:503::/home/andy:/bin/bash

As shown, the format of the file is

username:x:userID:groupID:fullname:homedirectory:shell

Normally the group ID will match the user ID.

There is a command line tool to edit this file, vipw

ali:$1$tkSdSEQD$x8pXvtDZ3Xta6zza9lKqh.:12733:0:99999:7:::
sara:$1$c4jofyxg$8zjaMTXWhW2hniTXKUt7V/:12733:0:99999:7:::

If a user account has been disabled with the usermod command, a "!" will be placed in front of the encrypted password in this file.

NIS is a network lookup service which consists of databases and processes. It works where a NIS master server stores the source files for the maps such as

/etc/passwd
/etc/group
/etc/hosts

A NIS master serves a NIS domain. You can have multiple NIS servers for a domain, but only 1 is the master, other NIS servers host read-only copies, i.e. they are slaves. NIS databases are in DBM format.

The NIS master server daemon is ypserv.

NIS client machines are those which get their configuration from the NIS Master. A NIS client runs the process ypbind.

Of note are the vmkernel, vmkwarning & messages file logs. These logs can be viewed with the more, cat, head and tail command line tools. We can also access these logs via the MUI via the following link in the Options tab.

If you use the sudo tool to run a command under a different security context then the log file /var/log/secure will contain the audit trail for such activity. Check the file /etc/syslog.conf for logging settings.

You can use less /var/log/logfile and then use SHIFT-f to enable dynamic update as new data is delivered to that file.

It is sometimes useful to add a line to the end of this file

local6.*      /dev/tty3

to get real-time logging of VMkernel to tty3.

lsof |grep IPv4.\*LISTEN

             total     used     free    shared     buffers    cached
Mem:           265      259        5         0          39       135
-/+ buffers/cache:       85      180
Swap:          541        0      541

Given these results, I would be thinking about either running fewer VMs, disconnecting unused devices from VMs, stopping any unnecessary applications or increasing service console RAM.

[root@esx1 root]# fdisk -l

Disk /dev/sda: 255 heads, 63 sectors, 17816 cylinders
Units = cylinders of 16065 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sda1   *         1         6     48163+  83  Linux
/dev/sda2             7       235   1839442+  83  Linux
/dev/sda3           236       304    554242+  82  Linux swap
/dev/sda4           305     17816 140665140    f  Win95 Ext'd (LBA)
/dev/sda5           305      1834  12289693+  83  Linux
/dev/sda6          1835      2063   1839411   83  Linux
/dev/sda7          2064      2076    104391   fc  Unknown
/dev/sda8          2077     17816 126431518+  fb  Unknown

Looking at the above output of the fdisk command, the last two partitions are for the VMkernel. Partitions of type "fc" correspond to the VMKcore dump partition. Partitions of type "fb" are VMFS volumes.

If you wanted to create a new VMFS volume from the service console command, then you could use fdisk to create the custom partition type.

fdisk /dev/sdf
 

In the following example, we have added a 2nd disk to the service console (appearing as SCSI disk "b" i.e. /dev/sdb). By using fdisk we have created a primary partition. Now, to create the file system we use makefs

makefs -t ext3 /dev/sdb1

e2label

du –h /home/ali/vmware
du –h ~
du –s summary
 

# df -h
Filesystem     Size  Used    Avail  Use%  Mounted on
/dev/sda2      2.0G  640M     1.2G   34%  /
/dev/sda1       45M   12M      31M   27%  /boot
/dev/sda7      2.0G   33M     1.8G    2%  /home
none            93M    0       93M    0%  /dev/shm
/dev/sda8      2.0G   33M     1.8G    2%  /tmp
/dev/sda6      2.0G  226M     1.6G   12%  /var
/dev/sda5      9.8G  2.9G     6.5G   31%  /vmimages
//win2k/share  137G   75G      61G   55%  /root/class

This is a great tool to run when first diagnosing an ESX server. The results of this command tell us whether the server was partitioned correctly and if any partitions are constrained for disk space.

# vdf -h
Filesystem     Size  Used    Avail  Use%  Mounted on
/dev/sda2      2.0G  640M     1.2G   34%  /
/dev/sda1       45M   12M      31M   27%  /boot
/dev/sda7      2.0G   33M     1.8G    2%  /home
none            93M    0       93M    0%  /dev/shm
/dev/sda8      2.0G   33M     1.8G    2%  /tmp
/dev/sda6      2.0G  226M     1.6G   12%  /var
/dev/sda5      9.8G  2.9G     6.5G   31%  /vmimages
//win2k/share  137G   75G      61G   55%  /root/class
vmhba0:0:0:10   48G   15G      33G   31%  /vmfs/vmhba0:0:0:10
vmhba1:0:10:1 10.0G  7.0M    10.0G    0%  /vmfs/vmhba1:0:10:1
vmhba1:0:11:1 10.0G  191M     9.8G    1%  /vmfs/vmhba1:0:11:1
vmhba1:0:25:1  136G   21G     114G   15%  /vmfs/vmhba1:0:25:1
vmhba1:0:26:1  136G  8.1G     128G    5%  /vmfs/vmhba1:0:26:1
vmhba1:0:27:1   14G  3.9G      11G   26%  /vmfs/vmhba1:0:27:1
vmhba1:0:28:1   14G  7.0M      14G    0%  /vmfs/vmhba1:0:28:1

When troubleshooting, make this your first command to run. You will be able to review if each partition for the service console and the VMkernel has enough disk space. Just take a quick look down the "Avail" column and if you see a zero there's likely a problem right there, or just look at the USE% column.

dd if=/dev/cdrom of=/vmimages/new.iso bs=32k

This tool could also be used to go from ASCII to EBCDIC etc.

This tool can be used to create an additional swap file. For example, if we did not allocate a big enough swap partition for the service console during ESX installation, we can create one now in a file of 64MB.

dd if=/dev/zero of=/swapfile bs=1M count=64

If we did add a swap file, we would need to make sure it is started when ESX starts. Therefore, an entry in the file system table /etc/fstab would be needed as this file describes the local and remote file systems to mount at boot. The total amount of service console swap space is the sum of the swap partition and any swap files that are active.

Filename                Type       Size   Used  Priority
/dev/sda3               partition  554232 0     -1
/swapfile               file       65528  0     -2
 

# touch newfile

However, this can be used to touch an existing file and update its last modified or last accessed attributes. This could be scripted if required. Be careful and avoid running touch against any file stored on a VMFS volume, as there appears to be a problem there. Remember that not all Linux tools are modified for VMFS awareness.

The VMFS is not an ext3 partition. but the directory /vmfs in the service console provides mount points to the VMkernel-mounted VMFS volumes.

This command is used frequently to view the contents of a text file, exactly as the command type in DOS or Windows command line. So to view a view we could enter

# cat /etc/vmware/netmap.conf

Technically, this is the tool to concatenate files together.

We can also use this tool to create text files quickly at the command line, by entering the text and then using the key sequence CTRL-D to write to file. In the following example, we create a new bare-minimum vmx file at the command line.

# cat > newVM.vmx

guestOS = "winxppro"
config.version = "6"
virtualHW.version = "3"
CTRL-D

Writes the text following echo command to file. This could be good for quickly creating files

echo modprobe usb-uhci > S92usb
echo modprobe usb-ohci >>S92usb

Another great use of this technique is to make changes to the ESX server configuration via the /proc hierarchy, e.g. changing the number of shares for a VM

echo 2500 > /proc/vmware/vm/nnn/cpu/shares

would change the VM CPU shares to 2500. However such a change would only exist for the duration of the world created for that VM. After the VM is powered off this in memory structure is lost. To make such a change persistent, we would need to add the line

sched.cpu.shares = "2476"

to the VMX file of the virtual machine. 

head server.dsk | file -

The “–“ is crucial to making the above command work. For an ESX virtual disk we would expect to see something like standard input: x86 boot sector.

If you are using this to view the last few entries in a log file, you can use the -f switch to "follow" changes as they happen to the file.
  

sort /etc/vmware/vm-list

or to sort a basic score sheet

sort –g –k 2 scores.txt
  

Grep can be used as a command directly e.g.

grep alistair /etc/passwd

or the output of a command can be piped directly into grep, for example the output of all running processes in the service console could be searched for the string "vmware"

ps -eaf |grep vmware 

# cat /proc/vmware/vm/*/names | cut -f1-5,25- -d" "

–mount         used to ensure it doesn't traverse to remote file systems
-size            obvious
-mtime -n     modified in the last n*24
-mmin -n      modified in the last n minutes
-ls               use output format as if ls were used
-name          name the file you are looking for (you just don’t know where it is!)

find –mmin -30                files modified in last 30 minutes
find –mtime -1                files modified in last 24 hours
find –size +10000             files in excess of 10,000 bytes
find –mount –size +10000 -ls  files on non-remote file system
find –name “hosts” -ls        file called hosts
find -exec ls -al {} \;       do ls on the files found
find -perm 666                find files with exactly rw-rw-rw-
find -perm +666               find files with at least rw-rw-rw
find -user ali                find files owned by ali

vi filename

The first thing that throws you is that to enter text into your file, you need to press "i" for Insert mode. You can then enter your text just as any other text editor. When you are done with text entering, just press the Escape (Esc) key to come out of insert mode. If you are happy with your file, then we need to Write & Quit (wq). To enter commands in this command line editor, rather than having menus, we have a command prompt in the application. To reach the vi command prompt, simply enter ":" - the colon character which will automatically place your cursor at the bottom of the session. Here you can enter the "wq" command to write and quit the editor. That's it!

Here is a summary of the vi commands

i                  Changes to insert mode where you can edit the text
:wq               Write the file and quit the editor
:q!               Quit the editor without saving changes

SHIFT ZZ       Quit the editor and save any changes made - just a fast way of doing ":wq"
Esc key          Exits the current mode, e.g. out of insert mode back to view mode.

These commands are just extra if you have the inclination to learn!

/                     search - if you entered /failed then the cursor would move to the first instance of "failed in the text
$                     jumps to the end of the opened file
yy                   copy - it's y for yank!
dd                   delete a line (cut) if you precede this with a number e.g. 8dd, then it would delete 8 lines
p                     paste
%s/old/new/g    substitute any occurrences of the world "old" with the world "new"

There are some great web sites which document the features of vi in superb depth, one of them is the staff site at University of Washington which helped me. Their site is at http://staff.washington.edu/rells/R110/

wc filename

authconfig
sysntv
mouseconfig
netconfig

 

List files in a directory including hidden (also known as dot files due to their prefix) files.

ls -dl */

List directories in long format (does not display files). Could add as a shell alias, say lsd.

If you want to organise files by their modification date.

ls -ltr

If you are interested in knowing where on the disk files are stored, based on their inode, use the -i switch.

ls -lia

more /etc/ssh/sshd_config

Or, to pipe the output of a command into the more utility

ls -al |more

#chown ali solaris.vmx

However if you wish to reset both the user owner and group owner, then rather than having to use chown and then chgrp straight after it, you can set user and group ownership in one operation by specifying the user owner and group owner separated by a colon as in the example shown.

#chown ali:ali netware5.vmx

#ll
-rwxr-wr-- bill bill w2k.vmx

Now we are going to change the group owner of the file to the group called vmadmins.

#chgrp vmadmins w2k.vmx
#ll
-rwxr-xr-- bill vmadmins w2k.vmx

So, in a full file listing, when you see 2 names, e.g. bill vmadmins, the first name is the user owner and the second name is the group owner. In Red Hat Linux, we have something called user private groups, which means that for each user account, there is a group account of the same name. So if you see a file owner and group owner as the same name, these are not the same security principals, one is the user account, the other is a group of the same name.

When we look at a file listing using ls -al the file & directory permissions are shown on the left.

-rwxr-xr-- 1 ali vmadmins 345 May 7 14:22 file.txt

In the above example, the file has 3 permissions described in the -rwxr-xr-- string. These are:

rwx for the User owner  - in our example above, this is the Linux user 'ali'
r-x for the Group owner - in our example above, this is the Linux group 'vmadmins'
r for all Others             - permission for any other user who is neither the user or group owner.

In this first chmod example, we are going to change the permissions on the file.txt by removing the read & execute permission for the user owner of the file and we are also going to remove the read permission for the group owner of the file.

# chmod u-wx,g-x file.txt
# ls -al
-r--r--r-- 1 ali vmadmins 345 May 7 14:22 file.txt

Note that using + or – indicates we are adding to or removing from the existing permissions. If we wish to reset the permission we use “=” to explicitly set the object permissions, overwriting anything that was already set.

# chmod u=rx,g=r,o=r file.txt yields r-xr—-r--

Sometimes you will see a chmod command using 'a' to specify all (user, group & other), so we could quickly set read permissions by

# chmod a-wx,a+r file.txt yields r—-r-—r--

A more common way to set permission is using chmod is using numeric equivalent values (4,2,1 for r,w,x) and permutations thereof.

chmod 777 windows2k.vmx would set permission to rwxrwxrwx
chmod 754 windows2k.vmx would set permission to rwxr-xr-- (default)

Watch for chmod commands with 4 digits, e.g. chmod 0754. This refers to additional attributes as described below.

Sticky bit

When the sticky bit (t) is set on executable files, it tells Linux to keep the application in memory. The reason for this is to improve load times for other users who wish to run the same executable. This relates to the multi-user nature of UNIX/Linux. Given the speed of memory and disk access nowadays the need to keep applications in memory is much less important and so the sticky bit isn't needed so much.

When the sticky bit is turned on for a directory, users can have read and/or write permissions for that directory, but they can only remove or rename files that they own.

If you see a "t" in a file or directory permission, this indicates the sticky bit is set. You can turn on the sticky bit with the chmod tool and specify "t".

chmod +t /directory

You can then view the directory with ls -al and note that the executable permissions indicator bit is shown as a "t" showing that the directory has the sticky bit set.

drwxr-xr-t 2 root root 4096 May 7 12:02 directory
 

SETUID (set uid)

The Set User ID bit is used on an executable file, so that when it is run, it is run under the security context of the file owner and not the current user who launched that executable. So, if I have an executable file whose owner is 'root' and it has the setuid bit set, then when I run this application as a normal user, that application would still run under 'root' privilege.

To set the UID bit, we use chmod with the "s" indicator. In the following example, the Perl script called listswitch.pl is has a user owner 'ali' and a group owner 'vmadmins'. Once the user id bit is set on this file, whoever launches the executable will not in fact be the owner of the process, the user 'ali' will be the process owner.

# chmod u+s listswitch.pl
# ls -al
-rwsr--r-- 1 ali vmadmins 396 May 7 12:09 listswitch.pl

You may have already been using a program with setuid set and not even known about it! The sudo command is owned by root and has the setuid bit set. You can check if the setuid bit is set by inspecting the file permissions

---s--x--x      1   root    root   80764  Jul  23   2001 /usr/bin/sudo

Set Group ID.

Just like SUID, setting the SGID bit for a file sets your group ID to the file's group while the file is executing. So again, we use the chmod tool with 's' but this time we set it on the group permission.

# chmod g+s listswitch.pl
# ls -al
-rwxr-sr-- 1 ali vmadmins 396 May 7 12:14 listswitch.pl

The group id bit is a great feature to enable easier management of permissions on the files in that directory. When the group id bit is set on a directory, any files or subdirectories created in that directory will automatically have their group ownership set to the same as the parent directory!

As we have seen above, to set any of these 3 attributes, we can use the 't' and 's' indicators. However, often we set permissions with chmod using numerical values like 777 to represent rwx. When setting user id, group id or sticky bits using chmod and numerical values, we use a 4th digit preceding the usual 3 used with chmod. That digit is set using the following:

4 Set user ID (s)
2 Set group ID (s)
1 Set sticky bit (t)

So if we want to set a file with permission -rwxr-xr-x and set the user ID bit we could use the following:

# chmod 4755

which would result in a new file permission of -rwsr-xr-x. Notice the "x" of the user permission is now an "s" indicating the setuid bit is set.

The most frequently used umask is 022, this would take away the write permission for the group owner and others in a permission list, i.e. full permission equals 777, corresponding to read(4), write(2) and execute (1).

Full permissions       777   rwxrwxrwx
Minus the umask        022   ----w--w-
Effective permission   755   rwxr-xr-x

1 NIC assigned to Service Console
1 NIC assigned to VMkernel
1 SCSI adapter assigned to be shared between Service Console and VMkernel
1 Fibre Channel adapter assigned to VMkernel

The vmkpcidivy tool is stored in the directory /usr/sbin/vmkpcidivy. This tool asks a series of questions and should be used with the –i switch for interactive mode. To assign a PCI card to either operating system, we use the 3 characters c, v & s.

[c] Assign to Service Console
[v] Assign to VMkernel
[s] Assign as shared between Service Console and VMkernel (the boot disk controller)

To run, we just type vmkpcidivy -i If you add a new NIC, SCSI or fibre channel PCI card to your physical server, you should boot the server into Linux and run the vmkpcidivy command. This way you can correctly assign the PCI card to the right operating system and also allows you to check that the new PCI card has not changed your existing PCI assignment. Once you have saved your changes, restart the server and boot ESX Server normally. This command is also used to refreshnames and –q vmhba_devs For example, if I had a SAN LUN of vmhba1:0:25 and lets say I removed the VMFS from this LUN and now wished to use it from the service console, I’d run

# vmkpcidivy -refreshnames

and then would run vmkpcidivy again this time with the query switch (-q)

# vmkpcidivy -q vmhba_devs

to find out what device name the service console was going to use for this LUN, e.g. vmhba0:0:0 /dev/sda

A very useful feature of this tool is the ability to create a new profile. This adds a new boot option to the LILO boot menu that will have its own allocation of memory and PCI devices. If you are unsure about the changes you are making, then create a new profile e.g. esx (modified)

vmkchdev -L

lists the PCI devices and reports whether they are assigned to VMkernel or the service console. We can also get this information from running vmkpcidivy, but if we only want a quick report of which device is owned by which OS, then this is great. Notice also that the PCI device ID is reported which is very helpful where we have more than one device of the same name, e.g. you could have 2 dual port Intel ethernet cards.

-i                   to import a virtual disk to VMFS
-e                   to export a virtual disk from VMFS
-m                   to commit changes from REDO log
-s                   to re-scan for new LUNs
-S                   to set vmfs metadata volume label
-X 6000M ./file.dsk  to extend an existing DSK to 6GB
-c 4000M ./file.dsk  to create a new empty virtual disk
-C                   to create a new vmfs volume
-l vmfsname          to list virtual disks on specified vmfs
-F                   to set the access mode e.g. public/shared
-k                   to create a VMkernel swap file
-w                   to activate a VMkernel swap file
-y                   to deactivate a VMkernel swap file
-T                   to convert a vmfs1 volume to vmfs2

Remember that the vmfs parameter always goes last on this command parameter set for vmkfstools. This can be confusing for the beginner as the source and target order is different for imports and exports.

If we want to simply list the files on a vmfs volumes we use the -l switch.

vmkfstools –l /vmfs/vmhba0:0:0:8

or if we wish to use the more friendly VMFS volume label;

vmkfstools –l <vmfs-metadatalabel>

which would produce an output similar to the following

Name: VMFS2-VOL1 (public)
Capacity 129465874944 (123461 file blocks * 1048576) avail
Permission Uid Gid Attr Bytes (Blocks)    Last Modified Filename
rw-------   0   0 swap 2146435072 ( 2047) Nov 18 18:25  Swap.vswp
rw------- 500 500 disk 4194304000 ( 4000) Nov 16 14:12  VM1.vmdk
rw------- 500 500 disk 6291456000 ( 6000) Nov 23 22:19  VM2.vmdk
rw------- 500 500 disk 2621440000 ( 2500) Nov 17 23:09  VM3.vmdk
rw------- 500 500 disk 4194304000 ( 4000) Nov 24 18:11  VM4.vmdk

If we use the command with the lh switch we get the results in human readable format. Notice that file sizes are shown rounded with the "G" symbol.

[root@esx4 W2Ktest]# vmkfstools -lh vmhba0:0:0:10

Name: Local (public) Capacity 48G, 33G avail, file block size 1.0M
Permission Uid Gid Attr Bytes Last Modified Filename
rw------- 0 0 swap 1.2G Apr 26 12:30 SwapFile.vswp
rw------- 0 0 disk 2.0G Apr 26 14:07 ad1-win2000server.vmdk
rw------- 0 0 disk 2.0G Apr 27 15:21 ad2-win2000adv.vmdk
rw------- 0 0 disk 2.0G Apr 27 08:41 Clone of ad2-win2000adv.vmdk

To create a new VMFS volume, we use the -C switch. In the following example, we are creating a VMFS volume on LUN16 on host bus adapter 1, typically the fibre channel adapter.

vmkfstools -C vmfs2 vmhba1:0:16:1

If someone has created a VMFS volume with an illegal character in the volume label, you may have problems removing that volume in the MUI. If this is the case, just overwrite the VMFS volume by creating a new volume over the top of the badly named one using the -C switch.

To create a new empty virtual disk on a VMFS volume we use the -c switch

vmkfstools –c 2048M /vmfs/vmhba0:0:0:8:newdisk.vmdk

This command would create a new virtual disk (monolithic) on the specified VMFS volume. Remember it is always better to use the VMFS name as this will not change even if your hba hardware does.

To import a virtual disk into the VMFS we use vmkfstools with the -i switch. This will take a virtual disk in sparse (COW) format into monolithic format without causing excessive SCSI reservations on the LUN holding the target VMFS.

vmkfstools –i /vmimages/template.vmdk /vmfs/vmhba1:0:25:1/new-vm.vmdk

As always with this command, the parameter specifying the VMFS location is always the last parameter.

If you just wish to view the properties of a VMFS volume, you can use the -P switch to print the volume properties. You can use either the logical name for the vmhba partition or the VMFS volume label.

[root@esx1 cpu]# vmkfstools -P VMFS2-VOL1

VMFS2-VOL1 is a VMFS-2.11 volume spanning 1 physical extents.
Volume label (if any): VMFS2-VOL1
UUID (if any): 6890b365-d911e933-7286-8497e91f9b7d
Physical Extents:
vmhba0:0:0:8 

There is no man page for this tool and --help doesn't yield anything beyond simply entering the command without parameters. Some additional information is visible if you enter

vmware-cmd -h

The first thing we can look at is to registering and un-registering a VM. We use the "-s" switch to indicate we performing a server operation, as opposed to VM operation.

vmware-cmd –s register /home/user/vmware/newvm/newvm.vmx

vmware-cmd –s unregister /home/user/vmware/oldvm/oldvm.vmx

The next use of this command is to list the VMs on the server. However, this will only list the registered VMs, i.e. the VMs which are listed in the file /etc/vmware/vm-list

# vmware-cmd –l

/home/vmware/vm1/vm1.vmx
/home/vmware/vm2/vm2.vmx
/home/alistair/vmware/alisrv1/alisrv1.vmx
/home/andy/vmware/andysolaris/andysolaris.vmx

Next we are looking at connecting or disconnecting a device. Typically this will be for the connection of IDE CD-ROM ISO files or floppy image files.

vmware-cmd /home/user/vmware/vm/vm.vmx connectdevice|disconnect

To perform power operations we unsurprisingly use the start and stop parameters. A stop operation type can be soft, try soft or hard. A stop hard is the last resort and equivalent to a forced VM power off. Here is an example of starting and then soft stopping a VM.

# vmware-cmd /home/user/vmware/server/server/vmx start
start() = 1

# vmware-cmd /home/user/vmware/server/server.vmx stop soft
stop(soft) = 1

If we wish to query the current heartbeat value for a VM, the getheartbeat parameter does the trick. Remember though, that in order to draw any meaning from this, we should query the heartbeat twice to prove the value is in fact increasing! For example,

# vmware-cmd /home/user/vmware/server/server.vmx getheartbeat
getheartbeat() = 29076

# vmware-cmd /home/user/vmware/server/server.vmx getheartbeat
getheartbeat() = 29079

If we want to determine simply if the VM is powered on or not, then we can use the getstate

# vmware-cmd /home/user/vmware/server/server.vmx getstate
getstate() = on

To find out the VMID (also known as the world ID) of a VM, we can use the getid parameter. The VMID is analogous to process ID (PID) but is the unique ID that the VMkernel is using for the Virtual Machine Monitor. The VMID of a VM is normally a 3 digit number greater than 100.

# vmware-cmd /home/user/vmware/server/server.vmx getid
getid() = 145

For every VM that is running with a VMID in the VMkernel, there are a parallel set of management processes running in the service console. These processes are there to allow operators interact with the VM, for example, power on and off, gain remote console access and to maintain the per-VM logging in the file vmware.log. To find the parent process ID (PID) of the management processes that correspond to a VM, we can use the getpid parameter.

# vmware-cmd /home/user/vmware/server/server.vmx getpid
getpid() = 12163

Both the VMID and PID remain unchanged while the VM is running. Once the VM is powered off, those IDs are removed and the VM will more than likely get a new VMID and PID the next time it is powered on.

We can also use this tool to answer questions such as the commit of a REDO file to virtual disk:

vmware-cmd "/home/vmware/SPPS 2003/SPPS 2003.vmx" answer

Question (id = 694724352) :No REDO log to be committed

0) OK
Select choice. Press enter for default <0> : 0
selected 0 : OK 

vmkdump -q              Query the VM kernel for which partition it will use
vmkdump -p vmhba0:0:0:3 Set VM kernel dump partition to partition 3
vmkdump –p none:0:0:0     Set VM kernel dump partition to none

Remember the vmkcore partition does not have a mount point in the service console and is not specified as ext3. We can use the fdisk -l command to view where the core dump partition is in relation to the disk layout.
 

# vmkload_mod -l

Name      R/O Addr Length R/W Addr  Length ID Loaded
vmklinux  0x4de000 0xf000 0x12516b0 0x53000 1 Yes
nfshaper  0x4ed000 0x1000 0x12a81b0 0x1000  2 Yes
e1000     0x4ee000 0xf000 0x12a91b8 0x6000  3 Yes
megaraid2 0x4fd000 0x6000 0x12f6008 0x3000  4 Yes
bond      0x503000 0x2000 0x138a158 0x2000  5 Yes

Note, the -l parameter can also be specified as --list
 

A useful function of this tool is to list running VMs using the -x switch.

VMware ESX Server Support Script 0.93

Available worlds to debug:

vmid=141 Windows 2000 Adv Server MOM 2000 SP1
vmid=142 ISA Server 2004 vmid=143 SUS 2
vmid=144 TAUPOMAIL2 vmid=149 SPPS 2003

[root@esx1 root]#

Watch out for the creation of empty subdirectories of the name "vm-support.<pid-of-process>" in the directory where you run this tool with the -x switch. It is safe to delete these directories.

vmware --new-sn "5c395-02a60-056aa-b8609"

vmware --new-smp-sn "9856a-091c7-6a7a4-8a679"

This command can also be used to display the ESX server version and patch level vmware -v would return something like:

VMware ESX Server 2.5.0 build-11548

alias parport_lowlevel parport_pc
alias eth0 e1000
# alias eth1 e1000
# alias eth2 e1000
# alias eth3 e1000
alias scsi_hostadapter megaraid2
alias usb-controller usb-uhci
alias usb-controller1 usb-ohci

options e1000 Speed=1000

# options bcm5700 line_speed=100 auto_speed=0 full_duplex=1
# options e100 e100_speed_duplex=2 # options e1000 Speed=[0,10,100,1000] Duplex=[0..2]
# options acenic link=[0x213,0x212,0x223,0x222,0x243,0x242,0x271,0x270]
# options 3c990 force=[0..4]

Notice the options available for network cards in this file. If we cannot use the mii-tool to force NIC speed and duplex, then we can remove the comment character ("#") from the appropriate options line in the this file. If there are problems with the interface eth0 disappearing after a rescan SAN operation, ensure that the Ethernet alias definitions above eth0 (i.e. eth1, eth2 etc.) in modules.conf are commented out.

modinfo e1000

Would produce a list of flow control settings for the Intel gigabit NIC.

modinfo cciss

Would produce the file details and version of the HP Smart Array controller.

You are only likely to encounter this command if you decide to share your service console physical network card with the VMkernel, when it is used to load the vmxnet_console device driver module.

The VMkernel device driver modules themselves are stored in /usr/lib/vmware/vmkmod

megaraid.o
nfshaper.o
tcpip
qla2200_604.o

lspci –v   verbose output
lspci –t   print tree of PCI devices (use –tv for verbose tree)

You may wish to examine /proc/pci also in order to correctly identify PCI devices and their slot configurations. One point to note is that when you are faced with PCI slot numbers is that not all hardware vendors number their slots in a straight forward left to right configuration. Make sure you know your slot numbers and their layout!

Gives out way more info that is actually required. Remember that USB devices cannot be presented to virtual machines in ESX Server. If you wish to use a USB device in ESX, then you will have to use a USB over IP device and install the appropriate driver software into your guest OS for this. The most common USB over IP device is AnywhereUSB from Digi. Details can be found at www.digi.com/products/usb. A company called Keyspan also produce a similar device, details at www.keyspan.com
 

network0.name = "SecuredGigabit"
network0.device = "vmnic2"
network1.name = "VirtualSwitch1"
network1.device = "vmnet_1"
network2.name = "InternetSwitch"
network2.device = "vmnic1"

Remember that

If a virtual switch has no physical adapters, then it is vmnet.
If a virtual switch has only 1 physical adapter it is a vmnic
If a virtual switch has two or more physical adapters, it is a bond.

A bond can be in one of three modes, out-mac (default), out-ip and standby

out-mac A VM virtual NIC is assigned to a pNIC in the bond and it uses only that
out-ip A VM TCP conversation is placed on an available pNIC
standby A VM will only use one NIC until a failure, then the other is used. There is no point in having more that 2 NICs in a bond in this mode.

002:14.0 megaraid vmhba0
005:04.1 nic vmnic0
011:07.0 nic vmnic1
012:08.0 nic vmnic2

Any changes to the vmware-devices.map.local file require a reboot, or at least an unload/reload of the VMkernel to take effect.

Entries in the vmware-devices.map.local files are used in addition to the entries in the vmware-devices.map file. The vmware-devices.map.local file does not need to mirror the vmware-devices.map file.

Any vmware-devices.map.local file entries that correspond to the vmware-devices.map file entries supercede the vmware-devices.map file entries.

libdir = "/usr/lib/vmware"
dhcpd.fullpath = "/usr/bin/vmnet-dhcpd"
authd.fullpath = "/usr/sbin/vmware-authd"
authd.client.port = "902"
loop.fullpath = "/usr/bin/vmware-loop"
vmware.fullpath = "/usr/bin/vmware"
control.fullpath = "/usr/bin/vmware-cmd"
serverd.fullpath = "/usr/sbin/vmware-serverd"
wizard.fullpath = "/usr/bin/vmware-wizard"
serverd.init.fullpath = "/usr/lib/vmware/serverd/init.pl"

serverd.vpxuser = "vpxuser"
serverd.snmpdconf.fullpath = "/etc/snmp/snmpd.conf"
snmp.enable = "TRUE"

prefvmx.useRecommendedLockedMemSize = "TRUE"
autoStart.defaultStartDelay = "240"

If this file is missing or corrupted then you will get some very weird behaviour, for example the inability to power on any virtual machine and no vmware.log file being created. If the root file system should become full, certain files can be corrupted, make sure this isn't one of them. I've seen this file truncated a number of times. To check if this may be your problem, try

vmware-cmd /home/vmware/vm/vm.vmx start

And if the result is

Malformed protocol error

then it could be that the /etc/vmware/config file is truncated, corrupt or simply missing. Normally, simply copying this file from another server will normally restore the server to normal operations.

grep –i vmnic /etc/vmware/hwconfig

So the first SCSI disk would be sda, the second would be sdb and so on.

Bus:Sl.F Vend:Dvid Subv:Subd Type Vendor ISA/irq/Vec P M Module Name Spawned bus
000:00.0 8086:3590 1028:016e Host/PCI Intel C
000:02.0 8086:3595 0000:0000 PCI/PCI Intel 001 C
000:03.0 8086:3596 0000:0000 PCI/PCI Intel 004 C
000:04.0 8086:3597 0000:0000 PCI/PCI Intel 007 C
000:05.0 8086:3598 0000:0000 PCI/PCI Intel 010 C
000:06.0 8086:3599 0000:0000 PCI/PCI Intel 013 C
000:29.0 8086:24d2 1028:016e USB Intel 11/ 16/0x69 A C
000:29.1 8086:24d4 1028:016e USB Intel 10/ 19/0x71 B C
000:29.2 8086:24d7 1028:016e USB Intel 7/ 18/0x79 C C
000:29.7 8086:24dd 1028:016e USB Intel 3/ 23/0x81 D C
000:30.0 8086:244e 0000:0000 PCI/PCI Intel 016 C
000:31.0 8086:24d0 0000:0000 PCI/ISA Intel C
000:31.1 8086:24db 1028:016e IDE Intel 0/ / A C
001:00.0 8086:0330 0000:0000 PCI/PCI Intel 002 C
001:00.2 8086:0332 0000:0000 PCI/PCI Intel 003 C
002:14.0 1028:0013 1028:016e RAID Dell 7/ 38/0x91 A S vmhba0
004:00.0 8086:0329 0000:0000 PCI/PCI Intel 005 C
004:00.2 8086:032a 0000:0000 PCI/PCI Intel 006 C
005:04.0 8086:1010 8086:1012 Ethernet Intel 7/ 58/0x99 A C
005:04.1 8086:1010 8086:1012 Ethernet Intel 10/ 59/0xa1 B V e1000 vmnic0
010:00.0 8086:0329 0000:0000 PCI/PCI Intel 011 C
010:00.2 8086:032a 0000:0000 PCI/PCI Intel 012 C
011:07.0 8086:1076 1028:016d Ethernet Intel 11/ 72/0xa9 A V e1000 vmnic1
012:08.0 8086:1076 1028:016d Ethernet Intel 11/ 73/0xb1 A V e1000 vmnic2
016:13.0 1002:5159 1028:016e Display ATI 7/ 18/0x79 A C

findnic –i 5 vmnic2 10.0.0.1 192.168.1.3

The above command will send ICMP echo requests to 192.168.1.3 every 5 seconds. We could also use the –f switch which would flood ping.

If you determined that one of these VMs temporarily needed more CPU shares, you could increase CPU shares on the fly at the command line, just by using echo to input a value into the shares file for that VM.

echo 10000 > /proc/vmware/vm/<number>/cpu/shares

# cat /proc/vmware/sched/ncpus
 4 logical
 2 physical

You can also get the same information from the top three lines of esxtop.

watch cat /proc/vmware/mem

to obtain a dynamic view of memory usage by the VMkernel. If you are viewing lots of output but can't see what is actually changing between refreshes, we can use the -d parameter to specify display differences, thus highlighting changes between refreshes.

watch –d cat /proc/vmware/mem

To specify a different refresh rate, we use the --interval=n parameter

<pic>

When we use the tool esxtop we get presented with two columns that look identical, VCPUID and WID. The VCPUID is the ID number of the virtual processor of that VM. This number will be the same as the world ID of the VMM, indicated in the WID column. Where things get interesting is when we have a VM with 2 virtual CPUs, i.e. we are using virtual SMP (symmetric multiprocessing). In this case, a VM gets two VCPUIDs, but is still only 1 world. So the output of esxtop when you have a vSMP VM would be similar to

VCPUID WID  WTYPE   %USED %READY %EUSED  %MEM
 129   129  idle    51.79   0.00  51.79  0.00
 131   131  idle    47.29   0.00  47.29  0.00
 130   130  idle    37.83   0.00  37.83  0.00
 128   128  idle    34.73   0.00  34.73  0.00
 145   145  vmm      8.37   3.85   8.37  7.00
 127   127  console  7.50   3.53   7.50  0.00
 160   160  vmm      5.32   1.61   5.32  4.00
 162   162  vmm      2.06   0.50   2.06  1.00
 164   164  vmm      2.04   0.14   2.04  0.00    <--- note world ID of 164
 165   164  vmm      0.32   0.26   0.32  0.00    <--- note world ID of 164

In the above example, it can be seen that there are two VCPUIDS (164 & 165) that correspond to the same world ID (164).

To avoid this and view the data in a more sensible fashion, pipe the output of the command into the less command with the -S switch as shown:

cat /proc/vmware/vm/133/disk/vmhba0:0:25 | less -S

It might also be a good idea to use the watch command on this file, as the disk queue length will be constantly changing and when you cat the file, you may only be sampling the queue while its on zero!

These directories are labelled using names vmnic, vmnet and bond. If you wish to reconcile a vmnic number to the virtual Ethernet switch name exposed in the MUI, then inspect /etc/vmware/netmap.conf.

In the subdirectories of each virtual switch (e.g. /proc/vmware/net/vmnic0 ) you will find files that correspond to per-virtual MAC address of each VM attached to that VM.

smbclient –I 192.168.115 –U username –L computername

Sharename Type Comment
--------- ---- -------
E$        Disk Default share
Microsoft Disk
IPC$      IPC Remote IPC
D$        Disk Default share
NETLOGON  Disk Logon server share
REMINST   Disk Remote Installation Share
ADMIN$    Disk Remote Admin
SYSVOL    Disk Logon server share
VPLOGON   Disk Symantec AntiVirus
C$        Disk Default share
VPHOME    Disk Symantec AntiVirus

To create a mount point to a Microsoft share is very straightforward. Remember, we are allowing the service console to access a remote file system. This is not related to what virtual machines are doing. Further, we need to be careful if we are attempting to do any file operations due to potential limits with 2GB file sizes.

1. Add entry to local hosts (/etc/hosts) for MS host
2. Create a local directory where you wish the mount point with mkdir
3. Run smbclient –I <ipaddress> -U user –L NetBIOSComputername to check you can see the shares
4. Add an entry to /etc/fstab

//server/share /mountdir smbfs ip=ipaddress,username=user,password=pass,noauto 0 0

5. Mount the remote file system with mount /mountdir
6. Change directory into mount point and the directory will no longer be empty but will appear just like a mapped drive.

Alternatively, if you just want to map temporarily to a Microsoft host and not have to modify fstab, then use smbclient interactively as shown:

smbclient //ipaddress/share –U NThost\NTuser Password: ******
Domain=[TAUPO] OS=[Windows 5.0] Server=[Windows 2000 LAN Manager] smb: \>

1. Change the ESX Server which is to be the NFS server, to use medium security (using the MUI is easiest for this).

2. Add an entry to the file /etc/exports on the NFS server by either using vi text editor (/vmimages *) or use the exportfs command

3. Check that the NFS client IP address or hostname is not excluded by the server file /etc/hosts.deny

4. It is up to you if you explicitly allow the NFS client by adding the NFS client to /etc/hosts.allow on the NFS server

5. Start the NFS server daemons
  a. /etc/rc.d/init.d/service portmap start
  b. /etc/rc.d/init.d/service nfslock start
  c. /etc/rc.d/init.d/service nfs start

6. Start the NFS client daemons
  a. /etc/rc.d/init.d/service netfs start
  b. /etc/rc.d/init.d/service portmap start

7. Mount remote export directory (on the NFS server) from the NFS client with the command
  a. mount –t nfs nfsserver:/export localdir/localmount

showmount –e nfsserver

This command can be specified with the hostname name or IP address of the NFS server holding the exported directories.

Amongst other things, NIS can ensure that the numeric user IDs are unique across the organisation. This is because numeric user IDs are used in NFS, so we can have a mistaken identity situation as user id 515 on a nfs client will not be the same as user id 515 on a nfs server.
 

vmware-mount.pl –p /vmfs/VMFS-VOL1/win2k3.vmdk

--------------------------------------------
VMware for Linux - Virtual Hard Disk Mounter Version: 1.0 build-9638
Copyright 1998 VMware, Inc. All rights reserved. -- VMware Confidential
--------------------------------------------

Nr      Start       Size Type Id Sytem
-- ---------- ---------- ---- -- ------------------------
1          63    8160957 BIOS  7 HPFS/NTFS

If we actually want to mount a partition then we need to be specific and create a directory (or use an existing) to be our mount point.

mkdir /myntfs
vmware-mount.pl /vmfs/VMFS-VOL1/win2k3.vmdk 1 –t ntfs –o ro /myntfs

This command will tie up the console window hence you’ll need to spawn a new window first to navigate to /myntfs to view the contents.

mount -t nfs [-o options] esx4:/vmimages /root/other_server_vmimages

To mount a cd, you could just use

mount /mnt/cdrom

For defined mount points we use the file /etc/fstab.

We can also use mount to gain access at the service console to an ISO image or floppy disk image using the following mount syntax:

mount -o loop cdname.iso mount-directory

If you want to make your own ISOs then you can use the utility mkisofs (not included in the service console) to select the files and create the ISO file, then use the cdrecord utility to write to device.

LABEL=/            /           ext3    defaults 1 1
LABEL=/boot        /boot       ext3    defaults 1 2
none               /dev/pts    devpts  gid=5,mode=620 0 0
LABEL=/home        /home       ext3    defaults 1 2
none               /proc       proc    defaults 0 0
none               /dev/shm    tmpfs   defaults 0 0
LABEL=/vmimages    /vmimages   ext3    defaults 1 2
/dev/sda3          swap        swap    defaults 0 0
/dev/cdrom         /mnt/cdrom  iso9660 noauto,owner,kudzu,ro 0 0
/dev/fd0           /mnt/floppy auto    noauto,owner,kudzu 0 0
//taupodc1/msfiles /root/share smbfs   ip=192.168.1.150,credentials=/root/.smbcreds,noauto 0 0

If we have smb mount points defined in the fstab file, then this file could end up with user credentials in it. The fstab file is readable by everyone so this would not be good. We can place the credentials for the smbmount in a hidden secured file in our home folder eg. /root/.smbcreds

echo username=user > .smbcreds echo password=pass >> .smbcreds chmod 600 .smbcreds

Then in the /etc/fstab file we substitute the username and password for credentials=/root/.smbcreds. Therefore the whole line in the fstab would be

//server/share /mountdir smbfs ip=ipaddress,credentials=/root/.smcreds,noauto 0 0

The noauto option specifies that this mount point should not be automatically mounted at boot. The administrator will mount and umount this as is required.

The 0 0 at the end of the line specifies backup pass and fsck pass

The backup pass flag relates to backup methods and generally you won't be backing up remote mounts in the service console.

The fsck pass flag is a kind of dirty flag. If a file system were un-cleanly closed, then fsck would check that flag on next boot and fix any errors found. Again, in the ESX service console, it's likely your mount points will be remote and will employ their own file system checks.

Therefore, your custom entries in /etc/fstab will typically terminate with the text noauto 0 0

wget http://woody.linif.org/vmconffix.sh

# ~/.bash_logout
clear

To re-use one of your previous commands, just enter an exclamation mark followed by the numeric ID of the command you wish to re-use.  For example, here we are using the history to view the commands and then re-using one by its numeric ID.

[ali@esx1 ali]$ history
 1  vdf -h
 2  ls -k
 3  cat /etc/hosts

[ali@esx1 ali]$ !2
-rw-rw-r--   1   ali   ali   27  Feb  4  2005  fruit
-rw-r--r--   1   ali   ali  463  Nov 23  2004  lilo.conf.backup
-rw-rw-r--   1   ali   ali   42  Feb  4  2005  scores

The history command is dependent upon the ~/.bash_history file being present and populated with data.

A great variation on this is just to use CTRL-R at the command line. This brings up a searchable command history which is very powerful...try it out!

This is the equivalent of the CLS command found in MS-DOS and the Windows command prompt. A great shortcut way of doing a clear is CTRL-L, what is very cool, is that if you are in the middle of typing a command you can do a CTRL-L and the screen is cleared but your command line is still maintained!

If the date is incorrect and you wish to reset it you would enter the command with the -s switch and specify date in dd/m/yyyy format.

# date -s "12/29/2007 23:48"

Once you have set the date, you will want to ensure that the hardware clock matches your newly entered date. We can do this with the hwclock command described below.

# hwclock

If we want to synchronise the hardware clock with the service console date and time, we use the following:

# hwclock -systohc
 

# cal -3
    March 2006            April 2006             May 2006
Su Mo Tu We Th Fr Sa  Su Mo Tu We Th Fr Sa  Su Mo Tu We Th Fr Sa
          1  2  3  4                     1      1  2  3  4  5  6
 5  6  7  8  9 10 11   2  3  4  5  6  7  8   7  8  9 10 11 12 13
12 13 14 15 16 17 18   9 10 11 12 13 14 15  14 15 16 17 18 19 20
19 20 21 22 23 24 25  16 17 18 19 20 21 22  21 22 23 24 25 26 27
26 27 28 29 30 31     23 24 25 26 27 28 29  28 29 30 31
                      30

Surprisingly useful!

vmware-cmd –l

When you register a VM, it is automatically added to this file. The order in which VMs appear listed in the MUI is dependent upon the order in which the VMs are listed in this file.

ps –A ps –eaf
ps –eaf |grep vmware-serverd
ps –efw

The -f switch

is useful as the “w” indicates wide format, so we can see the full directory path to the vmx file.

Another good option is the H option to show the process hierarchy in a similar way to pstree.

which might keep Solaris people happy as we don't have the ptree utility in Linux.

[root@esx1 root]# pstree

init-+-crond
     |-gpm
     |-httpd---3*[httpd]
     |-keventd
     |-khubd
     |-4*[kjournald]
     |-klogd
     |-5*[mingetty]
     |-scsi_eh_0
     |-snmpd
     |-sshd---sshd---bash---pstree
     |-syslogd
     |-vmfs_flush
     |-vmklogger
     |-vmkstatus---sleep
     |-vmware-ccagent---vmware-ccagent
     |-5*[vmware-vmx-+-vmware-mks]
     |               |-2*[vmware-vmx]]
     |               `-vmware-vmx---vmware-vmx]`-xinetd

You can display this process hierarchy with process ID numbers (PID) using the -p switch. To specify that the utility lists the processes with their command line arguments the -a switch should be used

# pstree -ap

# ps –eaf |grep vmware
# renice –p <pid of vmware-serverd>
# renice –p <pid of httpd.vmware>

To reset the PID of these processes back to their defaults, use renice again to set the priority to zero.

# pidof vmware-authd

$ sleep 900
CTRL-Z
[1]+ Stopped
$ bg 1
$ jobs

Now the job will be running in the background. If you want to start a process in the background just add a "&" to the end of the command.

$ sleep 900 &
jobs
[1]+  Running       sleep 900
$ fg 1
 

You still need to launch the process in the background when using the nohup command, i.e. after your command you need an "&" character. For example:

nohup sleep 900 &

# sleep 10 &

If we know a process will take a while and we need the interactive command prompt back, then it's easier to launch the process this way, rather than start it, CTRL-Z and using bg to place it in the background.

By default this value is set to 8, which means the VMkernel scans LUNs 0 to 7 on start up. So, if we what to scan up to LUN number N we must set Disk.MaxLUN to N+1. If you change this setting away from the VMkernel default, then the following file is created (or modified if it exists already).

/etc/vmware/vmkconfig

The safest way to update this parameter is by using the MUI in the Options tab, Advanced Settings. If you wish to inspect this value in the command line you can

# cat /etc/vmware/config/Disk/MaxLUN

vmhba0:0:4,6-255 would scan 0,1,2,3,5 i.e. skip 4 and skip 6 through 255

vmhba0:0:3,4,9-255 would scan 0,1,2,5,6,7,8 i.e. skip 3 & 4 and skip 9 through 255

If you have multiple paths to LUNs you wish to mask, you will need to supply a mask that masks LUNs on all available paths to those LUNs.

If you want to remove a setting from this file, either remove the offending line from this text file by manually editing it, or you can use the MUI (Options Tab, Advanced Settings) and enter a value of "" (i.e. two quotation marks). Unfortunately as of ESX 2.5 just deleting the existing value in the MUI will not work.

vmkfstools -s vmhba0

However, this has been known to cause problems in the past, hence the development of a script called cos-rescan.sh to help.

Minicom uses a configuration file to determine bit rates etc. This configuration file is placed in the /etc directory. We normally create the file with a meaningful name e.g. minirc.com1, so to launch the tool we enter

# ./minicom com1

The contents of the minirc.com1 file would typically be:

pr port		/dev/ttyS0
pu baudrate	19200
pu bits		8
pu parity	N
pu stopbits	1
pu rtscts	No

Much more detail on minicom can be found at http://www.cisl.ucar.edu/nets/intro/staff/siemsen/tools/minicom.html

vmkmultipath -q                     Query multipath

vmkmultipath -s -p policy     sets path policy

vmkmultipath -s -r path        sets active path

vmkmultipath -S                     save configuration

[root@esx1 vmimages]# free -m
             total       used     free      shared       buffers      cached
Mem:           186        180        5           0            11          36
-/+ buffers/cache:        133       53
Swap:          382          1      380

Frustratingly, this doesn't have a -h switch for human readable as the df tool does, so we need to specify -k, -m or -g for kilobytes, megabytes and gigabytes respectively.

 6:38pm up 2 days, 4:59, 17 worlds, load average: 0.00, 0.00, 0.00, 0.00
PCPU: 1.26%, 0.00% : 0.63% used total
LCPU: 1.26%, 0.00%, 0.00%, 0.00%
MEM: 3931136 managed(KB), 3668992 free(KB) : 6.67% used total
SWAP: 4127744 av(KB), 0 used(KB), 4086468 free(KB) : 0.00 MBr/s, 0.00 MBw/s
DISK vmhba0:0:0: 0.00 r/s, 0.98 w/s, 0.00 MBr/s, 0.00 MBw/s
DISK vmhba1:0:12: 0.00 r/s, 0.00 w/s, 0.00 MBr/s, 0.00 MBw/s
DISK vmhba1:0:11: 0.00 r/s, 0.00 w/s, 0.00 MBr/s, 0.00 MBw/s
DISK vmhba1:0:1: 0.00 r/s, 0.00 w/s, 0.00 MBr/s, 0.00 MBw/s
DISK vmhba1:0:0: 0.00 r/s, 0.00 w/s, 0.00 MBr/s, 0.00 MBw/s
NIC vmnic2: 0.00 pTx/s, 0.00 pRx/s, 0.00 MbTx/s, 0.00 MbRx/s
NIC vmnic1: 0.00 pTx/s, 0.98 pRx/s, 0.00 MbTx/s, 0.01 MbRx/s
NIC vmnic0: 0.00 pTx/s, 0.00 pRx/s, 0.00 MbTx/s, 0.00 MbRx/s

VCPUID WID WTYPE   %USED %READY %EUSED %MEM
 144   144 vmm      0.00   0.00   0.00 3.00
 142   142 migServ  0.00   0.00   0.00 0.00
 141   141 driver   0.00   0.00   0.00 0.00
 140   140 driver   0.00   0.00   0.00 0.00
 139   139 reset    0.00   0.00   0.00 0.00
 138   138 reset    0.00   0.00   0.00 0.00
 137   137 helper   0.00   0.00   0.00 0.00
 136   136 helper   0.00   0.00   0.00 0.00
 135   135 helper   0.00   0.00   0.00 0.00
 134   134 helper   0.00   0.00   0.00 0.00
 133   133 helper   0.00   0.00   0.00 0.00
 132   132 helper   0.00   0.00   0.00 0.00
 131   131 idle     0.00   0.00   0.00 0.00
 130   130 idle     0.00   0.00   0.00 0.00
 129   129 idle     0.00   0.00   0.00 0.00
 128   128 idle     0.00   0.00   0.00 0.00
 127   127 console  0.00   0.00   0.00 0.00

  6:44pm  up 2 days,  5:06,  1 user,  load average: 0.00, 0.00, 0.00
42 processes: 41 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  0.6% user,  0.4% system,  0.0% nice,  9.0% idle
Mem:   191240K av,  185412K used,    5828K free,      52K shrd,   12280K buff
Swap:  391672K av,    1892K used,  389780K free                   36980K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
18894 root      10   0  1024 1020   824 R     0.9  0.5   0:00 top
    1 root       8   0   504  492   436 S     0.0  0.2   0:04 init
    2 root       8   0     0    0     0 SW    0.0  0.0   0:00 keventd
    3 root      19  19     0    0     0 SWN   0.0  0.0   0:00 ksoftirqd_CPU0
    4 root       9   0     0    0     0 SW    0.0  0.0   0:00 kswapd
    5 root       9   0     0    0     0 SW    0.0  0.0   0:00 kreclaimd
    6 root       9   0     0    0     0 SW    0.0  0.0   0:00 bdflush
    7 root       9   0     0    0     0 SW    0.0  0.0   0:00 kupdated
   12 root       9   0     0    0     0 SW    0.0  0.0   0:01 kjournald
   88 root       9   0     0    0     0 SW    0.0  0.0   0:00 khubd
  283 root       9   0     0    0     0 SW    0.0  0.0   0:00 kjournald
  284 root       9   0     0    0     0 SW    0.0  0.0   0:00 kjournald
  285 root       9   0     0    0     0 SW    0.0  0.0   0:00 kjournald
  655 root       9   0     0    0     0 SW    0.0  0.0   0:00 vmfs_flush
  785 root       9   0     0    0     0 SW    0.0  0.0   0:00 scsi_eh_0
 1092 root       9   0   472  460   396 S     0.0  0.2   0:00 vmklogger
 1198 root       9   0 23252  22M  2292 S     0.0 12.1   7:17 vmware-ccagent

The sample output shown above is static, but the actual output of the tool is continually changing as the processes are running.

vmstat 2 3

would run the tool every 2 seconds for 3 times and then exit, producing an output similar to the following:

  procs                    memory   swap       io               cpu
r  b  w  swpd  free   buff  cache  si so   bi  bo   in    cs  us  sy  id
0  0  0     0 13132  51032  64924   0  0    0  28   20    41   1   1  41
0  0  0     0 13132  51032  64924   0  0    0   0  168   517   0   0 100
0  0  0     0 13132  51032  64924   0  0    0   0  167   500   0   2  98

Under the swap heading in the output are the column headers "si" and "so" which correspond to swapped-in and swapped-out.

Under the procs heading, the r b w column headers correspond to

r   = process is in run queue
b  = process is blocked for resources I/O
w = process is swapped

The general rule is that if r is consistently greater than the number of physical processors in the box, then the system will be slow. However, given that the ESX Server service console can only use physical CPU0, the service console rule should be if r is consistently greater than 1, the service console will be slow, directly impacting your ability to manage the ESX server. Poor performance could manifest itself as poor MUI or remote console performance.

Under the cpu heading, the us sy id headers correspond to

us  = cpu user time
sy  = system time
id  = idle time

There is something to be aware of in the Linux service console about swap. If the service console runs out of swap, then the survival instincts of the Linux kernel kick in! Linux will kill off other processes at random to keep itself alive, watch for this should your MUI go down, don't just re-start it, check why it stopped by checking service console RAM using the free command and the vmstat command.

cat /proc/vmware/meminfo

In a subdirectory called /vm there are subdirectories for each vm labelled by number How do you find out what number corresponds to what VM? Well, you can either just look in the "Status Monitor" tab in the MUI, or go to the command line and run B2V's listworlds.pl script or search each vmware.log file for each VM and locate it's worldid:

grep –i worldid= ~ali/vmware/ISAserver/vmware.log

Some other examples of great info that can be extracted are outlined in the following table:

/proc/vmware/vm/139/cpu affinity 0,1,2,3
/proc/vmware/vm/139/cpu shares 1000
/proc/vmware/vm/139/cpu status
/proc/vmware/vm/138/cpu hyperthreading

/proc/vmware/sched/cpu

vmkusagectl install

to install the utilisation web pages & setup a cron job vmkusage –graph to generate graph images.

ESX 2.5.0      vmkusage-v2.5.0v2

Sometimes this tool loses track of which VMs it should be charting. When this happens try using

vmkusage -rescan

If that fails, then the more aggressive command

vmkusage –regroove

can be used to to wipe the charting data database and start logging stats again.

If you are doing specific troubleshooting, the vmkusage tool has a hyper switch which enables sampling data at a much higher frequency, but logging is only recorded every 1 minute. It is recommended you only use hyper on non-production systems and even then only for specific troubleshooting.

vmkusage - hyper

There is another feature of vmkusage which allows generating of text performance reports at the command line.

vmkusage - report -reportfile /home/ali/vmkreport

There is a tool called vmktree which many customers like to use with this tool, it can be found at http://tihlde.org/~larstr/vmktree/

http://esxserver/hstatus

Not sure if we need to be logged in for this to work, but vmkusage does appear to be required. We get loads of output on this page, similar to running a number of command line tools. This is part of what looks like the legacy web interface to ESX server, i.e. it doesn’t look as cool as the MUI of ESX 2.x.

This legacy web output is not supported in the current release and could contain erroneous information.

When customers install vmkusage they see a message stating that a cron job has been added, however crontab will not list the new job as it is not added under the context of the root user id.

We supply the script the vmx file as a parameter to this script which will be read to find out where the virtual disk is.

If we use this script with the -l switch we are stating that the resulting snapshot should be stored on the local server. The default path will be /vmimages/backup?

If we choose to store the archive on another server, then the snapshot still occurs locally, but once complete, the archive is copied automatically using scp to the target archive server.

Remember the backups produced are crash consistent. This means when you restore a snapshot image, the OS will start a file system that was not cleanly shut down, and is therefore consistent with an OS that has just crashed.

In the following example, the -l switch has been used to specify the backup only occurs locally.

[root@esx1 root]# vmsnap.pl -c /home/vmware/w2k/w2k.vmx -l

vmsnap: VM config file is '/home/vmware/w2k/w2k.vmx'
vmsnap: This VM will only be backed up locally.

vmsnap: Disks found that are in use:
vmsnap: Disk found: scsi0:1 (VMFS-For-All-Servers:w2k.vmdk)
vmsnap: 1: scsi0:1
vmsnap: Creating the RedoLogs for cfg : /home/vmware/w2k/w2k.vmx
vmsnap: scsi0:1:Adding REDO
vmsnap: scsi0:1: BACKUP STARTING NOW...
local dir : /vmimages/localbackup has 5432340k space
/vmfs/VMFS-For-All-Servers/w2k.vmdk size : 2097152 k
2) diskname = VMFS-For-All-Servers:w2k.vmdk
2) disknameprefix = VMFS-For-All-Servers:w2k.vmdk
3) disknameprefix = VMFS-For-All-Servers:w2k
4) disknameprefix = VMFS-For-All-Servers:w2k
disknameprefix = VMFS-For-All-Servers:w2k
Exporting disk VMFS-For-All-Servers:w2k.vmdk:
Export: 100% done.

vmsnap: scsi0:1: Adding stacked REDO.REDO
vmsnap: scsi0:1: Committing REDO
Please wait, committing disks...
vmsnap: scsi0:1: Committing REDO.REDO
vmsnap: Backing up the cfg : /home/vmware/w2k/w2k.vmx

[root@esx1 root]#

Some users have run into problems with vmsnap.pl reporting that their virtual disk is already in REDO mode and cannot be snapshot backed up. Be sure to check that previous snapshot backups ran successfully and there was sufficient disk space.

[root@esx1 root]# wwpn.pl

vmhba1: 210000e08b17b3f6 (Qlogic) 6:1:0

If you run the script with the -v switch for verbose output, you also get the WWN information for the storage processors on SAN as well as the WWN for the ESX server hba.

[root@esx1 root]# wwpn.pl -v

WWPN 1.02 Copyright VMware 2003
Display WW port names and VMHBA information for fibre channel adapters
For each vmhba here are the corresponding Qlogic and Emulex WW Port Names
 Adapter                           WWPN PCI (decimal)
vmhba1: 210000e08b17b3f6 (Qlogic) 6:1:0 /proc/scsi/qla2300/0
vmhba1:0: 500508b30090ec31 scsi-qla0-port-0=500508b30090ec30:500508b30090ec31;
vmhba1:1: 500508b30090ec39 scsi-qla0-port-1=500508b30090ec30:500508b30090ec39;

The utility is called wwpn because it is listing the worldwide port name for your fibre channel hba. What can be confusing is that there is a WWN for the node.

In the following example, we add a REDO file to the powered-on VM called SUNone. As shown, the script parses the configuration file and then creates a REDO file of the same name as the virtual disk but with a file extension .REDO. The key point here is that this can be performed against a running VM.

[root@esx1 root]# vmAddRedo.pl -c /home/ali/vmware/SUNone/SUNone.vmx

vmAddRedo: VM config file is '/home/ali/vmware/SUNone/SUNone.vmx'

vmAddRedo: Disks found that are in use:
vmAddRedo: Disk found: scsi0:0 (VMFS2-VOL1:SUNOne.vmdk)
vmAddRedo:    1: scsi0:0
vmAddRedo: Creating the RedoLogs for cfg : /home/ali/vmware/SUNone/SUNone.vmx
vmAddRedo:    scsi0:0:Adding REDO
vmAddRedo: diskname :/vmfs/VMFS2-VOL1/SUNOne.vmdk

The REDO file is 16MB in size at creation and then grows further 16MB blocks as disk write operations dictate. The REDO file will not exceed the size of the original virtual disk. This is a delta file, not a REDO log, so a disk can only be 100% different from the original.

The command line options are:

vmAddRedo [-c config_file] [-g] [-h] [-m]
        -c config_file  Specify a VM configuration file to use for vmAddRedo
        -g              List all available VM's for backup
        -h              Help
        -m              Generate the man page for this program

 

The command line options are:

vmCommit [-c config_file] [-g] [-h] [-m]
        -c config_file Specify a VM configuration file to use for vmCommit
        -g List all available VM's for backup
        -h Help
        -m Generate the man page for this program

This Perl script runs on virtual terminal 1 (tty1) of the ESX server and provides the default console screen. This script is bound to tty1 in the initialisation table /etc/inittab. This script can be run at anytime from the command line and an output similar to the following would be seen.

cp source-file destination-file

cp -a

cp -l creates a link and is an alternative to using the ln utility to create a hard link.

Secure copy tool, used to copy files from one Linux host to another. For example if we are copying a virtual disk in COW (sparse) format from the service console of one ESX server to the /vmimages directory on another, then this should do the trick.

scp w2k*.vmdk root@new-server:/vmimages/

If you need to copy a directory, use the -r switch for recursion.

If you want to copy files from your Windows PC to the service console, e.g. you've just used your CD burning software to create an ISO file and now you want it up in /vmimages, then you could use the Windows freeware pscp which comes from the authors of the SSH client PuTTY. This Windows command line utility can be found at http://www.chiark.greenend.org.uk/~sgtatham/putty/

Alternatively, you could use a Windows GUI tool such as WinSCP, if using the command line is not your preference. This tool can be found at http://winscp.net/eng/index.php

A further option available to you is the free tool called Veeam, available from www.veeam.com. The advantage of this tool is that is extremely fast at file transfers, a typical transfer of 30 minutes could be be cut to 5 minutes! Try it and see!

mv old-filename /new-dir/new-filename

The purpose of the link is to allow you to access a file or directory that is located another directory by using a file in the current directory. For example, if you wanted to access the IP configuration file /etc/sysconfig/network-scripts/ifcfg-eth0 and you wanted to simply access this file using a filename in your home directory using a simple name like "ipconf", you could create a link to it using the ln utility.

# ln /etc/sysconfig/network-scripts/ifcfg-eth0 ipconf

You can confirm this has worked when you perform an ls -al as the number shown after the file permissions indicates the number of hard links to the same inode, in the following example, the link count is 2.

-rw------- 2 root root 83 Nov 11 2004 ipconf

There are actually 2 types of link that can be created, hard and soft. What we've just done above is a hard link. A hard link is where you have 2 file names either in the same or different directories which point at the same data on disk. As the two file names are linked directly to same file data and file attributes, if for example you change the permissions on one of the files, you are changing the other file as they are pointing at exactly the same file on disk - known as an inode.

A soft link, more commonly known as a symbolic link, is where you create a pointer file to the real file that contains the data. In many ways this is like a shortcut file in Windows - i.e. a LNK file. To create a symbolic link we still use the ln utility, but with the -s switch.

# ln -s /etc/sysconfig/network-scripts/ifcfg-eth0 ifconflink

If you now do a ls -al on the directory where you created the symbolic link, we get something like the following:

lrwxrwxrwx 1 root root 41 May 6 20:56 ifconf -> /etc/sysconfig/network-scripts/ifcfg-eth0

Notice that in the file description, the first byte of the file permissions, the "l" indicates that the file is in fact a symbolic link.

The best definition I've found so far for the exact differences between a hard and a soft link can be found at http://linuxgazette.net/105/pitcher.html. Thanks to Lew Pitcher for publishing this great article.

# rm testfile
rm: remove `testfile'? y

If you need to remove all the files in a directory then we could use recursion with the -r switch

# rm -r /olddata/

Be careful if using wildcards like * with this tool.

# shred secretfile -u

mkdir /vmimages/iso

You can create multiple directories at the same time using this tool simply by supplying multiple directory parameters separated by the space character as shown:

mkdir /vmimages/iso /home/alistair/scripts /tmp/downloads

which would create the 3 directories listed.

wall This server will be shutdown in 1 hour