Category Archives: VMware

Powershell, Tools, VMware

Simply check if login on an ESXi works and return it’s license

Leave a comment

Just a simple script to test ESXi Connectivity and return the current license state of the server by using PowerCLI

param(
    [Parameter(Mandatory = $true, HelpMessage = 'Provide username for login on ESXi')]
    [String] $username,
    [Parameter(Mandatory = $true, HelpMessage = 'Provide password for login on ESXi', ParameterSetName = 'Secret')]
    [Security.SecureString] $password

)

#check user/pwd
if($username -like "" -Or $password -like "")
{
    Write-Host -ForegroundColor Red "Username/Password seems wrong"
    exit(1)
}

#define domain
$domain="forensik.justiz.gv.at"

#disable certificate checking as we have self signed certs
Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false

$hosts="SITE01-ESX01","SITE01-ESX02","SITE01-ESX03","SITE01-ESX04","SITE01-ESX05","SITE01-ESX06","SITE02-ESX01","SITE02-ESX02","SITE02-ESX03","SITE02-ESX04","SITE02-ESX05","SITE02-ESX06" 



foreach($var_host in $hosts)
{
    Write-Host -ForegroundColor Yellow $var_host"."$domain
    Connect-VIServer -Server $var_host"."$domain -User $username -Password ([Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($password)))
    $(Get-VMHost).Name
    $(Get-VMHost).Version
    $(Get-VMHost).LicenseKey
    $(Get-VMHost).Uid
    Disconnect-VIServer $var_host"."$domain -WarningAction SilentlyContinue -Confirm:$false
    Write-Host -ForegroundColor Cyan "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
}

VMware

VM stuck in invalid state after export

Leave a comment

After trying to export a VM and cancelling the export it could happen that the VM is nolonger responsive (no start, unregister, delete, … possible)

The hostd.log (in the /var/log directory on the ESXi) will show an error similar to:

2021-09-07T10:55:19.583Z error hostd[2099544] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/XXXXXXXX-05c01598-574e-88d7f6d5ef52/myvm/myvm.vmx opID=esxui-6c72-aafe user=root] Invalid transition requested (VM_STATE_EXPORTING -> VM_STATE_DELETING): Invalid state
2021-09-07T10:55:19.583Z warning hostd[2099544] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/XXXXXXXX-05c01598-574e-88d7f6d5ef52/myvm/myvm.vmx opID=esxui-6c72-aafe user=root] Method fault exception during VM destroy: Fault cause: vim.fault.InvalidPowerState

It seems the VM is somehow stuck in the Exporting-state and therefore no other operation is possible on the VM .

As a workaround log in to the ESXi Host and restart the Management Agents (https://kb.vmware.com/s/article/1003490)

Powershell, Tools, VMware

PowerCLI batch revert to Snapshot

Leave a comment

The following script-let can be used to revert a bunch of VMs/all VMs in a folder back to the first snapshot made for the VM

$vms = Get-Folder "MY_VM_FOLDER_NAME" | Get-VM
 
foreach($vm in $vms)
{
    $vm.name
    $snap = Get-Snapshot -VM $vm | Sort-Object -Property Created -Descending | Select -First 1
    Set-VM -VM $vm -SnapShot $snap -Confirm:$false
    echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
}
Linux, Monitoring, VMware, Zabbix

vCenter alarm Polling (v2) – Cross platform version (Windows & Appliance)

Leave a comment

As it has been some time since my last post about a solution on how to get vCenter alarms to Zabbix, and VMware also evolved I followed a new approach on that topic as my initial post only supports Windows vCenters. Furthermore the solution is not as stable as I wished that it would be, so my new approach is to query all alarms from a vCenter via it’s SDK.
Initially all the alarms are discovered and created in Zabbix and in a second step the values for the discovered alarms are polled.
Currently the script used the data center object of the vCenter to discover alarms, so it can’t be used on a standalone ESXi-Server. However – if the code is changed to use whatever object is needed to get the alarms directly from the ESXi-server it should also be possible to get alarms directly from a server without the need of a vCenter (but I didn’t implement that till now as there wasn’t the need/time).

vCenter alarms – SDK (tested with ESXi 6.0+ and Zabbix 3.0 on RHEL 7)

To install the vCenter alarms the attached zip needs to be downloaded and the VMware Perl SDK must be installed on the Zabbix Server.
The template needs to be imported into Zabbix and the vCenter username and password need to be set in the username/password macros of the template.

The other two files (vcenterAlarms.pl & vcenterAlarms.wrapper) need to be extracted to the externalscripts folder of the Zabbix Server. The wrapper script is just a shell script that is executed by a Zabbix item to call the per script and send the that to Zabbix via Zabbix Sender.  As the VMware API is quite slow the wrapper also starts itself again with NOHUP because otherwise the timeout defined in the Zabbix Server configuration would cause an exit of the script. For my setup it always took longer than 30 Seconds till tall data where gathered and therefor the Zabbix Server would kill the script in the middle of the execution and no data would be sent to Zabbix. That’s why I added this workaround. Furthermore it also checks if there are less than eleven vcenterAlarms.wrapper processes running, and only starts if there are less, to ensure that Zabbix does not spawn hundreds of NOHUP-processes.

 

 

 

 

Network, VMware

VMware Workstation Player – No Bridge Adapter available

Leave a comment

Lately Microsoft convinced me to upgrade my Windows 7 @ home up to Windows 10. When I upgraded my Windows I checked all the installed tools for Upgrades and also upgrade my old VMware Player 6 to the new VMware Workstation Player 12.
Today I was playing with MDT at home and wanted to set up a test VM to check if everything is working, but I was not able to get the bridged interface working.
I was able to selct it, but I didn’t get an IP from my DHCP so I thought I’ll disable all adapter except the LOM which is connected to my Router, but there were no adapters. 🙁

VMwareWokrstationPlayerBridge

 

After a little bit of investigating I found out, that my LOM didn’t have the VMware Bridge Service installed. VMwareBridgeService

After installing the service I was able to set up the bridge adapter for the VM.

 

 

Monitoring, Powershell, VMware, Windows

Powershell/PowerCLI very slow execution Time

Leave a comment

Sometimes a PowerCLI-script can take quite some time till everything is executed. For example the PowerShell scripts used by Zabbix to gather the vCenter alarms into Zabbix (BlogPost) need some tuning to run fine.

So why are scripts running slow in some cases?
It seems to occur primary on systems which do not have a connection to the internet. As a matter of fact – most of the systems I’m setting up “lose” internet connection sooner, or later. :/

What exactly causes the problem?
While investigating that problem i found an interesting feature which seems to cause the problem – certificate checks!
There is an IE-setting which is named “Check for publisher’s certificate revocation ” and can be found at: Intenet Options -> Advanced -> Section: Security ->Disable: Check for publisher’s certificate revocation.

Disabling the certificate checks improves the execution time by about 60%.

certificate-check enabled:

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 41
Milliseconds      : 536
Ticks             : 415369143
TotalDays         : 0.000480751322916667
TotalHours        : 0.01153803175
TotalMinutes      : 0.692281905
TotalSeconds      : 41.5369143
TotalMilliseconds : 41536.9143

 

certificate-check disabled:

Days : 0
Hours : 0
Minutes : 0
Seconds : 16
Milliseconds : 262
Ticks : 162628208
TotalDays : 0.000188227092592593
TotalHours : 0.00451745022222222
TotalMinutes : 0.271047013333333
TotalSeconds : 16.2628208
TotalMilliseconds : 16262.8208

 

If the script is run from a normal user account everything should be fine and we have an improved execution time, BUT …

… if the script is run from an Service (and as a matter of fact I’m using the Zabbix agent service to run the script) we got a problem.
With default settings the Zabbix Agent is installed to run as nt authority\system, so if the IE-setting is changed for the current user, its working for this user, but not for the system user. 🙁
So a quick and dirty workaround could be to disable the setting for the system user.
ATTENTION: Running the Zabbix Agent as a system user is OK for a DEV-environment, but should not be used in an production environment. For production a dedicated service user should be used.

I disabled it by becoming a system user with

[pastacode lang=”bash” message=”” highlight=”” provider=”manual”]

PSEXEC -i -s -d CMD

[/pastacode]

and launching the IE from the command prompt. Afterwards I was able to disable the setting via the above method.

Otherwise the Key could also be found in the registry at:

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing\State
0x00023e00 / 146944 Check OFF
0x00023c00 / 146432 Check ON

A simple PS-Script to disable the setting would be:

[pastacode lang=”bash” message=”PowerShell to disable Publisher certificate checks” highlight=”” provider=”manual”]

Write-Host "Disable Check for publisher’s certificate Revocation"
set-ItemProperty -path "REGISTRY::\HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing\" -name State -value 146944

[/pastacode]

 

Linux, Monitoring, VMware, Zabbix

Get vCenter alarms into Zabbix via poll-method

Leave a comment

Some time ago i wrote a post on how to forward vCenter alarms to Zabbix ( https://blog.fawcs.info/2015/05/getting-vcenter-alarms-to-zabbix/) and I have to admit, that this solutions is kind of a pain in the ass. I’m getting the alarm info from environmental varaibles which are automatically set by the vCenter when an alarm changes its status, but it seems that there is a “littel” problem with “overlapping” alarms. For example if there are occuring multiple alarms within a short period only the first alarm will be forwarded to zabbix, but non of the follwoing alarms. Besides that this is not an ideal solution I personally do not like my former approach because it’s an event driven approach. So if one event goes missing we have an inconsistent system :/

It’s quite some time since I wanted to redesign the solution and now I’m finally having some time ( and the pressure) to do so. 🙂
The new approach is based on using userparameters to execute a powershellscript on the vCenter to discover all active alarms and create items in Zabbix. At the moment I’m creating three item prototyes. One for the Timestamp when the alarm became active, another item for the acknowledged-state of the alarm and the last one for the severity of the alarm.

There are two userparemeters which run two powershell scripts. The first one (vcenter.alarm.polling.discovery.ps1) does the discovery and the second one (vcenter.alarm.polling.itemdata.ps1) is to get the data for the discoverd items.
There are also three triggers (one for each severity gray, yellow, red) which will be active als long as the alarm is not acknowledged.

You can download the scripts, userparameters and the template down below:
vCenterAlarmPolling

 

Additional findings:
Ther can occure problems if there are different addresses used to connect to the vcenter (eg. 127.0.0.1, loclahost, vcenterhostname, …)
It seems that the vCenter creates a sperate datacenter instance for every connection, so if you use the three examples from abovve you will end up creating three instances and mess up the script.

 

If special characters want to be passed to the powershellscript (e.g. special chars in passwords ord login with administrator@vsphere.local) the “UnsafeUserParameters=1” – parameter from the zabbix-agent.conf needs to be set to 1. (default value is 0)

VMware

VMware 6.0 vCenter Webclient blank section in the middle (blank middle frame) – VSAN Health Plugin

Leave a comment

It seems that the VSAN Health Plugin could break the vCenter webclient if it’s not installed correctly. After installing the MSI-Packge on the Windows vCenter server and logging in via the web client everything seemed ok at the begining, but after selectin the datacenter, a cluster or a host/vm the middle section whoich sould display dietalled informatons about the selected property did not load and stayed blank.

After some searching I found an interssting VMware KB-article which described my problem. My vCenter looked like the screenshoot in the below article.

https://communities.vmware.com/thread/510468?start=0&tstart=0

It seems that I made the mistake and installed the VSAN health plugin as a domain admin – and that just does not work.  After uninstalling the plugin, restarting the vCenter, logging in as a local admin, starting a command prompt with admin priviliged and restarting the installation again, it woked fine.

 

BTW. in the new VSAN health plugin releases VMware fixed the DRS-dependency and now its possible to also install the plugin without activated DRS.  🙂

Before you had to install the Plugin while your system had the evaluation licens where you could activate DRS. If you use a license like  Essentials Plus + VSAN that could be a real problem.

Linux, Monitoring, Tools, VMware, Zabbix

Getting vCenter alarms to Zabbix

7 Comments

VMware is a relay nice product, but there is one little problem. It’s realy hard to monitor VMware products with SNMP or any other “old school” technologies.
The actual problem is to get an alarm in Zabbix if there occures an error on the vCenter. So Zabbix is used as an umbrella monitoring for the whole environment.
All this could also be done with SNMP-Traps what would be a lot easier – at first appereance, but Zabbix is … how do I say … not the best tool to monitor events. It’s designed to monitor statuses.

So it’s designed to continuously monitor as specific value – if this value raises over a defined alert-value an alert is displayed and when it falls below the value the problem disappears.
With events there is the problem that we get only one single value which describes the error. So firstly we have to analyze the received value/message and secondly – how do we know when the problem is okay again? And thats one of the design flaws of Zabbix – you do not have any possibilty to reset such events to “OK” if such an event happend.
So we need to monitor the vCenter alarms, because this alerts are raised if an problem occures and disappear if the problem changes to OK again.

So how do we get all the vCenter alarms to zabbix? I don’t want to copy/create all the alarms by hand because its a dynamic environment and alarms could be added or deleted, so the system has to “import” the alarms “on the fly” from the vCenter.
Since Zabbix 2.0 there exist discovery rules which are kind of helpful to import dynamic values. So I’m using a discovery to peridodically pull the data from the vCenter and create an item for every alarm. All the alarms in the vCenter need to be configured to run a custom alarm when an alarm becomes active which sends the current status to zabbix and voilá – we are done.

Continue reading Getting vCenter alarms to Zabbix →