Getting vCenter alarms to Zabbix

VMware is a relay nice product, but there is one little problem. It’s realy hard to monitor VMware products with SNMP or any other “old school” technologies.
The actual problem is to get an alarm in Zabbix if there occures an error on the vCenter. So Zabbix is used as an umbrella monitoring for the whole environment.
All this could also be done with SNMP-Traps what would be a lot easier – at first appereance, but Zabbix is … how do I say … not the best tool to monitor events. It’s designed to monitor statuses.

So it’s designed to continuously monitor as specific value – if this value raises over a defined alert-value an alert is displayed and when it falls below the value the problem disappears.
With events there is the problem that we get only one single value which describes the error. So firstly we have to analyze the received value/message and secondly – how do we know when the problem is okay again? And thats one of the design flaws of Zabbix – you do not have any possibilty to reset such events to “OK” if such an event happend.
So we need to monitor the vCenter alarms, because this alerts are raised if an problem occures and disappear if the problem changes to OK again.

So how do we get all the vCenter alarms to zabbix? I don’t want to copy/create all the alarms by hand because its a dynamic environment and alarms could be added or deleted, so the system has to “import” the alarms “on the fly” from the vCenter.
Since Zabbix 2.0 there exist discovery rules which are kind of helpful to import dynamic values. So I’m using a discovery to peridodically pull the data from the vCenter and create an item for every alarm. All the alarms in the vCenter need to be configured to run a custom alarm when an alarm becomes active which sends the current status to zabbix and voilá – we are done.

So the alerting will run through the follwoing steps:

vCenter Alerting Flow chart

A problem occured when sending the data directly after the discovery, because the zabbix server process syncs the items, which it is processing every 2 minutes from the database. Sof if a new alarm is triggerd, a new item is created which is note cached by the zabbix server process. But the zabbix server process has to know about the item first, before it is able to process data for this item. So the Alert-script tries to send the data multiple times (with a timeout in between) to make sure that the data are sent again when the zabbix server process should have knowledge about the items and is able to process the data. If it fails multiple times, the alert handler process will exit without sending the data to avoid being stuck in an endless loop. Otherwise – if multiple processes would be stuck in an endless loop the vCenter would eventually crash, and we do not want that 😉

To send the data to Zabbix the alert handler makes use of the zabbix agent configuration file & the zabbix sender utility, so a zabbix agent & zabbix sender utility have to be installed on the vCenter.
Another prerequisit would be that the vSphere Power CLI is installed on the system because several functions of the vSphere Power CLI are used.

The package for the vCenter monitoring is attached to this post and consits of the following files:

A readme file which contains nearly the same content as this blog post

redirect.bat
Is just a wrapper which is ran when a vCenter alarm is     trigged and calls the real alarm handler.
For debugging purpose the output could be piped to a file -> see the batch file for an example!
alarmHandler.ps1
The real alarm handler which forwards the data to Zabbix.     The alarm handler gathers the data from specific environmental variables and at first sends a discovery to     Zabbix. After the Discovery was sent to Zabbix the alarm handler tries to forward the data itself to Zabbix. If this is not successful, the alarm handler will try to send the     data for about 2 minutes. Specific settings can be provided in the alarmHandler.ps1-script. By default the variables should be sett correctly so there is no need to change them.
updateRunCommandAlarm.ps1
With this script all the alarms in the vCenter can be updated to use the alarm handler – so there is no need to add it by hand. By default the alarm is set to “repeat” so alarms are sent periodically.

So let me summarize:
To use the script the follwoing steps have to be taken:

  • make sure that vSphere Power CLI is installed
  • make sure that Zabbix Agent ist installed and configuration file is correct
  • extract the attached zip file to the vCenter and extract it to a path of your choice.
  • adopt the pathes in the batch and powershell scripts to match your path
  • run the updateRunCommandAlarm.ps1 to attach the alert handler to all alarms in your vCenter
  • Import the attached vCenter Template or create your own discovery to get the alarms to your Zabbix.

Now everything should be set up so that you can get all vcenter Alarms dynamically to your Zabbix 😉

vCenter Alerting Scripts: vCenterAlerting

vCenter Template: TMPL-NIS-Service-VMware-vCenter-WIN

7 thoughts on “Getting vCenter alarms to Zabbix”

  1. Hi,

    I’m trying to setup vcenter alerting in zabbix according to your instructions, but when I run the alarm handler it seems the file cannot be uploaded to the zabbix server and get the following:

    LastWriteTime : 25-1-2016 16:46:16
    Length : 202
    Name : VcAlarmVal_1453740376

    info from server: “processed: 0; failed: 1; total: 1; seconds spent: 0.000200”
    sent: 1; skipped: 0; total: 1

    LastWriteTime : 25-1-2016 16:46:16
    Length : 67
    Name : VcAlarmVal_1453740376

    zabbix_sender.exe [19364]: Warning: [line 1] ‘Key value’ required
    Sending failed. Use option -vv for more detailed output.

  2. Hi,

    Eventually we got the script running, but some alarms are not reset to green when the alarm disappears. Is there a possibility to do this in zabbix manually?

    1. Hi Ronald,

      ATM Zabbix doesn’t provide any possibility to reset triggers manually. It’s designed to use thresholds which also causes problems with SNMP-traps and events in general.

      I was thinking about coding a little php-script which is linked to in the trigger-URL and directly does the SQL-Update in the Database. As far as I know the SQL-Update in the DB is the only possibility to reset the trigger manually.

      But I also have to admit, that the push-based solution from the post isn’t the best implementation. We also have the problem that sometimes some alarm-states are not sent correctly to Zabbix which results in inconsistent states in Zabbix.

      I was thinking about replacing the push-base solution by a poll based solution – eg. querying active alarms via CIM. Unfortunately I dont’t have to much time for Zabbix because I’m woking on quite some other projects at the moment, but I’ve planned to redesign the vCenter-alerting solution (but that will take some time – I think around April 2016).

    2. Hi Ronald,
      I am having the same error. What did you do to get your script running? Thanks in advance

  3. I have solved the issue I reported earlier:

    here is a before and after of how to treat the type of data returned in the aray:
    #if($serverResponse.contains(“failed: 0;”))
    if($serverResponse -contains “failed: 0;” )

    NOW I have a different problem. zabbix_sender won’t send the alarm info that was created, and errors with the following:

    zabbix_sender.exe [10908]: ERROR: [line 1] ‘Key value’ required

    I think this is because the format of the text being sent is erroneous, but have yet to determine how to correct it. Please help

    1. Hi Tony,

      I know the answer is too late but it may help other readers.

      This because the file that Zabbix Sender tries to send to Zabbix is without key and value.
      The location of a file, you configured in alarmHandler as “containerFilePath”. You can try to send it manually in cmd : “C:\zabbix\zabbix_sender.exe -i E:\ZabbixVCenter\VcAlarmVal_XXXXXX -v -c C:\zabbix\conf\zabbix_agent.win.conf” (change paths accordingly), but now make sure that the file VcAlarmVal_XXXXXX is with Key and Value. This files should be created automatically if alarm triggered in vCenter.

  4. Has anyone managed to get this to work? I’m still stuck on the error “zabbix_sender.exe [20752]: Warning: [line 1] ‘Key value’ required
    Sending failed. Use option -vv for more detailed output.”

    Was this ever updated in April 2016 as the author said?

Leave a Reply

Your email address will not be published. Required fields are marked *