Tag Archives: monitoring

Zabbix Error “an array is expected” with Powershell

When trying to interact with the Zabbix-API via Powershell one might run into the problem, that an API request will not be executed successfully, but an error similar to the following one, will be displayed:

  code message         data                                                           
  ---- -------         ----                                                           
-32602 Invalid params. Invalid parameter "/1/filter/conditions": an array is expected.

Especially when using some code like in the below snippet which uses @(…) and @{…} to define data via arrays and hashtables:

function ZbxUpdateTestAction
{

    $params = @{
        body =  @{
            "jsonrpc"= "2.0"
            "method"= "action.update"
            "params"= @{
                "actionid" = "117"
                "status" = 0
                "filter" = @{
                    "evaltype" = 0
                    "conditions" = @(
                        @{
                            "conditiontype" = 4
                            "operator" = 5
                            "value"= 3
                        }
                        @{
                            "conditiontype" = 0
                            "operator" = 0
                            "value"= 435
                        }
                    )
                }
            }
            "id"= 1
            "auth" = "$api_token"
        } | ConvertTo-Json
        uri = $zbx_api_uri
        headers = @{
            "Content-Type" = "application/json"
            "Authorization" = "Bearer $api_token"
            }
        method = "Post"
    }
    $result = Invoke-WebRequest @params
    return $result.Content | ConvertFrom-Json
}

The above snippet just shows a test function to manually update an action and set conditions to send out alerts for severities of warning or higher for a specific host group.

However – when executed it will fail with the error from snippet 1.

This is because Powershell does the JSON-conversion only to a certain depth. If your structure reaches a deeper level, it will simply stop converting the dict or array which results in the above error.

Troubleshooting it, could be quite a pain in the ass as it’s not that simple to spot and when gradually extending the object it just works fine till you exceed a depth of 2.

To fix this – simple specify the depth for ConvertTo-Json

e.g.

function ZbxUpdateTestAction
{

    $params = @{
        body =  @{
            "jsonrpc"= "2.0"
            "method"= "action.update"
            "params"= @{
                "actionid" = "117"
                "status" = 0
                "filter" = @{
                    "evaltype" = 0
                    "conditions" = @(
                        @{
                            "conditiontype" = 4
                            "operator" = 5
                            "value"= 3
                        }
                        @{
                            "conditiontype" = 0
                            "operator" = 0
                            "value"= 435
                        }
                    )
                }
            }
            "id"= 1
            "auth" = "$api_token"
        } | ConvertTo-Json -Depth 5
        uri = $zbx_api_uri
        headers = @{
            "Content-Type" = "application/json"
            "Authorization" = "Bearer $api_token"
            }
        method = "Post"
    }
    $result = Invoke-WebRequest @params
    return $result.Content | ConvertFrom-Json
}

Compare line 29 between the two snippets

Refs:

Powershell/PowerCLI very slow execution Time

Sometimes a PowerCLI-script can take quite some time till everything is executed. For example the PowerShell scripts used by Zabbix to gather the vCenter alarms into Zabbix (BlogPost) need some tuning to run fine.

So why are scripts running slow in some cases?
It seems to occur primary on systems which do not have a connection to the internet. As a matter of fact – most of the systems I’m setting up “lose” internet connection sooner, or later. :/

What exactly causes the problem?
While investigating that problem i found an interesting feature which seems to cause the problem – certificate checks!
There is an IE-setting which is named “Check for publisher’s certificate revocation ” and can be found at: Intenet Options -> Advanced -> Section: Security ->Disable: Check for publisher’s certificate revocation.

Disabling the certificate checks improves the execution time by about 60%.

certificate-check enabled:

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 41
Milliseconds      : 536
Ticks             : 415369143
TotalDays         : 0.000480751322916667
TotalHours        : 0.01153803175
TotalMinutes      : 0.692281905
TotalSeconds      : 41.5369143
TotalMilliseconds : 41536.9143

 

certificate-check disabled:

Days : 0
Hours : 0
Minutes : 0
Seconds : 16
Milliseconds : 262
Ticks : 162628208
TotalDays : 0.000188227092592593
TotalHours : 0.00451745022222222
TotalMinutes : 0.271047013333333
TotalSeconds : 16.2628208
TotalMilliseconds : 16262.8208

 

If the script is run from a normal user account everything should be fine and we have an improved execution time, BUT …

… if the script is run from an Service (and as a matter of fact I’m using the Zabbix agent service to run the script) we got a problem.
With default settings the Zabbix Agent is installed to run as nt authority\system, so if the IE-setting is changed for the current user, its working for this user, but not for the system user. 🙁
So a quick and dirty workaround could be to disable the setting for the system user.
ATTENTION: Running the Zabbix Agent as a system user is OK for a DEV-environment, but should not be used in an production environment. For production a dedicated service user should be used.

I disabled it by becoming a system user with

[pastacode lang=”bash” message=”” highlight=”” provider=”manual”]

PSEXEC -i -s -d CMD

[/pastacode]

and launching the IE from the command prompt. Afterwards I was able to disable the setting via the above method.

Otherwise the Key could also be found in the registry at:

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing\State
0x00023e00 / 146944 Check OFF
0x00023c00 / 146432 Check ON

A simple PS-Script to disable the setting would be:

[pastacode lang=”bash” message=”PowerShell to disable Publisher certificate checks” highlight=”” provider=”manual”]

Write-Host "Disable Check for publisher’s certificate Revocation"
set-ItemProperty -path "REGISTRY::\HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing\" -name State -value 146944

[/pastacode]

 

Getting vCenter alarms to Zabbix

VMware is a relay nice product, but there is one little problem. It’s realy hard to monitor VMware products with SNMP or any other “old school” technologies.
The actual problem is to get an alarm in Zabbix if there occures an error on the vCenter. So Zabbix is used as an umbrella monitoring for the whole environment.
All this could also be done with SNMP-Traps what would be a lot easier – at first appereance, but Zabbix is … how do I say … not the best tool to monitor events. It’s designed to monitor statuses.

So it’s designed to continuously monitor as specific value – if this value raises over a defined alert-value an alert is displayed and when it falls below the value the problem disappears.
With events there is the problem that we get only one single value which describes the error. So firstly we have to analyze the received value/message and secondly – how do we know when the problem is okay again? And thats one of the design flaws of Zabbix – you do not have any possibilty to reset such events to “OK” if such an event happend.
So we need to monitor the vCenter alarms, because this alerts are raised if an problem occures and disappear if the problem changes to OK again.

So how do we get all the vCenter alarms to zabbix? I don’t want to copy/create all the alarms by hand because its a dynamic environment and alarms could be added or deleted, so the system has to “import” the alarms “on the fly” from the vCenter.
Since Zabbix 2.0 there exist discovery rules which are kind of helpful to import dynamic values. So I’m using a discovery to peridodically pull the data from the vCenter and create an item for every alarm. All the alarms in the vCenter need to be configured to run a custom alarm when an alarm becomes active which sends the current status to zabbix and voilá – we are done.

Continue reading Getting vCenter alarms to Zabbix

Get raid information from a Fujitsu RX200S8/Rx300S8 via BMC

With the S8 generation of Fujtitus RX300/200 servers, which use the iRMC S4 ,Fujitsu implementent the ability to poll SNMP-data from the BMC (iRMC). To enable the ability to poll data the BMC has to be flashed with an up to date Firmware version and SNMP-polling has to be enableed in the iRMC-web-GUI. I’ve tested it with 7.69F from Dec. 2014 .
The first shippings of the rx300/200 servers came with an older firmeware version which did not implment the needed Fujitsuu MIBs to query HW-status (see http://manuals.ts.fujitsu.com/file/11470/irmc-s4-ug-en.pdf Page: 18 for details about the supported MIBs).
The problem is, that there is no possibility to query the status of the Raid controller and its disks via SNMP (or I haven’t found it till now) but it’s displayed on the iRMC web-GUI. So I wrote a script which extracts the useful informations (controller status, disk status + details and logical drive status) from the web-interface.

Atm. it’s just an alpha release but I’ll modify the script to be used by Zabbix for an auto discovery and push all the data into Zabbix:

Source:

[pastacode lang=”php” message=”web-iRMCS RAID-query” highlight=”” provider=”manual”]

#!/usr/bin/php
<?php
/****************************************
################################################################
#   DESCRIPTION: Script to query RAID-Informations from the Fujitsu iRMC S4 webinterface.
#   COPYRIGHT ©: 2015 by fawcs, GPL freeware
#        AUTHOR: fawcs, weixeflo@fawcs.info
#       LICENSE: GPL freeware
# CREATION DATE: 2015-Mar-19
################################################################
*****************************************/
$username="admin";
$password="admin";
$host="127.0.0.1";

$siteIDs=array('controller'=>87,'disks'=>88,'logicalVolumes'=>89);
//get Raid-Controller-Data
//$returnController=getWebsiteContent($host."/".$siteIDs['controller'],$username,$password,true);
$returnDisks=getWebsiteContent($host."/".$siteIDs['disks'],$username,$password,true);
//$returnVolumes=getWebsiteContent($host."/".$siteIDs['logicalVolumes'],$username,$password,true);
//$id['controller']=parseWebSiteToIDs($returnController);
$id['disks']=parseWebSiteToIDs($returnDisks);
//$id['logicalVolumes']=parseWebSiteToIDs($returnVolumes);
//print_r($id);


$ctrl=$id['disks'][0]['id'][1];
foreach($id['disks'][0]['data'] AS $disk)
{
	//print_r($disk);
	$diskWebContent=getWebsiteContent($host."/pdrive?ctrl=".$ctrl."&pd=".$disk['detailID'][1],$username,$password,true);
	//print_r($diskWebContent);
	$diskTableArray=parseWebSiteToIDs($diskWebContent);
	print_r($diskTableArray);	
}
/**/

/**
retrieves an html-site
	
	@return: String -> whole HTML site
**/
function getWebsiteContent($url,$username,$password,$measuretime=false)
{
	if($measuretime===true) 
	{
		$timer=microtime(true);
	}
	$process = curl_init($url);
	curl_setopt($process, CURLOPT_USERPWD, $username . ":" . $password);  
	curl_setopt($process, CURLOPT_HTTPHEADER, array('Content-Type: application/xml'));
	curl_setopt($process, CURLOPT_HEADER, 1);
	curl_setopt($process, CURLOPT_USERPWD, $username . ":" . $password);
	curl_setopt($process, CURLOPT_TIMEOUT, 30);
	curl_setopt($process, CURLOPT_POST, 1);
	//curl_setopt($process, CURLOPT_POSTFIELDS, $payloadName);
	curl_setopt($process, CURLOPT_RETURNTRANSFER, TRUE);
	$return = curl_exec($process);
	curl_close($process);
	if($measuretime===true) 
	{
		echo "Content for ".$url." retrieved in ".round(microtime(true)-$timer,3)."s".chr(10);
	}
	return $return;
}

/**
retrieves the table-content of an IRMC Site / parses the listed sites for interesting data
possible sites are:
	http://$host/87 -> Controller Infos
	http://$host/88 -> Physical Disks
	http://$host/89 -> Logical Drives
	
	@return -> 	array which contains the ID of the controller, disks, LVs
				the table headings
				the table data
**/


function parseWebSiteToIDs($return)
{
	$ret=array(); 
	$cutFrom=strpos($return,"<!-- Menu end-->");
	$cutTo=strpos($return,'<div id="bottom">');
	$return=str_replace('</td>','</td>'.chr(10),substr($return,$cutFrom,$cutTo-$cutFrom));
	$return=str_replace('</th>','</th>'.chr(10),$return);
	$Loop=$return;
	
	//echo $table;
	$count=0;
	//go through all tatles which are found in the passed web-content-string & extract the tables 
	while(strpos($Loop,'<table')!==false)
	{
		
		//$table[]=substr($return, strpos($return,"<table"),strpos($return,"</table>")-strpos($return,"<table>"));
		$Loop=substr($Loop, strpos($Loop,'<table'),strlen($Loop)-strpos($Loop,'<table'));
		//get the table id
		$id=substr($Loop,strpos($Loop,"summary=\"")+9,strpos($Loop,"\">")-(strpos($Loop,"summary=\"")+9));
		$ret[$count]['id']=explode("_",$id);
		$Loop=substr($Loop, strpos($Loop,'<tr'),strlen($Loop)-strpos($Loop,'<tr'));
		//save just the html-table to the var & add ***###*** as a delimitter after every tablerow
		$currentTable=str_replace("</tr>","</tr>***###***",substr($Loop,0,strpos($Loop,"</table>")));
		$currentTable=substr($currentTable,0,strlen($currentTable)-9);
		//explode all table rows by the added delimiter
		$rows=explode("***###***",$currentTable);
		$count2=0;
		//iterate through the table rows and extract useful information
		foreach($rows AS $row)
		{
			unset($detailID);
			//check if submit button for details is listed in the table - if yes - get the submit-value for correct id -> is appended to the URL when querying the details page
			if(strpos($row,'<input class="submit" type="submit" value="Details"')!==false)
			{
				$detailID=trim(substr($row,strpos($row,'<input class="submit" type="submit" value="Details"')+strlen('<input class="submit" type="submit" value="Details"'),strpos(strtolower($row),'onclick=')-(strpos($row,'<input class="submit" type="submit" value="Details"')+strlen('<input class="submit" type="submit" value="Details"'))));
				//eg: extracted name="pd_10" from row
				$detailID=str_replace("\"","",$detailID);
				$detailID=explode("=",$detailID);
				$detailID=$detailID[1];
			}
			$row=trim(strip_tags($row));
			//if $count==0 -> first table row which stores only the headings/descriptions -> are saved in a seperate node in the array
			if($count2==0)
			{
				$ret[$count]["description"]=explode(chr(10),$row);
			} else {
			//all the other rows contain data -> also store the in the array
				$rowArray=explode(chr(10),$row);
				$ret[$count]["data"][]=$rowArray;
				if(isset($detailID))
				{
					$ret[$count]["data"][$rowArray[0]]["detailID"]=explode("_",$detailID);
				}
			}
			$count2++;
		}	
		$count++;
	}
	return $ret;
}



?>

[/pastacode]

Windows performance counters and zabbix

There already quite a lot of blog posts out there which describe how to add performance counters to zabbix. In fact – its not that hard – the tricky thing is, to have one template which gets performance counters from systems wich have different languages installed.

In this case it’s a good idea to use the index of a counter. Indexes of all performance counters can be obtained from teh registry.
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ Windows NT\CurrentVersion\Perflib
In this path there should be a key named “009” (would be englisch, “007” german) or similar, which contains all the key-value pairs.
The easiest way is to copy everything to an editor and look up the keys you need.

I tried to verify the counter ids by using them with “typeperf” – eg: typeperf \234(_Total)\202 but most times the result was:
Error: No valid counters.
If i used the names instead of the keys everything worked fine.

Atm I still don’t know what the reason for this error is, but, if you try it with  zabbix_get everything works fine.
zabbix_get -s server-01 -k “perf_counter[\234(_Total)\202]”
0.000000

Further details about performance counters in Zabbix can be found at:
https://www.packtpub.com/books/content/monitoring-windows-zabbix-18