Category Archives: Zabbix

Zabbix Error “an array is expected” with Powershell

When trying to interact with the Zabbix-API via Powershell one might run into the problem, that an API request will not be executed successfully, but an error similar to the following one, will be displayed:

  code message         data                                                           
  ---- -------         ----                                                           
-32602 Invalid params. Invalid parameter "/1/filter/conditions": an array is expected.

Especially when using some code like in the below snippet which uses @(…) and @{…} to define data via arrays and hashtables:

function ZbxUpdateTestAction
{

    $params = @{
        body =  @{
            "jsonrpc"= "2.0"
            "method"= "action.update"
            "params"= @{
                "actionid" = "117"
                "status" = 0
                "filter" = @{
                    "evaltype" = 0
                    "conditions" = @(
                        @{
                            "conditiontype" = 4
                            "operator" = 5
                            "value"= 3
                        }
                        @{
                            "conditiontype" = 0
                            "operator" = 0
                            "value"= 435
                        }
                    )
                }
            }
            "id"= 1
            "auth" = "$api_token"
        } | ConvertTo-Json
        uri = $zbx_api_uri
        headers = @{
            "Content-Type" = "application/json"
            "Authorization" = "Bearer $api_token"
            }
        method = "Post"
    }
    $result = Invoke-WebRequest @params
    return $result.Content | ConvertFrom-Json
}

The above snippet just shows a test function to manually update an action and set conditions to send out alerts for severities of warning or higher for a specific host group.

However – when executed it will fail with the error from snippet 1.

This is because Powershell does the JSON-conversion only to a certain depth. If your structure reaches a deeper level, it will simply stop converting the dict or array which results in the above error.

Troubleshooting it, could be quite a pain in the ass as it’s not that simple to spot and when gradually extending the object it just works fine till you exceed a depth of 2.

To fix this – simple specify the depth for ConvertTo-Json

e.g.

function ZbxUpdateTestAction
{

    $params = @{
        body =  @{
            "jsonrpc"= "2.0"
            "method"= "action.update"
            "params"= @{
                "actionid" = "117"
                "status" = 0
                "filter" = @{
                    "evaltype" = 0
                    "conditions" = @(
                        @{
                            "conditiontype" = 4
                            "operator" = 5
                            "value"= 3
                        }
                        @{
                            "conditiontype" = 0
                            "operator" = 0
                            "value"= 435
                        }
                    )
                }
            }
            "id"= 1
            "auth" = "$api_token"
        } | ConvertTo-Json -Depth 5
        uri = $zbx_api_uri
        headers = @{
            "Content-Type" = "application/json"
            "Authorization" = "Bearer $api_token"
            }
        method = "Post"
    }
    $result = Invoke-WebRequest @params
    return $result.Content | ConvertFrom-Json
}

Compare line 29 between the two snippets

Refs:

(35) error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small

When using Zabbix on a Centos8/RHEL8 machine the following error occurred whil trying to monitor an HTTPS-website via the build in web scenarieos:

(35) error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small 

The error itself also shows up when trying to use curl to connect to the website:

$ curl -D - https://<some-legacy-website-> -k
curl: (35) error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small

That error occurs if the server uses an older cipher-suite that’s considered unsafe by the default crypto policy used in Centos8/RHEL8.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/using-the-system-wide-cryptographic-policies_security-hardening

To work around that problem, the legacy cipher suites must be enabled by:

# update-crypto-policies --set LEGACY

Although a restart is recommended after issuing the command, for me it also worked without the need of issuing a reboot.

Monitoring Tasmota with Zabbix

The repository contains a python-script which can be used to subscribe to an MQTT-server (e.q. mosquitto) and forward all MQTT-message to Zabbix. There the Template can be used to monitore the values sent via MQTT. Currently the Template is mainly intended as an example and requires adaption to your current setup/configuration of tasmote.

https://gitlab.com/fawcs/zabbix_tasmota_mqtt

Zabbix SELinux policy generation

Commands to query the auditlog for Zabbix relevant queries and create/import a compiled policy file within Zabbix

Could be adapted to generate policies for any other system.

The suggestion is to set SELinux to permissive (setenforce=0) execute the action and afterwards create the policy based on the logged events. If the policy does not work on the first try after re-enabeling SELinux again it it could happen that a call was blocked (which is also logged within the auditlog) that was not blocked with SELinux in permissive mode. Therefore it could help creating a new human readable policy (.te-file) and checking the first version vs. the second version + merging them. 

filename=zabbix-server
cat /var/log/audit/audit.log  | grep zabbix | audit2allow -m $filename >> $filename.te
checkmodule -M -m -o $filename.mod $filename.te
semodule_package -o $filename.pp -m $filename.mod
semodule -i $filename.pp
 
 
#restorecon -R -v /run/zabbix/zabbix_server_alerter.sock    #suggested by the policygenerator

vCenter alarm Polling (v2) – Cross platform version (Windows & Appliance)

As it has been some time since my last post about a solution on how to get vCenter alarms to Zabbix, and VMware also evolved I followed a new approach on that topic as my initial post only supports Windows vCenters. Furthermore the solution is not as stable as I wished that it would be, so my new approach is to query all alarms from a vCenter via it’s SDK.
Initially all the alarms are discovered and created in Zabbix and in a second step the values for the discovered alarms are polled.
Currently the script used the data center object of the vCenter to discover alarms, so it can’t be used on a standalone ESXi-Server. However – if the code is changed to use whatever object is needed to get the alarms directly from the ESXi-server it should also be possible to get alarms directly from a server without the need of a vCenter (but I didn’t implement that till now as there wasn’t the need/time).

vCenter alarms – SDK (tested with ESXi 6.0+ and Zabbix 3.0 on RHEL 7)

To install the vCenter alarms the attached zip needs to be downloaded and the VMware Perl SDK must be installed on the Zabbix Server.
The template needs to be imported into Zabbix and the vCenter username and password need to be set in the username/password macros of the template.

The other two files (vcenterAlarms.pl & vcenterAlarms.wrapper) need to be extracted to the externalscripts folder of the Zabbix Server. The wrapper script is just a shell script that is executed by a Zabbix item to call the per script and send the that to Zabbix via Zabbix Sender.  As the VMware API is quite slow the wrapper also starts itself again with NOHUP because otherwise the timeout defined in the Zabbix Server configuration would cause an exit of the script. For my setup it always took longer than 30 Seconds till tall data where gathered and therefor the Zabbix Server would kill the script in the middle of the execution and no data would be sent to Zabbix. That’s why I added this workaround. Furthermore it also checks if there are less than eleven vcenterAlarms.wrapper processes running, and only starts if there are less, to ensure that Zabbix does not spawn hundreds of NOHUP-processes.

 

 

 

 

Zabbix – Clear hosts from untemplated items

Sometimes you run into the problem, that you have a host which had a template attached but somebody wanted to replace the template or something like that and unfortunately hit just “Unlink” instead of “Unlink and Clear” and all the items are still in the host.

If you have only one host it’s normaly no problem to delete all items per hand, but if you have multiple of those hosts it’s quite some work do remove the old items.

Solution nr. one would be tu use the filters to select all items in a specified host group and delte those items, but the applications, discovery rules and so on will still remain in the hosts and have to be deleted in a 2nd/3rd step.

See the Screenshot below:Zabbix Host configuration - item filters

 

My preferred solution for this problem is a simple regex based find/replace with Notepad++.
Herefor an export of the affected hosts is needed. The xml-file could be opend with NPP and the following regexes are needed for find/replace (CTRL+H) to remove the unwanted items.

[pastacode lang=”markdown” manual=”Find%20what%3A%20(%3Cdiscovery_rules%3E%5B%5Cs%5CS%5D*%3F%3C%5C%2Fdiscovery_rules%3E)%7C(%3Ctriggers%3E%5B%5Cs%5CS%5D*%3F%3C%5C%2Ftriggers%3E)%7C(%3Cinventory%3E%5B%5Cs%5CS%5D*%3F%3C%5C%2Finventory%3E)%7C(%3Citems%3E%5B%5Cs%5CS%5D*%3F%3C%5C%2Fitems%3E)%0A%0AReplace%20with%3A%20(%3F1%3Cdiscovery_rules%20%2F%3E)(%3F2%3Ctriggers%20%2F%3E)(%3F3%3Cinventory%20%2F%3E)(%3F4%3Citems%20%2F%3E)” message=”” highlight=”” provider=”manual”/]

In the above example multiple regexes with multiple replace-patterns are used to replace the items, discovery rules, triggers an inventory and reset it.

Zabbix 1.8 to 2.2 Upgrade

Lately I was asked to help to upgrade Zabbix from 1.8 to 2.2 in a project. It wasn’t a problem to upgrade the templates – that was easily done with a xml-export/import but the hosts where kind of a challenge because the exported xml-files for the hosts itself pretty differs between 1.8 and 2.2.

Because i already had the PhpZabbixApi (https://github.com/confirm/PhpZabbixApi/blob/master/README.md) installed on the tared system i decided to write a little script which pareses the 1.8-host export and creates the hosts in 2.2. The script inc. the lib is attached at the end of the post.

I tested the script with Zabbix 1.8.6->2.2.10 and everything worked fine. Currently the script is capable of creating the hosts (with Zabbix-agent & SNMP-interface), creating the host groups and adding the hosts to the correct host group and also linking the correct templates to the host. However, the templates need to be already available on the target system to be linked correctly.

After extracting the script on the target Zabbix server the xml-import from the old system needs to be uploaded into the same directory as the script (scp) and the login data for Zabbix need to be adapted in the script. Afterwards the import can be started from a bash via:

[pastacode lang=”bash” manual=”” message=”” highlight=”” provider=”manual”/]

Zabbix1.8_2.2_upgrade

 

Get vCenter alarms into Zabbix via poll-method

Some time ago i wrote a post on how to forward vCenter alarms to Zabbix ( https://blog.fawcs.info/2015/05/getting-vcenter-alarms-to-zabbix/) and I have to admit, that this solutions is kind of a pain in the ass. I’m getting the alarm info from environmental varaibles which are automatically set by the vCenter when an alarm changes its status, but it seems that there is a “littel” problem with “overlapping” alarms. For example if there are occuring multiple alarms within a short period only the first alarm will be forwarded to zabbix, but non of the follwoing alarms. Besides that this is not an ideal solution I personally do not like my former approach because it’s an event driven approach. So if one event goes missing we have an inconsistent system :/

It’s quite some time since I wanted to redesign the solution and now I’m finally having some time ( and the pressure) to do so. 🙂
The new approach is based on using userparameters to execute a powershellscript on the vCenter to discover all active alarms and create items in Zabbix. At the moment I’m creating three item prototyes. One for the Timestamp when the alarm became active, another item for the acknowledged-state of the alarm and the last one for the severity of the alarm.

There are two userparemeters which run two powershell scripts. The first one (vcenter.alarm.polling.discovery.ps1) does the discovery and the second one (vcenter.alarm.polling.itemdata.ps1) is to get the data for the discoverd items.
There are also three triggers (one for each severity gray, yellow, red) which will be active als long as the alarm is not acknowledged.

You can download the scripts, userparameters and the template down below:
vCenterAlarmPolling

 

Additional findings:
Ther can occure problems if there are different addresses used to connect to the vcenter (eg. 127.0.0.1, loclahost, vcenterhostname, …)
It seems that the vCenter creates a sperate datacenter instance for every connection, so if you use the three examples from abovve you will end up creating three instances and mess up the script.

 

If special characters want to be passed to the powershellscript (e.g. special chars in passwords ord login with administrator@vsphere.local) the “UnsafeUserParameters=1” – parameter from the zabbix-agent.conf needs to be set to 1. (default value is 0)

Install Zabbix on Raspberry PI 2

Wouldn’t it be cool to monitor your home? For example all your devices, but also temperature and other sensors an have all that data accessible via a web interfaces?

I think it would so, i thought about setting up Zabbix for home monitoring, but on the RaPi B and B+ it’s not the most performant setup, So i decided to try it again with the PI2.

This post provides a short log on how I set it up.

At first we have to download the source from Zabbix’ SF-page because there is no official package for the ARM-architecture available.

[pastacode lang=”bash” message=”zabbix installation” highlight=”” provider=”manual”]

cd /opt
wget http://downloads.sourceforge.net/project/zabbix/ZABBIX%20Latest%20Stable/2.4.6/zabbix-2.4.6.tar.gz?r=http%3A%2F%2Fwww.zabbix.com%2Fdownload.php&ts=1441447329&use_mirror=skylink^C
mv zabbix-2.4.6.tar.gz\?r\=http\:%2F%2Fwww.zabbix.com%2Fdownload.php zabbix-2.4.6.tar.gz
tar xfvz zabbix-2.4.6.tar.gz
cd zabbix-2.4.6/


#With ./configure --help we can see all the availalbe switches which can be used to compile zabbix.
root@raspberrypi /opt/zabbix-2.4.6 # groupadd zabbix
root@raspberrypi /opt/zabbix-2.4.6 # useradd -g zabbix zabbix
root@raspberrypi /opt/zabbix-2.4.6 # ./configure --help


#I used the follwoing switches to compile the zabbix server and agent, use a MySQL-DB, enable jabber-support, lib-xml2 - which is needed for webmonitoring, net-snmp, ssh and curl which is alos needed for webmonitoring. IPMI can be useful if you also hav a realy server with a BMC to monitor. But for most homeusers the IPMI-option is not neede if you only want to monitor your home and thats it. If you have a LDAP/AD-environment where you want to integrate zabbix you also should use the ldap-switch, but I think most home users also do not have a directory service running at home. ;)

#If this command is run there will ocure some erroes in most cases because there are missing dependencies

./configure --enable-server --enable-agent --with-mysql --with-jabber --with-libxml2 --with-net-snmp --with-ssh2 --with-libcurl


apt-get install apache2 php5-mysql mysql-server mysql-common mysql-utilities libiksemel-dev libiksemel-utils libxml2-dev libxml2-utils libxml2 snmp libsnmp-dev libsnmp-perl libssh2-1-dev libssh2-1 libcurl3 libghc-curl-dev libmysql++-dev php5-gd

#now all dependencies should be resolved
./configure --enable-server --enable-agent --with-mysql --with-jabber --with-libxml2 --with-net-snmp --with-ssh2 --with-libcurl

#copy init scripts
cp /opt/zabbix-2.4.6/misc/init.d/debian/* /etc/init.d/
#copy webfrontend
cp -r /opt/zabbix-2.4.6/frontends/php/* /var/www/zabbix/
chown -R www-data:www-data /var/www/zabbix/


#create the database
#at first log in to your mysql-server as a root useradd and runn the following commands
mysql -uroot -p
create database zabbix character set utf8 collate utf8_bin;
CREATE USER 'zabbix'@'localhost' IDENTIFIED BY 'zabbix';
GRANT ALL PRIVILEGES ON zabbix.* TO 'zabbix'@'%' WITH GRANT OPTION;
mysql -uzabbix -pzabbix zabbix < /opt/zabbix-2.4.6/database/mysql/schema.sql
mysql -uzabbix -pzabbix zabbix < /opt/zabbix-2.4.6/database/mysql/images.sql
mysql -uzabbix -pzabbix zabbix < /opt/zabbix-2.4.6/database/mysql/data.sql

#adapt configuration files at /usr/local/etc/ like in the attached examples
#create dircetories for logfiles:
mkdir -p /var/log/zabbix
chown -R zabbix:zabbix /var/log/zabbix/

#create dirs for alert & external scripts 
mkdir -p /var/zabbix/alertscripts
mkdir -p /var/zabbix/externalscripts
chown -R zabbix:zabbix /var/zabbix/


#configure php-settings
vim /etc/php5/apache2filter/php.init
post_max_size = 16M
max_execution_time = 300
max_input_time = 300
#select timezone from http://php.net/manual/en/timezones.php and set:
date.timezone = 

#restart/reload webserver to accept changes
service apache2 restart
service zabbix-server restart
service zabbix-agent restart

#open http:///zabbix in browser and finish installation



[/pastacode]

 

zabbix-conf

Getting vCenter alarms to Zabbix

VMware is a relay nice product, but there is one little problem. It’s realy hard to monitor VMware products with SNMP or any other “old school” technologies.
The actual problem is to get an alarm in Zabbix if there occures an error on the vCenter. So Zabbix is used as an umbrella monitoring for the whole environment.
All this could also be done with SNMP-Traps what would be a lot easier – at first appereance, but Zabbix is … how do I say … not the best tool to monitor events. It’s designed to monitor statuses.

So it’s designed to continuously monitor as specific value – if this value raises over a defined alert-value an alert is displayed and when it falls below the value the problem disappears.
With events there is the problem that we get only one single value which describes the error. So firstly we have to analyze the received value/message and secondly – how do we know when the problem is okay again? And thats one of the design flaws of Zabbix – you do not have any possibilty to reset such events to “OK” if such an event happend.
So we need to monitor the vCenter alarms, because this alerts are raised if an problem occures and disappear if the problem changes to OK again.

So how do we get all the vCenter alarms to zabbix? I don’t want to copy/create all the alarms by hand because its a dynamic environment and alarms could be added or deleted, so the system has to “import” the alarms “on the fly” from the vCenter.
Since Zabbix 2.0 there exist discovery rules which are kind of helpful to import dynamic values. So I’m using a discovery to peridodically pull the data from the vCenter and create an item for every alarm. All the alarms in the vCenter need to be configured to run a custom alarm when an alarm becomes active which sends the current status to zabbix and voilá – we are done.

Continue reading Getting vCenter alarms to Zabbix