When running infrastructure in production, it is best practice to setup monitoring to identify and resolve any issues proactively before they arise. With infrastructure, such as the SwiftStack Object Storage System, which is able to withstand component failures including disk drives, networks, storage nodes, or even entire data centers, it is not the end of the world if something fails or becomes unresponsive. Nonetheless, it is still important to monitor your SwiftStack storage environment.
The SwiftStack Object Storage System is built using OpenStack Swift, an open source object storage engine, at its core. SwiftStack software is installed on standard Linux distributions (RedHat, CentOS, and Ubuntu) and can be monitored like regular Linux servers. However, most monitoring solutions don't currently include hooks for monitoring Swift-based object storage by default.
As part of the SwiftStack Object Storage System, metrics including CPU load, utilization and memory consumption, disk I/O for the storage nodes, as well as overall system status and utilization can be monitored from the SwiftStack Controller. However, most of our customers have existing monitoring infrastructure in place and prefer their SwiftStack Object Storage to plug into their monitoring workflows.
To integrate with existing monitoring infrastructure, SwiftStack provides instructions for third party tools such as Nagios. Recently we've come across a lot of new customers that use Zabbix, a monitoring solution for the enterprise, and they have requested help with integration.
In this blog, we'll show you how to:
- Install Zabbix Server (v2.2) on Ubuntu Precise 12.04
- Import a SwiftStack monitor template for SwiftStack Nodes
- Install Zabbix Agent on a SwiftStack Node
- Add a SwiftStack Node to a Zabbix Server
- Apply the SwiftStack template to SwiftStack Node for monitoring
Note: Zabbix announced a newer version 2.4, but setup is no different than version 2.2.
Install Zabbix Server on Ubuntu 12.04 LTS:
$ wget http://repo.zabbix.com/zabbix/2.2/ubuntu/pool/main/z/zabbix-release/zabbix-release_2.2-1+precise_all.deb $ dpkg -i zabbix-release_2.2-1+precise_all.deb $ apt-get update
Install Zabbix packages and configure the database
If you're planning to monitor your SwiftStack environment using Zabbix, you probably already have a Zabbix Server set up. But, for sake of completeness we'll also show how to install and set up a Zabbix Server. While installing Zabbix it will also include MySQL. So, you'll need to enter an admin password for the database.
$ apt-get install zabbix-server-mysql zabbix-frontend-php
Modify php configuration and restart apache2
Modify date.timezone to your timezone in
/etc/apache2/conf.d/zabbix. For this example we used
php_value max_execution_time 300 php_value memory_limit 128M php_value post_max_size 16M php_value upload_max_filesize 2M php_value max_input_time 300 php_value date.timezone America/Los_Angeles
Restart Apache server:
$ service apache2 restart * Restarting web server apache2 ... waiting
After restarting Apache server, the Zabbix Server should be running on your machine.
Log into your Zabbix Server
Navigate to the Zabbix Server you just created:
Log in with the default credentials:
- Username: Admin
- Password: zabbix
Step through the Welcome screen. Before continuing, the system will verify the pre-requisites have been met.
Next, configure the MySQL database connection by filling out the database host, port, name, username, and password fields. Select Test Connection to verify a successful connection can be made to the MySQL database. Upon success, select Next to continue.
In this step, fill out the Zabbix server details and select Next to continue.
Review and verify the the Pre-Installation summary looks correct. Select Next to continue.
Once the install process has completed, select Finish to complete the setup.
After you click on the Finish button, you'll be taken to the login screen. Again, to log in, the default username is Admin and the password iszabbix.
Once you're logged in, your Zabbix dashboard will look similar to the following:
Import the SwiftStack Zabbix template
The first thing we'll need to do is to create a SwiftStack host group. Under Configuration > Host groups you can create a new host group. Because the
ss_template_zabbix.xml file is using the group naming convention 'SwiftStack', you should call your new host groupSwiftStack. After you've created the SwiftStack group, it's time to import the monitoring template.
Before setting up a Zabbix agent to monitor a SwiftStack Node, we need to import the SwiftStack monitoring template.
You can download the SwiftStack Zabbix template here. or directly on the command line using
Once you have downloaded the monitoring template, selectConfiguration > Templates > Import.
From the Import screen, select Choose File to and select the SwiftStack monitoring template you previously downloaded.
You will see an Imported successfully message if everything worked correctly.
Install Zabbix agent on a SwiftStack node
At this point, you should have a Zabbix server with the SwiftStack monitoring template imported.
The next step is to install the Zabbix agent on your SwiftStack Node. To do so, log in to your SwiftStack Node (via ssh) to install the required Zabbix packages. Note: make sure you have root permission.
` root@charz-test-swift:~# wget http://repo.zabbix.com/zabbix/2.2/ubuntu/pool/main/z/zabbix/zabbix-agent2.2.8-1+preciseamd64.deb root@charz-test-swift:~# dpkg -i zabbix-agent2.2.8-1+preciseamd64.deb Selecting previously unselected package zabbix-agent. (Reading database ... 128744 files and directories currently installed.) Unpacking zabbix-agent (from zabbix-agent2.2.8-1+preciseamd64.deb) ... Setting up zabbix-agent (1:2.2.8-1+precise) ...
Creating config file /etc/zabbix/zabbixagentd.conf with new version * Starting Zabbix agent zabbixagentd [ OK ] Processing triggers for ureadahead ... Processing triggers for man-db ...
Zabbix user permissions
Grant passwordless sudo permission to the
zabbix user on the SwiftStack node:
$ sudo visudo
Add the following lines:
Cmnd_Alias ZABBIX_CMD = /usr/bin/check_swift_sweep_time, /usr/bin/check_swift_background_daemon zabbix ALL = (other_user) NOPASSWD: ALL zabbix ALL = (root) NOPASSWD: ZABBIX_CMD
Modify the Zabbix agent configuration file
Modify the content of zabbix_agentd.conf (
/etc/zabbix/zabbix_agentd.conf ) and make sure the hostname is the same as defined on the Zabbix server:
` Server=<ZabbixServerIP> ServerActive=<ZabbixServerIP> Hostname=<SwiftStackNodehostname>
Now, restart the Zabbix agent service:
$ /etc/init.d/zabbix-agent restart
Check the Zabbix agent log
In this step, you'll notice the agent returns an error message
host [charz-test-swift] not found.
root@charz-test-swift:~# sudo tailf /var/log/zabbix/zabbix_agentd.log 8529:20150312:023635.743 Got signal [signal:15(SIGTERM),sender_pid:16171,sender_uid:0,reason:0]. Exiting ... 8529:20150312:023635.747 Zabbix Agent stopped. Zabbix 2.2.8 (revision 51174). 16185:20150312:023635.810 Starting Zabbix Agent [charz-test-swift]. Zabbix 2.2.8 (revision 51174). 16185:20150312:023635.810 using configuration file: /etc/zabbix/zabbix_agentd.conf 16186:20150312:023635.811 agent #0 started [collector] 16187:20150312:023635.811 agent #1 started [listener #1] 16188:20150312:023635.812 agent #2 started [listener #2] 16189:20150312:023635.812 agent #3 started [listener #3] 16190:20150312:023635.812 agent #4 started [active checks #1] 16190:20150312:023635.831 no active checks on server [192.168.201.152:10051]: host [charz-test-swift] not found
The error message exists because there is currently no host matching the host entry in the Zabbix server. Let's fix that by adding the Node as a host in the Zabbix server configuration and restart the Zabbix agent.
Create a host to monitor
On the Zabbix server, navigate to: Configuration > Hosts.
Fill in the SwiftStack Node host information including:
- Host name
- Agent Interface - IP Address
Restart Zabbix agent on Swift node
root@charz-test-swift:~/etc/init.d/zabbix-agent restartg * Stopping Zabbix agent zabbix_agentd [ OK ] * Starting Zabbix agent zabbix_agentd [ OK ]
Check the Zabbix agent log again
root@charz-test-swift:~# sudo tailf /var/log/zabbix/zabbix_agentd.log 16190:20150312:023635.831 no active checks on server [192.168.201.152:10051]: host [charz-test-swift] not found 16185:20150312:024040.421 Got signal [signal:15(SIGTERM),sender_pid:19615,sender_uid:0,reason:0]. Exiting ... 16185:20150312:024040.444 Zabbix Agent stopped. Zabbix 2.2.8 (revision 51174). 19631:20150312:024040.505 Starting Zabbix Agent [charz-test-swift]. Zabbix 2.2.8 (revision 51174). 19631:20150312:024040.505 using configuration file: /etc/zabbix/zabbix_agentd.conf 19632:20150312:024040.506 agent #0 started [collector] 19633:20150312:024040.506 agent #1 started [listener #1] 19634:20150312:024040.507 agent #2 started [listener #2] 19635:20150312:024040.507 agent #3 started [listener #3] 19636:20150312:024040.507 agent #4 started [active checks #1] 19633:20150312:024053.668 Executing command '/usr/bin/sudo /usr/bin/check_swift_background_daemon -w 4 -c 12 object-replicator' 19635:20150312:024053.669 Executing command '/usr/bin/sudo /usr/bin/check_swift_background_daemon -w 4 -c 12 object-updater' 19634:20150312:024053.682 Executing command '/usr/bin/sudo /usr/bin/check_swift_sweep_time -w 8 -c 12 account-auditor' 19635:20150312:024053.898 Executing command '/usr/bin/sudo /usr/bin/check_swift_sweep_time -w 8 -c 12 account-reaper' 19634:20150312:024053.904 Executing command '/usr/bin/sudo /usr/bin/check_swift_sweep_time -w 8 -c 12 account-replicator' 19635:20150312:024054.026 Executing command '/usr/bin/sudo /usr/bin/check_swift_sweep_time -w 8 -c 12 container-auditor' 19634:20150312:024055.051 Executing command '/usr/bin/sudo /usr/bin/check_swift_sweep_time -w 8 -c 12 container-replicator' 19635:20150312:024056.146 Executing command '/usr/bin/sudo /usr/bin/check_swift_sweep_time -w 8 -c 12 container-updater' 19634:20150312:024057.188 Executing command '/usr/bin/sudo /usr/bin/check_swift_sweep_time -w 8 -c 12 object-auditor' 19635:20150312:024058.280 Executing command '/usr/bin/sudo /usr/bin/check_swift_sweep_time -w 8 -c 12 object-replicator' 19633:20150312:024059.317 Executing command '/usr/bin/sudo /usr/bin/check_swift_sweep_time -w 8 -c 12 object-updater'
At this point the Zabbix installation is done and the agent has been activated on the SwiftStack node. Go check on the Reports >Availability report page to see all current reports for all hosts.