How to Set Up Nagios Alerts on a Dedicated Server?

Nagios is a powerful monitoring system that allows you to track the status of servers, services, and network equipment. However, to make Nagios truly useful, you need to properly configure the alert system. This article provides a detailed guide on configuring Nagios alerts on a dedicated server so you can quickly respond to any issues and minimize downtime. We will explore various ways to configure alerts, including using email and SMS, as well as integration with other systems.

Contents:

• Configuring Email Alerts
• Configuring SMS Alerts
• Integration with Slack and Telegram
• Configuring Alert Escalation

Configuring Email Alerts

Configuring email alerts is one of the most common and straightforward ways to receive notifications from Nagios. This method allows you to receive detailed reports on the status of your servers and services directly in your email. In this section, we will thoroughly examine how to configure sending email alerts from Nagios, what parameters to consider, and what common issues may arise.

Configuring Email Parameters in Nagios

The first step is configuring the email parameters in the Nagios configuration files. The main parameters that need to be specified include the SMTP server, port, credentials (if required), and sender address. These parameters are usually specified in the /etc/nagios/nagios.cfg file. Open this file with a text editor and find the section dedicated to email configuration.

# grep email /etc/nagios/nagios.cfg
service_notification_options  w,u,c,r,f,s
host_notification_options   d,u,r,f,s
email_notification_options  w,u,c,r,f,s

Ensure that the parameters listed above are enabled. They determine when notifications are sent (e.g., on warning (w), critical error (c), recovery (r)).

Next, you need to configure the command to send the email. Typically, the notify-service-by-email and notify-host-by-email commands, defined in the /etc/nagios/commands.cfg file, are used.

# grep notify-service-by-email /etc/nagios/commands.cfg
define command {
        command_name    notify-service-by-email
        command_line    /usr/bin/printf "%b" "*** Nagios *\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTNAME$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /usr/bin/mail -s " $NOTIFICATIONTYPE$ Service Alert: $HOSTNAME$/$SERVICEDESC$ is $SERVICESTATE$ " $CONTACTEMAIL$
        }

# grep notify-host-by-email /etc/nagios/commands.cfg
define command {
        command_name    notify-host-by-email
        command_line    /usr/bin/printf "%b" "* Nagios *\n\nNotification Type: $NOTIFICATIONTYPE$\n\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$HOSTOUTPUT$\n" | /usr/bin/mail -s " $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$
        }

Important: Make sure you have the mail package or similar installed, which is necessary for sending mail from the command line. For example, in Debian/Ubuntu: sudo apt-get install mailutils. In CentOS/RHEL: sudo yum install mailx.

Configuring Contacts to Receive Alerts

After configuring the email parameters, you need to define the contacts that will receive alerts. This is done in the /etc/nagios/conf.d/contacts.cfg file (or in a separate file specified in nagios.cfg). You need to create a contact definition, specifying their name, email address, and other parameters.

define contact {
        contact_name                    nagiosadmin
        alias                           Nagios Admin
        email                           nagios@example.com
        service_notification_period     24x7
        host_notification_period        24x7
        service_notification_options    w,u,c,r,f,s
        host_notification_options     d,u,r,f,s
        service_notification_commands   notify-service-by-email
        host_notification_commands      notify-host-by-email
        }

In this example, we created a contact named nagiosadmin, specified the email address nagios@example.com, defined the notification period (24×7), and specified the commands that will be used to send notifications by email.

Expert Tip: To prevent your mailbox from being flooded, consider using filters in your email program to automatically sort Nagios notifications.

Linking Contacts to Hosts and Services

The final step is linking the created contacts to the hosts and services for which you want to receive alerts. This is done in the host and service definitions in the /etc/nagios/conf.d/hosts.cfg and /etc/nagios/conf.d/services.cfg files (or in the corresponding files specified in nagios.cfg).

define host {
        use                     linux-server
        host_name               webserver1
        alias                   Web Server 1
        address                 192.168.1.100
        contacts                nagiosadmin
        }

define service {
        use                     generic-service
        host_name               webserver1
        service_description     HTTP
        check_command           check_http
        contacts                nagiosadmin
        }

In these examples, we linked the contact nagiosadmin to the host webserver1 and the service HTTP. Now, when the state of webserver1 or HTTP changes, nagiosadmin will receive an email notification.

After making changes to the Nagios configuration files, you need to restart the Nagios service for the changes to take effect:

sudo systemctl restart nagios

Example: Suppose you want to configure alerts for disk space on the fileserver server. You need to add a service to the /etc/nagios/conf.d/services.cfg file:

define service {
        use                     generic-service
        host_name               fileserver
        service_description     Disk Space
        check_command           check_disk!20%!10%
        contacts                nagiosadmin
        }

In this example, we use the check_disk command to monitor disk space. The parameters 20%!10% indicate the warning and critical error levels (20% and 10% free space, respectively). After adding this service and restarting Nagios, you will receive email notifications when the disk space on fileserver approaches these levels.

Important: Always check the syntax of the Nagios configuration files before restarting the service. You can do this using the command nagios -v /etc/nagios/nagios.cfg. This command will check the configuration files for errors and output the corresponding messages.

Configuring SMS Alerts

SMS alerts are an effective way to receive notifications about critical issues, especially when you don’t have access to email. Configuring SMS alerts requires integration with an external SMS gateway service. In this section, we will explore various ways to configure SMS alerts, including using free and paid services, as well as configuring commands and contacts in Nagios.

Choosing an SMS Gateway

The first step is to choose a suitable SMS gateway. There are many services that offer SMS gateway services, both paid and free. Paid services usually offer higher reliability and message delivery speed, as well as broader customization options. Free services may have limitations on the number of messages sent or delays in delivery.

Some popular SMS gateways:

• Twilio
• Nexmo (Vonage)
• Clickatell
• TextMagic

For testing purposes, you can use free SMS gateways, but for production systems, it is recommended to use paid services.

Configuring a Command to Send SMS

After selecting an SMS gateway, you need to configure a command in Nagios to send SMS messages. This command will use the API of the selected SMS gateway to send messages. You need to create a new command in the /etc/nagios/commands.cfg file.

Example: Configuring a command to send SMS via Twilio:

define command {
        command_name    notify-service-by-sms
        command_line    /usr/bin/curl -X POST 'https://api.twilio.com/2010-04-01/Accounts/ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/Messages.json' \
        --data-urlencode 'To=$CONTACTPAGER$' \
        --data-urlencode 'From=+1234567890' \
        --data-urlencode 'Body=Nagios: $NOTIFICATIONTYPE$ - $HOSTNAME$ - $SERVICEDESC$ is $SERVICESTATE$' \
        -u ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:your_auth_token
        }

define command {
        command_name    notify-host-by-sms
        command_line    /usr/bin/curl -X POST 'https://api.twilio.com/2010-04-01/Accounts/ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/Messages.json' \
        --data-urlencode 'To=$CONTACTPAGER$' \
        --data-urlencode 'From=+1234567890' \
        --data-urlencode 'Body=Nagios: $NOTIFICATIONTYPE$ - $HOSTNAME$ is $HOSTSTATE$' \
        -u ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:your_auth_token
        }

In this example, replace ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxx with your Account SID from Twilio, +1234567890 with your Twilio phone number, and your_auth_token with your Auth Token from Twilio. Also, make sure you have the curl package installed.

Important: Store your API credentials in a safe place and do not publish them publicly.

Configuring Contacts to Receive SMS Alerts

Now you need to configure the contacts that will receive SMS alerts. This is done in the /etc/nagios/conf.d/contacts.cfg file. You need to specify the contact’s phone number in the pager parameter.

define contact {
        contact_name                    admin_sms
        alias                           Admin SMS
        pager                           +79991234567
        service_notification_period     24x7
        host_notification_period        24x7
        service_notification_options    w,u,c,r,f,s
        host_notification_options     d,u,r,f,s
        service_notification_commands   notify-service-by-sms
        host_notification_commands      notify-host-by-sms
        }

In this example, we created a contact named admin_sms and specified the phone number +79991234567. Note that the phone number must be specified in international format.

Example: If you are using Nexmo, the command to send SMS will look like this:

define command {
        command_name    notify-service-by-sms
        command_line    /usr/bin/curl -d "api_key=YOUR_API_KEY&api_secret=YOUR_API_SECRET&to=$CONTACTPAGER$&from=Nagios&text=Nagios: $NOTIFICATIONTYPE$ - $HOSTNAME$ - $SERVICEDESC$ is $SERVICESTATE$" 'https://rest.nexmo.com/sms/json'
        }

define command {
        command_name    notify-host-by-sms
        command_line    /usr/bin/curl -d "api_key=YOUR_API_KEY&api_secret=YOUR_API_SECRET&to=$CONTACTPAGER$&from=Nagios&text=Nagios: $NOTIFICATIONTYPE$ - $HOSTNAME$ is $HOSTSTATE$" 'https://rest.nexmo.com/sms/json'
        }

Replace YOUR_API_KEY and YOUR_API_SECRET with your credentials from Nexmo.

Expert Tip: Use the SMS sending limits provided by your SMS gateway to avoid unexpected costs.

Linking Contacts to Hosts and Services

As with email alerts, you need to link contacts to the hosts and services for which you want to receive SMS alerts. This is done in the host and service definitions in the /etc/nagios/conf.d/hosts.cfg and /etc/nagios/conf.d/services.cfg files.

define host {
        use                     linux-server
        host_name               dbserver
        alias                   Database Server
        address                 192.168.1.101
        contacts                admin_sms
        }

define service {
        use                     generic-service
        host_name               dbserver
        service_description     Database Connection
        check_command           check_tcp!3306
        contacts                admin_sms
        }

After making changes to the Nagios configuration files, you need to restart the Nagios service:

sudo systemctl restart nagios

Important: Test sending SMS alerts to make sure they are working correctly. You can manually trigger an error to generate an alert.

Integration with Slack and Telegram

Integrating Nagios with messaging platforms like Slack and Telegram allows you to receive real-time notifications of issues directly in your communication channels. This can significantly speed up the incident response process. In this section, we will explore how to configure Nagios integration with Slack and Telegram to receive notifications about the status of your servers and services.

Integration with Slack

To integrate Nagios with Slack, you need to create a Slack App and configure an incoming webhook. A webhook is a URL to which Nagios will send notifications in JSON format.

Steps to configure integration with Slack:

• Create a Slack App in your Slack workspace.
• Activate Incoming Webhooks in your Slack App settings.
• Get the webhook URL.
• Configure a command in Nagios to send notifications to the webhook.

Example: Configuring a command in /etc/nagios/commands.cfg to send notifications to Slack:

define command {
        command_name    notify-service-by-slack
        command_line    /usr/bin/curl -X POST --data-urlencode "payload={\"channel\": \"#nagios-alerts\", \"username\": \"Nagios\", \"text\": \"*$NOTIFICATIONTYPE* Service Alert: $HOSTNAME$/$SERVICEDESC$ is $SERVICESTATE$ - $SERVICEOUTPUT$\", \"icon_emoji\": \":warning:\"}" YOUR_SLACK_WEBHOOK_URL
        }

define command {
        command_name    notify-host-by-slack
        command_line    /usr/bin/curl -X POST --data-urlencode "payload={\"channel\": \"#nagios-alerts\", \"username\": \"Nagios\", \"text\": \"*$NOTIFICATIONTYPE* Host Alert: $HOSTNAME$ is $HOSTSTATE$ - $HOSTOUTPUT$\", \"icon_emoji\": \":warning:\"}" YOUR_SLACK_WEBHOOK_URL
        }

Replace YOUR_SLACK_WEBHOOK_URL with the webhook URL you obtained when configuring the Slack App. The channel parameter specifies the channel to which notifications will be sent. The icon_emoji parameter specifies the emoji that will be displayed next to the notification.

Next, configure contacts in /etc/nagios/conf.d/contacts.cfg, specifying the commands for sending notifications to Slack:

define contact {
        contact_name                    admin_slack
        alias                           Admin Slack
        email                           ignore@example.com
        service_notification_period     24x7
        host_notification_period        24x7
        service_notification_options    w,u,c,r,f,s
        host_notification_options     d,u,r,f,s
        service_notification_commands   notify-service-by-slack
        host_notification_commands      notify-host-by-slack
        }

Finally, link the contact admin_slack to the hosts and services for which you want to receive notifications in Slack.

Important: Slack requires that the webhook URL be protected by the HTTPS protocol. Make sure your server is configured to use HTTPS.

Integration with Telegram

To integrate Nagios with Telegram, you need to create a Telegram Bot and get its API Token and Chat ID. The API Token is a unique identifier for your bot, and the Chat ID is the identifier of the chat to which notifications will be sent.

Steps to configure integration with Telegram:

• Create a Telegram Bot using BotFather in Telegram.
• Get the API Token of your bot.
• Get the Chat ID of the chat to which you want to send notifications.
• Configure a command in Nagios to send notifications via the Telegram API.

Example: Configuring a command in /etc/nagios/commands.cfg to send notifications to Telegram:

define command {
        command_name    notify-service-by-telegram
        command_line    /usr/bin/curl -s -X POST "https://api.telegram.org/botYOUR_TELEGRAM_BOT_TOKEN/sendMessage" -d "chat_id=YOUR_TELEGRAM_CHAT_ID&text=*$NOTIFICATIONTYPE*%20Service%20Alert:%20$HOSTNAME$/$SERVICEDESC$%20is%20$SERVICESTATE$%20-%20$SERVICEOUTPUT$"
        }

define command {
        command_name    notify-host-by-telegram
        command_line    /usr/bin/curl -s -X POST "https://api.telegram.org/botYOUR_TELEGRAM_BOT_TOKEN/sendMessage" -d "chat_id=YOUR_TELEGRAM_CHAT_ID&text=*$NOTIFICATIONTYPE*%20Host%20Alert:%20$HOSTNAME$%20is%20$HOSTSTATE$%20-%20$HOSTOUTPUT$"
        }

Replace YOUR_TELEGRAM_BOT_TOKEN with the API Token of your bot, and YOUR_TELEGRAM_CHAT_ID with the Chat ID of the chat. The %20 characters are used to encode spaces in the URL.

Next, configure contacts in /etc/nagios/conf.d/contacts.cfg, specifying the commands for sending notifications to Telegram:

define contact {
        contact_name                    admin_telegram
        alias                           Admin Telegram
        email                           ignore@example.com
        service_notification_period     24x7
        host_notification_period        24x7
        service_notification_options    w,u,c,r,f,s
        host_notification_options     d,u,r,f,s
        service_notification_commands   notify-service-by-telegram
        host_notification_commands      notify-host-by-telegram
        }

Finally, link the contact admin_telegram to the hosts and services for which you want to receive notifications in Telegram.

Example: If you want to send notifications to Telegram in Markdown format, use the parse_mode=Markdown parameter in the request:

define command {
        command_name    notify-service-by-telegram
        command_line    /usr/bin/curl -s -X POST "https://api.telegram.org/botYOUR_TELEGRAM_BOT_TOKEN/sendMessage" -d "chat_id=YOUR_TELEGRAM_CHAT_ID&text=*$NOTIFICATIONTYPE*%20Service%20Alert:%20$HOSTNAME$/$SERVICEDESC$%20is%20$SERVICESTATE$%20-%20$SERVICEOUTPUT$&parse_mode=Markdown"
        }

Expert Tip: Use different channels or chats for different types of notifications to make filtering and prioritization easier.

Comparison of notification methods:

Notification Method	Advantages	Disadvantages
Email	Detailed information, easy to set up	Delivery delay, may be missed
SMS	Fast delivery, reliability	Limited message size, cost
Slack/Telegram	Instant notifications, collaboration	Requires internet connection, can be distracting

Configuring Alert Escalation

Alert escalation is a mechanism that allows sending notifications to different groups of people depending on the severity of the problem and the time elapsed since its detection. This ensures that critical issues are not ignored and are resolved as quickly as possible. In this section, we will explore how to configure alert escalation in Nagios so that notifications are sent to different contacts or contact groups depending on various conditions.

Defining Notification Timeperiods

The first step in configuring alert escalation is to define notification timeperiods. Timeperiods define what time of day and which days of the week notifications should be sent. This avoids sending notifications during non-working hours if it is not necessary.

Timeperiods are defined in the /etc/nagios/conf.d/timeperiods.cfg file (or in a separate file specified in nagios.cfg).

define timeperiod {
        timeperiod_name     24x7
        alias               24 Hours A Day, 7 Days A Week
        use                 generic-timeperiod
        start_time          00:00
        end_time            24:00
        }

define timeperiod {
        timeperiod_name     workhours
        alias               Work Hours
        use                 generic-timeperiod
        monday              09:00-18:00
        tuesday             09:00-18:00
        wednesday           09:00-18:00
        thursday            09:00-18:00
        friday              09:00-18:00
        }

In this example, we defined two notification timeperiods: 24x7 (around the clock) and workhours (working hours from Monday to Friday).

Defining Contact Groups

The next step is to define contact groups. Contact groups allow you to combine multiple contacts into one group to simplify notification management. This is useful when you need to send notifications to multiple people simultaneously.

Contact groups are defined in the /etc/nagios/conf.d/contacts.cfg file.

define contactgroup {
        contactgroup_name   admins
        alias               Nagios Administrators
        members             nagiosadmin, admin_sms, admin_telegram
        }

define contactgroup {
        contactgroup_name   developers
        alias               Developers Team
        members             developer1, developer2
        }

In this example, we defined two contact groups: admins (administrators) and developers (developers). The admins group includes the contacts nagiosadmin, admin_sms, and admin_telegram, and the developers group includes the contacts developer1 and developer2.

Configuring Escalation for Hosts and Services

Now you need to configure escalation for hosts and services. This is done in the host and service definitions in the /etc/nagios/conf.d/hosts.cfg and /etc/nagios/conf.d/services.cfg files. The notification_interval, first_notification_delay, notification_period, and contacts/contact_groups parameters are used to configure escalation.

Example: Configuring escalation for a host:

define host {
        use                     linux-server
        host_name               important_server
        alias                   Important