Graylog2 install Slack Notifications

The graylog Marketplace has a great document on getting started, but I ran into one snag adding HTTPS certificate to java store for hooks.slack.com.

For reference here is link to marketplace 

Required Graylog version: 2.0 and later. Detailed alarm notification and message output:

This plugin can notify Slack or Mattermost channels about triggered alerts in Graylog (Alarm Callback) and also forward each message routed into a stream (Message Output) in realtime.

Short mode message output:

Great for streams with higher message throughput. The screenshot shows the output of a nightly task that updates information of the Graylog Marketplace.

Changes in v3.0

*Templated message are now supported. They use the same format as email alerts.

Installation

Download the plugin and place the .jar file in your Graylog plugin directory. The plugin directory is the plugins/ folder relative from your graylog-server directory by default and can be configured in your graylog.conf file.

Restart graylog-server and you are done.

Usage

For Slack:

Step 1: Create Slack Incoming Webhook

Create a new Slack Incoming Webhook (https://<organization>.slack.com/services/new/incoming-webhook) and copy the URL it will present to you. It will ask you to select a Slack channel but you can override it in the plugin configuration later.

For Mattermost:

Step 1: Create Mattermost Incoming Webhook

Enable Webhooks in general and create an incoming Webhook for Graylog as described in the Mattermost docs.

Step 2: Create alarm callback or message output

Create a “Slack alarm callback” on the “Manage alerts” page of your stream. Enter the requested configuration (use the Incoming Webhook URL you created in step 1) and save. Make sure you also configured alert conditions for the stream so that the alerts are actually triggered.

The same applies for message outputs which you can configure in Stream – > Manage Outputs.

Troubleshooting

HTTPS connection fails

If the Java runtime environment and the included SSL certificate trust store is too old, HTTPS connections to Slack might fail with the following error message:

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Alternate Error Messages

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Setting Certificate in Java Store

openssl s_client -servername hooks.slack.com -connect hooks.slack.com:443 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' >certificate.pem
sudo cp -a "${JAVA_HOME}/jre/lib/security/cacerts" /opt/certs/cacerts.jks
keytool -importcert -file certificate.cer -keystore /opt/certs/cacerts.jks -alias "slack"
keytool -keystore /opt/certs/cacerts.jks -storepass changeit -list

Edit /etc/default/graylog-server

# Default Java options for heap and garbage collection.
GRAYLOG_SERVER_JAVA_OPTS="-Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/opt/certs/cacerts.jks"

Where “-Djavax.net.ssl.trustStore=/opt/certs/cacerts.jks” was added to end of Java Opts line.

Restart the Service and Test Output Notification to Graylog.