Graylog HA with Rancher

Introduction:

This is an example of how I have deployed Graylog2 in my Rancher infrastructure.  There are other posts that describe more about the actual Rancher infrastructure itself.  I am not proposing that is this the “right way” or even the “best way” to deploy Graylog2.  This is what worked for me and the result has been very stable and durable, withstanding the loss of an instance without loosing data.

Components

Rancher Server
AWS – Launch Configuration
AWS – Auto-Scaling Group
3 AWS t2.small instances generated via AutoScaling Group
MongoDB Cluster
ElasticsearchCluster
Graylog Server

I have also previously created a GlusterFS Cluster which is where my persistent data is stored.

Graylog HA Architecture

Graylog Cluster

Created an AMI based on the Docker Gold, but with 40GB of storage for the Elasticsearch storage.  Called it Graylog-Gold

Launch Configuration:

This launch configuration will create an instance with the rancher agent and label it “GraylogPool=yes” and mount the GlusterFS volume at /app/

#!/bin/bash
sudo perl -pi -e 's/^#?Port 22$/Port 2222/' /etc/ssh/sshd_config
sudo service ssh restart
sudo docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher -e CATTLE_HOST_LABELS=GraylogPool=yes rancher/agent:v1.0.1 http://rch.domain.com:8080/v1/scripts/D67D5110CBA5A0577D90:1464310800000:R2PLB0VagVreCqEUp94tPNPw5g
sudo mount -t glusterfs glusterfs1.domain.com:vol1 /app

Note: The code in bold comes from Rancher. Click on Add Host and choose Custom.  Copy the text from section 5 and add your host label.

AutoScaling Group

AutoScalingGroup
Example of AWS Autoscaliing Group for Graylog Servers

Folders on GlusterFS

(do not create folders directly on  the GlusterFS servers.  Use a server where the volume has been mounted.)

cd /app/data

mkdir graylog/config
 cd graylog/config
 wget https://raw.githubusercontent.com/Graylog2/graylog2-images/2.0/docker/config/graylog.conf
 wget https://raw.githubusercontent.com/Graylog2/graylog2-images/2.0/docker/config/log4j2.xml

In the same graylog directory I also created the following folders: journal, mongo, plugins

Docker/Rancher Compose for ElasticSearch (This should be launched first)

docker-compose.yml

elasticsearch-datavolume-clients:
  labels:
    io.rancher.scheduler.affinity:container_label_soft_ne: io.rancher.stack_service.name=es/elasticsearch-clients
    io.rancher.container.start_once: 'true'
    io.rancher.scheduler.affinity:host_label: graylogpool=yes
    io.rancher.container.hostname_override: container_name
    elasticsearch.datanode.config.version: '0'
  entrypoint:
  - /bin/true
  image: elasticsearch:2.2.1
  links:
  - elasticsearch-masters:es-masters
  volumes:
  - /opt/rancher/bin
  - /usr/share/elasticsearch/config
  - /usr/share/elasticsearch/data
elasticsearch-base-datanode:
  labels:
    io.rancher.scheduler.affinity:host_label: graylogpool=yes
    io.rancher.container.hostname_override: container_name
    io.rancher.scheduler.affinity:container_label_ne: io.rancher.stack_service.name=$${stack_name}/$${service_name}
    elasticsearch.datanode.config.version: '0'
  entrypoint:
  - /opt/rancher/bin/run.sh
  image: elasticsearch:2.2.1
  links:
  - elasticsearch-masters:es-masters
  volumes_from:
  - elasticsearch-datavolume-datanode
  net: container:elasticsearch-datanodes
elasticsearch-clients:
  log_driver: ''
  labels:
    io.rancher.sidekicks: elasticsearch-datavolume-clients,elasticsearch-base-clients
    io.rancher.scheduler.affinity:container_label_soft_ne: io.rancher.stack_service.name=es/elasticsearch-clients
    io.rancher.scheduler.affinity:host_label: graylogpool=yes
    io.rancher.container.hostname_override: container_name
  log_opt: {}
  image: rancher/elasticsearch-conf:v0.5.0
  links:
  - elasticsearch-masters:es-masters
  volumes_from:
  - elasticsearch-datavolume-clients
elasticsearch-datavolume-datanode:
  labels:
    io.rancher.container.start_once: 'true'
    io.rancher.scheduler.affinity:host_label: graylogpool=yes
    io.rancher.container.hostname_override: container_name
    io.rancher.scheduler.affinity:container_label_ne: io.rancher.stack_service.name=$${stack_name}/$${service_name}
    elasticsearch.datanode.config.version: '0'
  entrypoint:
  - /bin/true
  image: elasticsearch:2.2.1
  links:
  - elasticsearch-masters:es-masters
  volumes:
  - /opt/rancher/bin
  - /usr/share/elasticsearch/config
  - /usr/share/elasticsearch/data
elasticsearch-datavolume-masters:
  labels:
    io.rancher.container.start_once: 'true'
    io.rancher.scheduler.affinity:host_label: graylogpool=yes
    io.rancher.container.hostname_override: container_name
    io.rancher.scheduler.affinity:container_label_ne: io.rancher.stack_service.name=es/elasticsearch-masters
    elasticsearch.datanode.config.version: '0'
  entrypoint:
  - /bin/true
  image: elasticsearch:2.2.1
  volumes:
  - /opt/rancher/bin
  - /usr/share/elasticsearch/config
  - /usr/share/elasticsearch/data
kopf:
  ports:
  - 8090:80/tcp
  environment:
    KOPF_ES_SERVERS: es-clients:9200
    KOPF_SERVER_NAME: es.dev
  labels:
    io.rancher.scheduler.affinity:host_label: graylogpool=yes
    io.rancher.container.hostname_override: container_name
  image: rancher/kopf:v0.4.0
  links:
  - elasticsearch-clients:es-clients
elasticsearch-datanodes:
  log_driver: ''
  labels:
    io.rancher.sidekicks: elasticsearch-base-datanode,elasticsearch-datavolume-datanode
    io.rancher.scheduler.affinity:host_label: graylogpool=yes
    io.rancher.container.hostname_override: container_name
    io.rancher.scheduler.affinity:container_label_ne: io.rancher.stack_service.name=$${stack_name}/$${service_name}
  log_opt: {}
  image: rancher/elasticsearch-conf:v0.5.0
  links:
  - elasticsearch-masters:es-masters
  volumes_from:
  - elasticsearch-datavolume-datanode
elasticsearch-base-clients:
  labels:
    io.rancher.scheduler.affinity:container_label_soft_ne: io.rancher.stack_service.name=es/elasticsearch-clients
    io.rancher.scheduler.affinity:host_label: graylogpool=yes
    elasticsearch.client.config.version: '0'
    io.rancher.container.hostname_override: container_name
  entrypoint:
  - /opt/rancher/bin/run.sh
  image: elasticsearch:2.2.1
  links:
  - elasticsearch-masters:es-masters
  volumes_from:
  - elasticsearch-datavolume-clients
  net: container:elasticsearch-clients
elasticsearch-base-master:
  labels:
    io.rancher.scheduler.affinity:host_label: graylogpool=yes
    elasticsearch.master.config.version: '0'
    io.rancher.container.hostname_override: container_name
    io.rancher.scheduler.affinity:container_label_ne: io.rancher.stack_service.name=es/elasticsearch-masters
  entrypoint:
  - /opt/rancher/bin/run.sh
  image: elasticsearch:2.2.1
  volumes_from:
  - elasticsearch-datavolume-masters
  net: container:elasticsearch-masters
elasticsearch-masters:
  log_driver: ''
  labels:
    io.rancher.sidekicks: elasticsearch-datavolume-masters,elasticsearch-base-master
    io.rancher.scheduler.affinity:host_label: graylogpool=yes
    io.rancher.container.hostname_override: container_name
    io.rancher.scheduler.affinity:container_label_ne: io.rancher.stack_service.name=es/elasticsearch-masters
  log_opt: {}
  image: rancher/elasticsearch-conf:v0.5.0
  volumes_from:
  - elasticsearch-datavolume-masters

rancher-compose.yml

elasticsearch-datavolume-clients:
  scale: 3
  metadata:
    elasticsearch: &id001
      yml:
        cluster.name: graylog
        network.host: _site_
        node.data: 'false'
        node.master: 'false'
        node.name: $${HOSTNAME}
elasticsearch-base-datanode:
  scale: 3
  metadata:
    elasticsearch: &id002
      yml:
        cluster.name: graylog
        http.enabled: 'false'
        network.host: _site_
        node.data: 'true'
        node.master: 'false'
        node.name: $${HOSTNAME}
elasticsearch-clients:
  scale: 3
  metadata:
    elasticsearch: *id001
elasticsearch-datavolume-datanode:
  scale: 3
  metadata:
    elasticsearch: *id002
elasticsearch-datavolume-masters:
  scale: 3
  metadata:
    elasticsearch: &id003
      yml:
        cluster.name: graylog
        network.host: _site_
        node.data: 'false'
        node.master: 'true'
        node.name: $${HOSTNAME}
kopf:
  scale: 1
elasticsearch-datanodes:
  scale: 3
  metadata:
    elasticsearch: *id002
elasticsearch-base-clients:
  scale: 3
  metadata:
    elasticsearch: *id001
elasticsearch-base-master:
  scale: 3
  metadata:
    elasticsearch: *id003
elasticsearch-masters:
  scale: 3
  metadata:
    elasticsearch: *id003

ElasticSearch Configuration

There are some configuration steps that need to be performed after the Graylog stack is running.

Docker/Rancher Compose for Graylog

docker-compose.yml

gray-mongo:
  labels:
    io.rancher.scheduler.affinity:host_label: graylogpool=yes
    io.rancher.container.hostname_override: container_name
  image: mongo
  volumes:
  - /app/data/graylog/mongo:/data/db
graylog:
  ports:
  - 12201:12201/tcp
  - 12201:12201/udp
  - 12900:12900/tcp
  - 1514:1514/udp
  - 9000:9000/tcp
  environment:
    GRAYLOG_PASSWORD_SECRET: PASSWORD$$
    GRAYLOG_REST_TRANSPORT_URI: http://graylog.domain.com:12900/
    GRAYLOG_ROOT_PASSWORD_SHA2: 5de84f473602e86256d3673b9d0000000000000e1341ba65de5d3619b65342ad
    GRAYLOG_TIMEZONE: PST8PDT
    GRAYLOG_ELASTICSEARCH_CLUSTER_DISCOVERY_TIMEOUT: '55000'
    GRAYLOG_ELASTICSEARCH_CLUSTER_NAME: graylog
   GRAYLOG_transport_email_enabled: true
   GRAYLOG_transport_email_hostname: email-smtp.us-west-2.amazonaws.com
   GRAYLOG_transport_email_port: 587
   GRAYLOG_transport_email_use_auth: true
   GRAYLOG_transport_email_use_tls: true
   GRAYLOG_transport_email_use_ssl: true
   GRAYLOG_transport_email_auth_username: SES-USER-NAME
   GRAYLOG_transport_email_auth_password: SES-PASSWORD
   GRAYLOG_transport_email_subject_prefix: [graylog]
   GRAYLOG_transport_email_from_email: graylog@domain.com
   GRAYLOG_transport_email_web_interface_url: http://graylog.domain.com:9000
  external_links:
  - es/elasticsearch-masters:elasticsearch
  log_driver: ''
  labels:
    io.rancher.scheduler.affinity:host_label: graylogpool=yes
    io.rancher.container.hostname_override: container_name
  tty: true
  log_opt: {}
  image: graylog2/server:latest
  links:
  - gray-mongo:mongo
  volumes:
  - /app/data/graylog/journal:/usr/share/graylog/data/journal
  - /app/data/graylog/config:/usr/share/graylog/data/config
  - /app/data/graylog/plugins:/usr/share/graylog/data/plugin
  stdin_open: true

rancher-compose.yml

gray-mongo:
  scale: 1
graylog:
  scale: 1

Post Installation Configuration

ElasticSearch Configuration

This article was helpful:  http://rancher.com/running-elasticsearch-on-rancher/#more-4051

Using KOPF (This should be done after Graylog is installed)

Graylog installation will create an index with 4 shards. This is named graylog_0.  By default it will only exist on one data node. We want to use all our ES nodes.

From the Cluster screen, click the down arrow next to the index name.

kopf2

Click “edit settings”

kopf3

In the Routing section: Enter 3 in the total_shards_per_node. section.

kopf-shards

In the Index section: Enter 1 in number of replicas

kopf1

The replicas are spread across nodes almost immediately.

kopf5

Configuring Graylog

Graylog Inputs

These are required in order for Graylog to begin receiving logs.

WinServer (GELF TCP)

This one is for the IIS and windows logs on WinServer. Other than the port, most of the settings are just default for the input type.

bind_address: 0.0.0.0
max_message_size: 2097152
override_source: <empty>
port: 12201
recv_buffer_size: 1048576
tcp_keepalive: false
tls_cert_file: <empty>
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: false
tls_key_file: <empty>
tls_key_password: ********
use_null_delimiter: true

Syslog-UDP (Syslog UDP)

allow_override_date: true
bind_address: 0.0.0.0
expand_structured_data: false
force_rdns: false
override_source: <empty>
port: 1514
recv_buffer_size: 262144
store_full_message: false

Getting Logs Into Graylog

Windows Server running IIS

Used: graylog-collector-0.4.1

Note:  I know this is deprecated but it was already installed and running from my previous Graylog install.

Created the following collector.conf

server-url = "http://graylog.domain.com:12900/"
collector-id = WinServer
enable-registration = false

inputs {
 win-eventlog-application {
   type = "windows-eventlog"
   source-name = "Application"
   poll-interval = "1s"
 }
 win-eventlog-system {
   type = "windows-eventlog"
   source-name = "System"
   poll-interval = "1s"
 }
 win-eventlog-security {
   type = "windows-eventlog"
   source-name = "Security"
   poll-interval = "1s"
 }
 iis {
   type = "file"
   path-glob-root = "G:\\app\\logs\\W3SVC1"
    path-glob-pattern = "u_ex*.log"
    reader-interval = "15s"
 }
}

outputs {
 gelf-tcp {
   type = "gelf"
   host = "graylog.domain.com"
   port = 12201
 }
}

 

Note: There is a newer way to do this that involves using NXLog and a Graylog Collector Sidecar.  This was not available to me at the time I was setting this up.

http://docs.graylog.org/en/latest/pages/collector_sidecar.html

Docker Logs

For the Docker logs I am using Logspout.  This is a docker container that reads all the “standard out and standard err” logs from other docker containers. It’s in the catalog, but I modified it to work with graylog.  It can only pull logs from containers that are not started with the tty option set to true.

logspout:
  environment:
    LOGSPOUT: ignore
    ROUTE_URIS: syslog://graylog.domain.com:1514
  log_driver: ''
  labels:
    io.rancher.scheduler.global: 'true'
    io.rancher.container.hostname_override: container_name
    io.rancher.scheduler.affinity:host_label_ne: service=fileserver,graylogpool=yes
  tty: true
  log_opt: {}
  image: rancher/logspout-logstash:v0.2.0
  volumes:
  - /var/run/docker.sock:/var/run/docker.sock
  stdin_open: true

Note that I have it configured not to run on the graylog servers.

Graylog Extractors

An extractor will parse an incoming log and extract out various fields which can then become the basis for searches, tables and charts.  These are configured per Input.

Example Extractor for input WinServer (GELF TCP)

Condition

Will only attempt to run if the message includes the string GET /MYAPPNAME.aspx

Configuration

grok_pattern: reportname=%{DATA:REPORT}&_userName=%{DATA:UserName}&_userOrg=%{DATA:UserOrg}&_org=%{DATA:Customer}&_app=%{DATA:Application}&

 

Graylog Dashboards

Dashboards are first created then then you can add widgets. Each widget is based on a search parameters.

graylog-dashboard1

I clicked on Create Dashboard and created one called WinServer

I then performed this search:

WINSERVER AND "GET \/MYAPPNAME.aspx"

Results looked like this:

graylog-dashboard2
Add to Graylog Dashboard from search result

I clicked on Add to dashboard and chose my dashboard.

Because I created an extractor for these messages, I can click on any of the Fields to have them show up in the Messages columns.  I can also expand them.

Generate Graylog Quick Value Widget
Generate Graylog Quick Value Widget

If I choose Quick values, Graylog will produce a pie chart and data table like this:

graylog-quick-values
Graylog Quick Values

I can then click on Add to dashboard and add that to my dashboard.

If I do the same thing for the Customer field I get this widget.

Graylog_Quick_Values
Graylog Quick Values

I can also add a message count to my dashboard by clicking on Add count to dashboard.

graylog-dashboard6

Once I have added everything I want to my dashboard I can arrange and resize them as I prefer.

 

Follow by Email
Facebook
Google+
http://cloudlady911.com/index.php/2016/06/23/graylog-ha-with-rancher/
LinkedIn

2 thoughts on “Graylog HA with Rancher

  1. Bernie Carolan Reply

    Great article, has helped a lot with a project I’m working on.
    With the docker-compose for Elasticsearch, have you looked into use routing allocation for hot/warm nodes?
    Just trying to find a way to expand that configuration for 3 hot nodes & 3 warm nodes without too many changes.

    • cloudlady Post authorReply

      Bernie, I have not attempted that. Let me know if you figure out a way to do that. I would love to post it here.

Leave a Reply

Your email address will not be published. Required fields are marked *