Jump to contents

User Guide for GridDB Logstash Output Plugin

Revision: 1.5.0-9417-7fa03c1d

1 Overview

The GridDB output plugin for Logstash (logstash-plugin-griddb) is a Logstash output plugin to transfer insert data into GridDB using the HTTP/HTTPS method.

1.1 Installation

There are two ways to install the plugin: manually or by using Docker.

1.1.1 (1) Manual installation

1.1.1.1 Requirements

Building of the plugin and execution have been checked in the following environment:

  • Operation system: CentOS 7 or higher (x64)
  • Ruby version 2.4 or higher
  • Logstash version 7.x (currently does not work with logstash version 8.x)

1.1.1.2 Install Ruby

See https://www.techoism.com/install-ruby-2-3-3-centosrhel-rvm/ to set up and install Ruby.

rvm install ${RUBY_VER}

1.1.1.3 Install Logstash

See https://www.elastic.co/guide/en/logstash/current/installing-logstash.html to set up and install Logstash.

Download and install the public signing key.

$ rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Go to /etc/yum.repos.d/ and create a repo file for the yum command (e.g., logstash.repo) with the following content:

[logstash-7.x] 
name=Elastic repository for 7.x packages 
baseurl=https://artifacts.elastic.co/packages/7.x/yum 
gpgcheck=1 
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch 
enabled=1 
autorefresh=1 
type=rpm-md

Install Logstash using the following yum command.

$ yum install logstash

1.1.1.4 Install the plugin from local source code

Step 1: Copy the logstash-plugin directory to any directory on your machine that is easy to remember, such as root.

Step 2: Run the following commands to install logstash-output-griddb to your machine.

# cd logstash-plugin/source/logstash-output-griddb
# gem build logstash-output-griddb.gemspec
# /usr/share/logstash/bin/logstash-plugin install logstash-output-griddb-1.0.0.gem

1.1.2 (2) Installation using Docker

1.1.2.1 Install Docker

Follow the steps at https://docs.docker.com/engine/install/centos/.

[Note]: If you are using proxy, follow the guide at https://docs.docker.com/config/daemon/systemd

1.1.2.2 Build and run a container

  • Copy the directory /logstash-plugin/development_procedure/logstash-docker to your machine.
  • Run the command docker build -t logstash-griddb-output . in the /logstash-docker directory.
  • Run the command docker run -it logstash-griddb-output bash to access the Docker container logstash-griddb-output.

2 How to use

Edit your config file according to the section Output plugin configuration.

$ cd /etc/logstash/
$ vi mylogstash.conf

Run the following command to run Logstash:

$ /usr/share/logstash/bin/logstash -f /etc/logstash/mylogstash.conf --path.settings /etc/logstash

To automatically reload files after making changes, add --config.reload.automatic to the command and run the following:

$ /usr/share/logstash/bin/logstash -f /etc/logstash/mylogstash.conf --config.reload.automatic --path.settings /etc/logstash

2.1 Input plugin configuration

See https://www.elastic.co/guide/en/logstash/current/input-plugins.html for more details about the input plugin. This manual describes the file input plugin as an example.

2.1.1 File input

parameter value
path This is a required setting.
The value type is an array.
There is no default value for this setting.
type Types are used mainly for filter activation.

Example: Using a file input

input { 
    file {
        path => "/usr/share/logstash/run/input/sys.log"
        type => "syslog"
    }
}

2.2 Filter plugin configuration (optional)

See https://www.elastic.co/guide/en/logstash/current/filter-plugins.html for more details about filter plugins.

2.3 Output plugin configuration

2.3.1 Parameters

parameter value
host URL of GridDB WebAPI (required)
[Note]: The URL must not contain the context path of WebAPI.
e.g. Valid URL: http://192.168.0.11:8081
e.g. Invalid URL: http://192.168.0.11:8081/griddb/v2
cluster The name of a cluster in GridDB (required)
database The name of a database in GridDB into which you want to insert data. If null or empty, then the default is "public".
container The name of a container that you want to insert into GridDB (required)
insert_mode Insert mode option: append or replace. If null or empty, then the default is "append".
username GridDB user name (required)
password Password for a GridDB user (required)
proxy Proxy configuration (required if your machine is behind a proxy server)
[Note]: The proxy contains a URL, a username, and a password
e.g. full configuration:
proxy => {
url => 'http://10.116.16.12:3128'
user => 'admin'
password => 'admin'
}

2.3.2 Examples

Below are some detailed examples of using logstash-plugin-griddb.

Example 1: Using the append mode

output {
    griddb {
        host => "{host}:{port}"
        cluster => "mycluster2"
        database => "public"
        container => "logstashTest2"    
        username => "{username}"
        password => "{password}"
        insert_mode => "append"
        proxy => {
            url =>  '{host}:{port}'
            user => '{username}'
            password => '{password}'
        }
    }
}

When the append mode is used to insert data and a container does not exist, a new one is created to insert data. If a container already exists, the current one is kept into which data is inserted.

Example 2: Using the replace mode

output {
    griddb {
        host => "{host}:{port}"
        cluster => "mycluster2"
        database => "public"
        container => "logstashTest2"    
        username => "{username}"
        password => "{password}"
        insert_mode => "replace"
        proxy => {
            url =>  '{host}:{port}'
            user => '{username}'
            password => '{password}'
        }

    }
}

When the replace mode is used to insert data and a container does not exist, a new one is created to insert data. If a container already exists, the current one is deleted and a new one is created to insert data.

Example 3: Using https

output {
    griddb {
        host => "https://{full_uri}"
        cluster => "mycluster2"
        database => "public"
        container => "logstashTest2"    
        username => "{username}"
        password => "{password}"
        insert_mode => "replace"
        proxy => {
            url =>  '{host}:{port}'
            user => '{username}'
            password => '{password}'
        }
    }
}

2.4 Examples for input-output plugin configuration

Example 1: The full configuration with file input and no proxy:

input { 
    file {
        path => "/var/log/msg/logstash-tutorial-dataset"
        type => "syslog"
    }
}

output {
    stdout {}
    
    griddb {
        host => "{host}:{port}"
        cluster => "mycluster2"
        database => "public"
        container => "logstashTest2"    
        username => "{username}"
        password => "{password}"
        insert_mode => "replace"
    }
}

Example 2: The full configuration with system log input and proxy without authentication:

input { 
    file {
        path => "/var/log/msg/logstash-tutorial-dataset"
        type => "syslog"
    }
}

output {
    stdout {}
    
    griddb {
        host => "{host}:{port}"
        cluster => "mycluster2"
        database => "public"
        container => "logstashTest2"    
        username => "{username}"
        password => "{password}"
        insert_mode => "replace"
        proxy => "{host}:{port}"
    }
}
Copyright