Jump to contents

User Guide for GridDB Fluentd Output Plugin

Revision: 2.5.0-13372-ad2ab4f6

1 Overview

The GridDB output plugin for Fluentd (fluent-plugin-griddb) is a Fluentd output plugin to insert data into GridDB using the HTTP/HTTPS method.

2 Installation

2.1 Requirements

Building of the plugin and execution have been tested in the following environment:

  • Operation system: CentOS 7.6 (x64)
  • Ruby version 2.4 or higher
  • Fluentd version 1.12.0 or higher

2.2 Install Ruby

See https://www.techoism.com/install-ruby-2-3-3-centosrhel-rvm/ to set up and install Ruby.

rvm install ${RUBY_VER}

Notes : Some users need to run the following command before installing RVM :

echo insecure >> ~/.curlrc

2.3 Install fluentd

See https://docs.fluentd.org/installation/install-by-gem to set up and install fluentd. For quick installation, run:

$ gem install fluentd -v 1.12.0 --no-doc

Note: It is recommended that you use fluentd v1.12.0 if you use Ruby 2.4 because the latest version of fluentd only supports Ruby the version of which is 2.4 or higher

Note: If problems occur with 'Proxy Authentication', try:

$ gem install fluentd --no-doc --http-proxy http://host:port
or
$ gem install fluentd --no-doc --http-proxy http://username:password@host:port

Note: If your environment requires yajl-ruby and msgpack to install fluentd, please use the following two commands before installing fluentd

$ gem install yajl-ruby -v 1.4.1 
$ gem install msgpack -v 1.6.1

2.4 Install the plugin from the local source code

Step 1: Download the fluent-plugin-griddb plugin source code.

Step 2: Run the following command:

# cd fluent-plugin-griddb
# gem build fluent-plugin-griddb.gemspec
# gem install --force --local fluent-plugin-griddb-1.0.2.gem

3 How to use

3.1 Quick start

Run the following command to create a sample configuration file:

$ fluentd --setup ./fluent

Configure ./fluent/fluent.conf according to your requirements. For details, see the corresponding sections in https://docs.fluentd.org/installation/install-by-gem .

If you want to run it as a daemon, run the following command:

$ fluentd -c ./fluent/fluent.conf --v &

Run the following command to stop it:

$ pkill -f fluentd

If you want to run it as a foreground process, run the following command:

$ fluentd -c ./fluent/fluent.conf

You can use the following command to send a JSON message with a tag such as debug.test.

$ echo '{"json":"message"}' | fluent-cat debug.test

3.2 Input plugin configuration

The input plugin is out of scope of this guideline. See https://docs.fluentd.org/input for detail.

Here are some examples you can use:

3.2.1 Apache log

Parameters

parameter value
@type tail (required)
path The path of the log file to read (required)
pos_file Fluentd will record the position it last read from this file.
tag The tag of the event (required)
parse The format of the log (required)

Examples

Configuration of ./fluent/fluent.conf.

  • If Apache is already installed on the system, then use the following input:
<source>
  @type tail
  path /var/log/httpd-access.log
  pos_file /var/log/td-agent/httpd-access.log.pos
  tag griddb
  <parse>
    @type apache2
  </parse>
</source>

  • If Apache is not installed on the system, create the following two files:

    • file1.log, which has the following formatted content:
    192.168.0.1 - - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0"
192.168.0.1 - - [28/Feb/2013:12:00:00 +0900] "POST / HTTP/1.1" 200 777 "-" "Opera/12.0"
192.168.0.1 - - [28/Feb/2013:12:00:00 +0900] "PUT / HTTP/1.1" 200 777 "-" "Opera/12.0"
192.168.0.1 - - [28/Feb/2013:12:00:00 +0900] "DELETE / HTTP/1.1" 200 777 "-" "Opera/12.0"
    • file2.log.pos, which is an empty file
<source>
  @type tail
  path your_folder/file1.log
  pos_file your_folder/file2.log.pos
  tag griddb
  <parse>
    @type apache2
  </parse>
</source>

3.2.2 Tail

Note: See https://docs.fluentd.org/input/tail for detail. Apache log is a special case of tail_input plugin.

Parameters

parameter value
@type tail (required)
path The path of the log file to read (required)
pos_file Fluentd will record the position it last read from this file
tag The tag of the event (required)
parse The format of the log (required)

Examples

  • The parser plugin type is csv.
CSV format in file1.log:
2013/02/28 12:00:00,192.168.0.1,111,text

<source>
  @type tail
  path your_folder/file1.log
  pos_file your_folder/file2.log.pos
  <parse>
    @type csv
    keys time,host,req_id,user
    time_key time
  </parse>
  tag griddb
</source>
  • The parser plugin type is none
NONE format in file1.log:
Hello world. I am a line of log!

<source>
  @type tail
  path your_folder/file1.log
  pos_file your_folder/file2.log.pos
  <parse>
    @type none
  </parse>
  tag griddb
</source>
  • The parser plugin type is json.
JSON format in file1.log:
{"field1":"type", "field2":"type", "field3":"type:option", "field4":"type:option"}

<source>
  @type tail
  path your_folder/file1.log
  pos_file your_folder/file2.log.pos
  <parse>
    @type json
  </parse>
  tag griddb
</source>
  • The parser plugin type is regexp.
REGEXP format in file1.log:
[2013-02-28 12:00:00 +0900] name engineer 1

<source>
  @type tail
  path your_folder/file1.log
  pos_file your_folder/file2.log.pos
  <parse>
    @type regexp
    expression /^\[(?<logtime>[^\]]*)\] (?<name>[^ ]*) (?<title>[^ ]*) (?<id>\d*)$/
    time_key logtime
    time_format %Y-%m-%d %H:%M:%S %z
    types id:integer
  </parse>
  tag griddb
</source>
  • The parser plugin type is apache_error.
APACHE_ERROR format in file1.log:
[Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server configuration

<source>
  @type tail
  path your_folder/file1.log
  pos_file your_folder/file2.log.pos
  <parse>
    @type apache_error
    expression /^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])? \[client (?<client>[^\]]*)\] (?<message>.*)$/
  </parse>
  tag griddb
</source>
  • The parser plugin type is nginx.
NGINX format in file1.log:
127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -

<source>
  @type tail
  path your_folder/file1.log
  pos_file your_folder/file2.log.pos
  <parse>
    @type nginx
  </parse>
  tag griddb
</source>

3.2.3 System logs

Note: See https://docs.fluentd.org/input/syslog for detail.

Parameters

parameter value
@type syslog (required)
port The port to listen to
bind The bind address to listen to
parse The format of the log (required)
tag The prefix of the tag

Setting up rsyslogd

Open /etc/rsyslogd.conf and append the following line:

*.* @127.0.0.1:5140

If you are using rsyslogd, add the following lines to /etc/rsyslog.conf:

# Send log messages to Fluentd
*.* @127.0.0.1:5140

Then restart the rsyslogd service:

$ sudo systemctl restart rsyslog

This tells rsyslogd to forward logs to port 5140 to which Fluentd will listen.

Example

Configuration of ./fluent/fluent.conf.

<source>
  @type syslog
  port 5140
  bind 0.0.0.0
  <parse>
    @type syslog
  </parse>
  tag griddb
</source>

3.3 Filter plugin configuration (optional)

<filter></filter>

Note: See https://docs.fluentd.org/filter for detail.

Example 1: filter_grep filter plugin

 <filter griddb.**>
   @type grep
   <regexp>
    key ident
    pattern /^sudo$/
  </regexp>
  <regexp>
    key message
    pattern /COMMAND/
  </regexp>
 </filter>

Example 2: filter_stdout filter plugin

<filter >
  @type stdout
</filter>

Example 3: filter_record_transformer filter plugin

<filter>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
    tag ${tag}
  </record>
</filter>

3.4 Output plugin configuration

3.4.1 Parameters

parameter value
@type griddb (required)
host URL of GridDB Web API (required)
Note: The URL must not contain the context path of Web API.
e.g. Valid URL: http://192.168.0.11:8081
Invalid URL: http://192.168.0.11:8081/griddb/v2
cluster cluster name (required)
database If null or empty, then the default is "public".
container container name (required)
insert_mode append/replace. If null or empty, then the default is "append".
username GridDB user name (required)
password password for a GridDB user (required)
p_addr proxy address (required if the machine is behind a proxy server)
p_port proxy port (required if the machine is behind a proxy server)
p_user proxy username (required if the machine is behind a proxy server with authentication)
p_pass proxy password (required if the machine is behind a proxy server with authentication)

3.4.2 Examples

Below are some detailed examples where fluent-plugin-griddb is used.

Example 1: Use http for the host and specify insert_mode as append.

<match griddb>
   @type griddb
   host http://localhost:8080/
   cluster defaultCluster
   database public
   container container_1
   insert_mode append
   username admin
   password admin
</match>

When the append mode is used to insert data and a container does not exist, a new one is created to insert data. If a container already exists, the current one is kept into which data is inserted.

Example 2: Use https for the host and specify insert_mode as replace.

<match griddb.**>
   @type griddb
   host https://{please_specify_full_uri}
   cluster defaultCluster
   database public
   container container_2
   insert_mode replace
   username admin
   password admin
   p_addr proxyAddress
   p_port proxyPort
</match>

When the replace mode is used to insert data and a container does not exist, a new one is created to insert data. If a container already exists, the current one is deleted and a new one is created to insert data.

3.5 Examples for full configuration

Example 1: The full configuration with http, Apache log files, and no proxy:

<source>
  @type tail
  path /var/log/httpd-access.log
  pos_file /var/log/td-agent/httpd-access.log.pos
  tag griddb
  <parse>
    @type apache2
  </parse>
</source>

<match griddb>
  @type griddb
  host http://{please_specify_full_uri}
  cluster clusterName
  database defaultDB/databaseName
  container containerName
  insert_mode mode
  username username
  password password
</match>
<match>
  @type stdout
</match>

Example 2: The full configuration with https, system log type, and proxy without authentication:

<source>
  @type syslog
  port 5140
  tag griddb
</source>

<filter griddb.**>
  @type grep
  <regexp>
    key ident
    pattern /^sudo$/
  </regexp>
  <regexp>
    key message
    pattern /COMMAND/
  </regexp>
  <exclude>
     key message
     pattern /COMMAND/
  </exclude>
</filter>

<filter griddb.**>
  @type parser
  key_name message
  <parse>
    @type regexp
    expression /USER=(?<sudoer>[^ ]+) ; COMMAND=(?<command>.*)$/
  </parse>
</filter>

<match griddb>
  @type griddb
  host https://{please_specify_full_uri}
  cluster clusterName
  database defaultDB/databaseName
  container containerName
  insert_mode mode
  username username
  password password
  p_addr proxyAddress
  p_port proxyPort
</match>

<match>
  @type stdout
</match>

Example 3: The full configuration with proxy authentication

<source>
  @type tail
  path /var/log/httpd-access.log
  pos_file /var/log/td-agent/httpd-access.log.pos
  tag griddb
  <parse>
    @type apache2
  </parse>
</source>

<match griddb>
 @type griddb
 host https://{please_specify_full_uri}
 cluster clusterName
 database defaultDB/databaseName
 container containerName
 insert_mode mode
 username username
 password password
 p_addr proxy-address
 p_port proxy-port
 p_user proxy-username
 p_pass proxy-password
</match>

<match>
  @type stdout
</match>
Copyright