GridDB technical reference

As the scale of a system expands, the data volume handled increases and thus the system needs to be expanded so as to quickly process the big data.

System expansion can be broadly divided into 2 approaches - scale-up (vertical scalability) and scale-out (horizontal scalability).

• What is scale-up?

  This approach reinforces the system by adding memory to the operating machines, using SSD for the disks, adding processors, and so on. Generally, there is a need to stop the nodes once during scale-up operation as it is not a cluster application using multiple machines even though each individual processing time is shortened and the system processing speed is increased. When a failure occurs, failure recovery is also time-consuming.

• What is scale-out?

  This approach increases the number of nodes constituting a system to improve the processing capability. Generally, there is no need to completely stop service when a failure occurs and during maintenance as multiple nodes are linked and operating together. However, the application management time and effort increases as the number of nodes increases. This architecture is suitable for performing highly parallel processing.

In GridDB, in addition to the scale-up approach to increase the number of operating nodes and reinforce the system, new nodes can be added to expand the system with a scale-out approach to incorporate nodes into an operating cluster.

As an in-memory processing database, GridDB can handle a large volume of data with its scale-out model. In GridDB, data is distributed throughout the nodes inside a cluster that is composed of multiple nodes. Therefore, a large-scale memory database can be provided as the memories of multiple nodes can be used as a single, large memory space.

In addition, since data management of a hybrid composition that combines the use of disk with memory is also possible, data exceeding the memory size can be retained and accessed even when operating with a standalone node. A large capacity that is not limited by the memory size can also be realized.

System expansion can be carried out online with a scale-out approach. As a result, a system in operation can be supported without having to stop it as it will support the increasing volume of data as the system grows.

In the scale-out approach, data is arranged in an appropriate manner according to the load of the system in the nodes built into the system. As GridDB will optimize the load balance, the application administrator does not need to worry about the data arrangement. Operation is also easy because a structure to automate such operations has been built into the system.

GridDB data adopts a Key-Container data model that is expanded from Key-Value. Data is stored in a device equivalent to a RDB table known as a container. (A container can be considered a RDB table for easier understanding.)

When accessing data in GridDB, the model allows data to be short-listed with a key thanks to its Key-Value database structure, allowing processing to be carried out at the highest speed. A design that prepares a container serving as a key is required to support the entity under management.

Besides being suitable for handling a large volume of time series data (TimeSeries container) that is generated by a sensor or the like and other values paired with the time of occurrence, space data such as position information, etc. can also be registered and space specific operations (space intersection) can also be carried out in a container. A variety of data can be handled as the system supports non-standard data such as array data, BLOB and other data as well.

A unique compression function and a function to release data that has expired and so on are provided in a TimeSeries container, making it suitable for the management of data which is generated in large volumes.

A variety of architectural features is embedded in GridDB to achieve high-speed processing.

Duplicate data (hereinafter referred to replicas) are created in the cluster and processing can be continued by using these replicas even when a failure occurs in any of the nodes constituting a cluster. Special operating procedures are not necessary as the system will also automatically perform re-arrangement of the data after a node failure occurs (autonomous data arrangement). Data arranged in a failed node is restored from a replica and then the data is re-arranged so that the set number of replicas is reached automatically.

Duplex, triplex or multiplex replica can be set according to the availability requirements.

Each node performs persistence of the data update information using a disk, and all registered and updated data up to that point in time can be restored without being lost even if a failure occurs in the entire cluster system.

In addition, since the client also possesses cache information on the data arrangement and management, upon detecting a node failure, it will automatically perform a failover and data access can be continued using a replica.

There are 2 GridDB status, nodeStatus and clusterStatus that can be checked with a gs_stat command. The status of a node is determined by these 2 statuses.

nodeStatus indicates the operating status of the node while clusterStatus indicates the role of each node in the constituted cluster. The status of the entire cluster is determined by the status of these multiple nodes belonging to the cluster.

• Transition in the node status

  The node status may be one of the following statuses shown in the diagram below.

  

  Node status

  • [STOP]: State in which the GridDB server has not been started in the node.
  • [STARTING]: State in which the GridDB server is starting in the node. Depending on the previous operating state, start-up processes such as recovery processing of the database are carried out. The only possible access from a client is checking the status of the system with a gs_stat command or gs_sh command. Access from the application is not possible.
  • [STARTED]: State in which the GridDB server has been started in the node. However, continued access from the application is not possible as the node has not joined the cluster. To obtain the cluster composition, a command is issued to join a cluster with the gs_joincluster or gs_sh cluster operating command.
  • [WAIT]: State in which the system is waiting for the cluster composition. Nodes have been informed to join a cluster but the number of nodes constituting a cluster is insufficient, so the system is waiting for the number of nodes constituting a cluster to be reached. It also indicates the node status when the number of nodes constituting a cluster drops below the majority and the cluster service is stopped.
  • [SERVICING]: State in which a cluster has been constituted and access from the application is possible. However, access may be delayed if synchronization between the clusters of the partition occurs due to a re-start after a failure when the node is stopped or the like.
  • [STOPPING]: Intermediate state in which a node has been instructed to stop but has not stopped yet.
  • [ABNORMAL]: SERVICING state or state in which an error is detected by the node in the middle of the state transition. A node in the ABNORMAL state will be automatically separated from the cluster. After obtaining the operating information of the system, the system needs to be stopped by force and then re-started. By re-starting the system, recovery processing will be automatically carried out.
• Description of state transition

  A description of events that serve as an opportunity to change the status of a node.

  State transition	State transition event	Description
  1	Command execution	Node start-up using gs_startnode command, gs_sh, service start-up
  2	System	Automatic transition at the end of recovery processing or loading of database files
  3	Command execution	Cluster participation using gs_joincluster/gs_appendcluster command, gs_sh, service start-up
  4	System	State changes when the required number of component nodes join a cluster
  5	System	When other nodes that make up a cluster are detached from the service due to a failure, etc., and the number of nodes constituting a cluster drops below half of the value set.
  6	Command execution	Detaches a node from a cluster using a gs_leavecluster command or gs_sh
  7	Command execution	Detaches a node from a cluster using a gs_leavecluster/gs_stopcluster command or gs-sh
  8	Command execution	Stops a node using gs_stopnode command, gs_sh, service stop
  9	System	Stops the server process once the final processing ends
  10	System	Detached state due to a system failure. In this state, the node needs to be stopped by force once.

• Node status and nodeStatus, clusterStatus

  By using a gs_stat command, the detailed operating information of the node can be checked with text in the json format. The relationship between the clusterStatus and the nodeStatus which is a json parameter to indicate the gs_stat is shown below.

  Status	/cluster/nodeStatus	/cluster/clusterStatus
  STARTING	INACTIVE	SUB_CLUSTER
  STARTED	INACTIVE	SUB_CLUSTER
  WAIT	ACTIVATING or DEACTIVATING	SUB_CLUSTER
  SERVICING	ACTIVE	MASTER or FOLLOWER
  STOPPING	NORMAL_SHUTDOWN	SUB_CLUSTER
  ABNORMAL	ABNORMAL	SUB_CLUSTER

  The status of the node can be checked with gs_sh or gs_admin.

The cluster operating status is determined by the state of each node, and the status may be one of 3 states - IN OPERATION/INTERRUPTED/STOPPED.

Cluster service starts when all the nodes that make up a cluster (number of nodes constituting a cluster) specified by the user during initial system construction have joined the cluster.

During initial cluster construction, the state in which the cluster is waiting to be composed when all the nodes that make up the cluster have not been incorporated into the cluster is known as [INIT_WAIT]. When the number of nodes constituting a cluster has joined the cluster, the state will automatically change to the operating state.

There are 2 operating states. These are [STABLE] and [UNSTABLE].

• [STABLE] state
  • State in which a cluster has been formed by the number of nodes specified in the number of nodes constituting a cluster and service can be provided in a stable manner.
• [UNSTABLE] state
  • State in which the number of nodes constituting a cluster has not been fulfilled.
  • Cluster service will continue for as long as a majority of the number of nodes constituting a cluster is in operation.

A cluster can be operated in an [UNSTABLE] state as long as a majority of the nodes are in operation even if they are detached from a cluster due to maintenance and other reasons.

Cluster service is interrupted automatically in order to prevent a split brain from occurring when the number of nodes making up a cluster falls below the majority of the number of nodes constituting a cluster. The state in which cluster service has been interrupted is known as [WAIT] state.

• What is split brain?

  A split brain is an action where multiple cluster systems performing the same process provide simultaneous service when a system is divided due to a hardware or network failure in a tightly-coupled system that works like a single server interconnecting multiple nodes. If the operation is continued in this state, data saved as replicas in multiple clusters will be treated as master data, resulting in data consistency being lost.

To restart cluster service from the [WAIT] state, new nodes are added to a cluster and nodes with errors are restored. The state will become [STABLE] once the number of nodes constituting a cluster has joined the cluster again.

When the number of clusters constituting a cluster falls below half due to a failure in a node constituting the cluster and the cluster operation is disrupted, new nodes are added to a cluster and nodes with errors are restored. Cluster service is automatically restarted once a majority of the nodes has joined the cluster.

A STABLE state is a state in which the value of the json parameter shown in gs_stat, /cluster/activeCount, is equal to the value of /cluster/designatedCount.

%gs_stat -u admin/admin -s 
{
    "checkpoint": {
        "archiveLog": 0,
          :
          :
    },
    "cluster": {
        "activeCount":4,                     // Nodes in operation within the cluster
        "clusterName": "test-cluster",
        "clusterStatus": "MASTER",
        "designatedCount": 4,                // Number of nodes constituting a cluster
        "loadBalancer": "ACTIVE",
        "master": {
            "address": "192.168.0.1",
            "port": 10040
        },
        "nodeList": [                        // Node list constituting a cluster
            {
                "address": "192.168.0.1",
                "port": 10040
            },
            {
                "address": "192.168.0.2",
                "port": 10040
            },
            {
                "address": "192.168.0.3",
                "port": 10040
            },
            {
                "address": "192.168.0.4",
                "port": 10040
            },
                      :
        ],
                      :
                      :

The status of the cluster can be checked with gs_sh or gs_admin. An example on checking the cluster status with gs_sh is shown below.

% gs_sh
gs> setuser admin admin gsadm                  // Setting connecting user
gs> setnode node1 192.168.0.1 10040            // Definition of a node constituting the cluster
gs> setnode node2 192.168.0.2 10040
gs> setnode node3 192.168.0.3 10040
gs> setnode node4 192.168.0.4 10040
gs> setcluster cluster1 test150 239.0.0.5 31999 $node1 $node2 $node3 $node4 // Definition of cluster
gs> startnode $cluster1                        // Start-up of all nodes making up the cluster
gs> startcluster $cluster1                     // Instructing cluster composition
Waiting for cluster to start.
Cluster has started.
gs> configcluster  $cluster1                   // Checking status of cluster
Name                  : cluster1
ClusterName           : test-cluster
Designated Node Count : 4
Active Node Count     : 4
ClusterStatus         : SERVICE_STABLE         // Stable state

Nodes:
  Name    Role Host:Port              Status
-------------------------------------------------
  node1     M  192.168.0.1:10040    SERVICING
  node2     F  192.168.0.2:10040    SERVICING
  node3     F  192.168.0.3:10040    SERVICING
  node4     F  192.168.0.4:10040    SERVICING

gs> leavecluster $node2
Waiting for node to separate from cluster
Node has separated from cluster.
gs> configcluster  $cluster1
Name                  : cluster1
ClusterName           : test150
Designated Node Count : 4
Active Node Count     : 3
ClusterStatus         : SERVICE_UNSTABLE       // Unstable state

Nodes:
  Name    Role Host:Port              Status
-------------------------------------------------
  node1     M  192.168.0.1:10040    SERVICING  // Master node
  node2     -  192.168.0.2:10040    STARTED
  node3     F  192.168.0.3:10040    SERVICING  // Follower node
  node4     F  192.168.0.4:10040    SERVICING  // Follower node

An OS user has the right to execute operating functions in GridDB and a gsadm user is created during GridDB installation. This OS user is hereinafter referred to gsadm.

All GridDB resources will become the property of gsadm. In addition, all operating commands in GridDB are executed by a gsadm.

A check is conducted to see whether the user has the right to connect to the GridDB server and execute the operating commands. This authentication is performed by a GridDB user.

• Administrator user and general user

  There are 2 types of GridDB user, an administrator user and a general user, which differ in terms of which functions can be used. Immediately after the installation of GridDB, 2 users, a system and an admin user, are registered as default administrator users.

  An administrator user is a user created to perform GridDB operations while general users are users used by the application system.

  For security reasons, administrator users and general users need to be used differently according to the usage purpose.

• Creating a user

  An administrator user can register or delete a gsadm, and the information is saved in the password file of the definition file directory as a GridDB resource. As an administrator user is saved/managed in a local file of the OS, it has to be placed so that the settings are the same in all the nodes constituting the cluster. In addition, administrator users need to be set up prior to starting the GridDB server. After the GridDB server is started, administrative users are not valid even if they are registered.

  A general user can be created after an administrator user starts cluster operations in GridDB. A general user cannot be registered before the start of cluster services. A general user can only be registered using an operating command against a cluster as it is created after a cluster is composed in GridDB and maintained as management information in the GridDB database.

  

  GridDB users

  Since information is not communicated automatically among clusters, an administrator user needs to make the same settings in all the nodes and perform operational management such as determining the master management node of the definition file and distributing information from the master management node to all the nodes that constitute the cluster.

• Rules when creating a user

  There are naming rules to be adopted when creating a user name.

  • Administrator user: Specify a user starting with “gs#”. After “gs#”, the name should be composed of only alphanumeric characters and the underscore mark. Since the name is not case-sensitive, gs#manager and gs#MANAGER cannot be registered at the same time.
  • General user: Specify using alphanumeric characters and the underscore mark. However, the first character cannot be a number. In addition, since the name is not case-sensitive, user and USER cannot be registered at the same time. System and admin users cannot be created as default administrator users.
  • Password: No restrictions on the characters that can be specified.

  A string consisting of up to 64 characters can be specified for the user name and password.

The operations that can be carried out by an administrator and a general user are shown below. Among the operations, commands which can be executed by a gsadm without using a GridDB user are marked with “✓✓”.

Operations	Operating details	Operating tools used	gsadm	Administrator user	General user
Node operations	Starting a node	gs_startnode/gs_sh		✓	✗
	Stopping a node	gs_stopnode/gs_sh		✓	✗
Cluster operations	Building a cluster	gs_joincluster/gs_sh		✓	✗
	Adding a note to a cluster	gs_appendcluster/gs_sh		✓	✗
	Detaching a node from a cluster	gs_leavecluster/gs_sh		✓	✗
	Stopping a cluster	gs_stopcluster/gs_sh		✓	✗
User management	Registering an administrator user	gs_adduser	✓✓	✗	✗
	Deleting an administrator user	gs_deluser	✓✓	✗	✗
	Changing the password of an administrator user	gs_passwd	✓✓	✗	✗
	Creating a general user	gs_sh		✓	✗
	Deleting a general user	gs_sh		✓	✗
	Changing the password of a general user	gs_sh		✓	✓: Individual only
Database management	Creating/deleting a database	gs_sh		✓	✗
	Assigning/cancelling a user in the database	gs_sh		✓	✗
Data operations	Creating/deleting a container or table	gs_sh		✓	✓: Only in the DB of the individual
	Registering data in a container or table	gs_sh		✓	✓: Only in the DB of the individual
	Searching for a container or table	gs_sh		✓	✓: Only in the DB of the individual
	Creating index to a container or table	gs_sh		✓	✓: Only in the DB of the individual
Backup management	Creating a backup	gs_backup		✓	✗
	Restoring a backup	gs_restore	✓✓	✗	✗
	Displaying a backup list	gs_backuplist		✓	✗
System status management	Acquiring system information	gs_stat		✓	✗
	Changing system parameter	gs_paramconf		✓	✗
Data import/export	Importing data	gs_import		✓	✓: Only in accessible object
	Exporting data	gs_export		✓	✓: Only in accessible object

Access to a cluster database in GridDB can be separated on a user basis. The separation unit is known as a database. The following is a cluster database in the initial state.

• public
  • The database can be accessed by all administrator user and general users.
  • This database is used when connected without specifying the database at the connection point.

Multiple databases can be created in a cluster database. Creation of databases and assignment to users are carried out by an administrator user.

The rules for creating a database are as shown below.

• The maximum no. of users and the maximum no. of databases that can be created in a cluster database is 128.
• A string consisting of alphanumeric characters, the underscore mark, the hyphen mark, the dot mark, the slash mark and the equal mark can be specified for the database. However, the first character cannot be a number.
• A string consisting of 64 characters can be specified for the database name.
• Although the case sensitivity of the database name is maintained, a database which has the same name when it is not case-sensitive cannot be created.
• “public” and “information_schema” cannot be specified for default DB.

Only assigned general users and administrator users can access the database. Administrator user can access all databases. The following rules apply when assign a general user to a database.

• Only 1 general user can be assigned to 1 database
• Multiple databases can be assigned to 1 user

There are 2 container (table) data types.

A timeseries container (timeseries table) is a data type which is suitable for managing hourly data together with the occurrence time while a collection (table) is suitable for managing a variety of data.

The schema can be set in a container (table).

The basic data types that can be registered in a container (table) are the basic data type and array data type .

GEOMETRY data are widely used in map information systems, etc. This type can only be specified for container and not be supported for table.

For GEOMETRY, data is written in WKT (Well-known text). WKT is formulated by the Open Geospatial Consortium (OGC), a nonprofit organization promoting standardization of information on geospatial information.

The following WKT format data can be stored in the GEOMETRY column.

• POINT
  • Point represented by two or three-dimensional coordinate.
  • Example) POINT(0 10 10)
• LINESTRING
  • Set of straight lines in two or three-dimensional space represented by two or more points.
  • Example) LINESTRING(0 10 10, 10 10 10, 10 10 0)
• POLYGON
  • Closed area in two or three-dimensional space represented by a set of straight lines.
  • Example) POLYGON((0 0,10 0,10 10,0 10,0 0)), POLYGON((35 10, 45 45, 15 40, 10 20, 35 10),(20 30, 35 35, 30 20, 20 30))
• POLYHEDRALSURFACE
  • Area in the three-dimensional space represented by a set of the specified area.
  • Example) POLYHEDRALSURFACE(((0 0 0, 0 1 0, 1 1 0, 1 0 0, 0 0 0)), ((0 0 0, 0 1 0, 0 1 1, 0 0 1, 0 0 0)), ((0 0 0, 1 0 0, 1 0 1, 0 0 1, 0 0 0)), ((1 1 1, 1 0 1, 0 0 1, 0 1 1, 1 1 1)), ((1 1 1, 1 0 1, 1 0 0, 1 1 0, 1 1 1)), ((1 1 1, 1 1 0, 0 1 0, 0 1 1, 1 1 1)))
• QUADRATICSURFACE
  • Two-dimensional curved surface in a three-dimensional space represented by defining equation f(X) = <AX, X> + BX + c.

Operations using GEOMETRY can be executed with API or TQL.

With TQL, management of two or three-dimensional spatial structure is possible. Generating and judgement function are also provided.

SELECT * WHERE ST_MBRIntersects(geom, ST_GeomFromText('POLYGON((0 0,10 0,10 10,0 10,0 0))'))

See “GridDB API Reference” (GridDB_API_Reference.html) for details of the functions of TQL.

A ROWKEY is the data set in the row of a container. The uniqueness of a row with a set ROWKEY is guaranteed.

In NewSQL I/F, ROWKEY is called as PRIMARY KEY.

A ROWKEY can be set in the first column of the row. (This is set in Column No. 0 since columns start from 0 in GridDB.)

• For a timeseries container (timeseries table)
  • ROWKEY (PRIMARY KEY) is a TIMESTAMP
  • Must be specified.
• For a collection (table)
  • A ROWKEY (PRIMARY KEY) is either a STRING, INTEGER, LONG or TIMESTAMP column.
  • Need not be specified.

A default index prescribed in advance according to the column data type can be set in a column set in ROWKEY (PRIMARY KEY).

In the current version, the default index of all STRING, INTEGER, LONG or TIMESTAMP data that can be specified in a ROWKEY (PRIMARY KEY) is the TREE index.

A condition-based search can be processed quickly by creating an index for the columns of a container (table).

There are 3 types of index - hash index (HASH), tree index (TREE) and space index (SPATIAL). A hash index is used in an equivalent-value search when searching with a query in a container. Besides equivalent-value search, a tree index is used in comparisons including the range (bigger/same, smaller/same etc.).

The index that can be set differs depending on the container (table) type and column data type.

• HASH INDEX
  • An equivalent value search can be conducted quickly but this is not suitable for searches that read the rows sequentially.
  • Columns of the following data type can be set in a collection. Cannot be set in a timeseries container, a table, and a timeseries table.
    • STRING
    • BOOL
    • BYTE
    • SHORT
    • INTEGER
    • LONG
    • FLOAT
    • DOUBLE
    • TIMESTAMP
• TREE INDEX
  • Besides equivalent-value search, a tree index is used in comparisons including the range (bigger/same, smaller/same etc.).
  • This can be used for columns of the following data type in any type of container (table), except for columns corresponding to a rowkey in a timeseries container (timeseries table).
    • STRING
    • BOOL
    • BYTE
    • SHORT
    • INTEGER
    • LONG
    • FLOAT
    • DOUBLE
    • TIMESTAMP
• SPATIAL INDEX
  • Can be used for only GEOMETRY columns in a collection. This is specified when conducting a spatial search at a high speed.

Although there are no restrictions on the no. of indices that can be created in a container, creation of an index needs to be carefully designed. An index is updated when the rows of a configured container are inserted, updated or deleted. Therefore, when multiple indices are created in a column of a row that is updated frequently, this will affect the performance in insertion, update or deletion operations.

An index is created in a column as shown below.

• A column that is frequently searched and sorted
• A column that is frequently used in the condition of the WHERE section of TQL
• High cardinality column (containing few duplicated values)

In order to manage data from a sensor etc. occurring at a high frequency, data is placed in accordance with the data placement algorithm (TDPA: Time Series Data Placement Algorithm) making maximum effective use of the memory . In a timeseries container (timeseries table), memory is allocated while classifying internal data by its periodicity. When hint information is given in an affinity function, the placement efficiency rises further. Expired data in a timeseries container is released at almost zero cost while being expelled to a disk where necessary.

A timeseries container (timeseries table) has a TIMESTAMP ROWKEY (PRIMARY KEY).

Time data may deviate slightly from the expected time due to the timing of the collection and the contents of the data to be collected. Therefore when conducting a search using time data as a key, a function that allows data around the specified time to be acquired is also required.

The functions for searching the timeseries container (timeseries table) and acquiring the specified row are as follows:

• TIME_NEXT(*, timestamp)

  Selects a time-series row whose timestamp is identical with or just after the specified timestamp.

• TIME_NEXT_ONLY(*, timestamp)

  Select a time-series row whose timestamp is just after the specified timestamp.

• TIME_PREV(*, timestamp)

  Selects a time-series row whose timestamp is identical with or just before the specified timestamp.

• TIME_PREV_ONLY(*, timestamp)

  Selects a time-series row whose timestamp is just before the specified timestamp.

In addition, the functions for interpolating the values of the columns are as follows:

• TIME_INTERPOLATED(column, timestamp)

  Returns a specified column value of the time-series row whose timestamp is identical with the specified timestamp, or a value obtained by linearly interpolating specified column values of adjacent rows whose timestamps are just before and after the specified timestamp, respectively.

• TIME_SAMPLING(*|column, timestamp_start, timestamp_end, interval, DAY|HOUR|MINUTE|SECOND|MILLISECOND)

  Takes a sampling of Rows in a specific range from a given start time to a given end time.

  Each sampling time point is defined by adding a sampling interval multiplied by a non-negative integer to the start time, excluding the time points later than the end time.

  If there is a Row whose timestamp is identical with each sampling time point, the values of the Row are used. Otherwise, interpolated values are used.

An affinity is a function to connect related data. There are 2 types of affinity function in GridDB, data affinity and node affinity.

In order to improve the operation speed of applications connected to multiple nodes of the GridDB cluster, it is important to arrange the data to be processed in memory as much as possible. For huge table with a large number of rows, by distributing rows of the table to multiple nodes, processors and memory of multiple nodes can be effectively used. Distributed rows are stored in the internal containers called "data partition". The allocation of each row to the data partition is determined by a "partitioning key" column specified at the time of the table creation.

GridDB supports hash partitioning, interval partitioning and interval-hash partitioning as table partitioning methods.

Table partitioning is a function only for GridDB AE/VE. Creating and Deleting tables can be performed only through the NewSQL interface. Data registration, update and search can be performed through the NewSQL/NoSQL interface. (There are some restrictions. See TQL and SQL for the details.)

• Data registration

  When data is registered into a table, the data is stored in the appropriate data partition according to the partitioning key value and the partitioning method. It is not possible to specify a data partition to be stored.

• Index

  When creating an index on a table, a local index for each data partition is created. It is not possible to create a global index for the whole table.

• Data handling

  An error occurs for updating the partitioning key value. If updating the partitioning key is needed, delete and reregister the data.

• Functions of timeseries tables

  The expiry release function can be used for partitioned timeseries tables. The compression function cannot be used for the tables.

• Notes

  When specifying the column as a partitioning key other than the primary key, the primary key constraint is ensured in each data partition, but it is not ensured in the whole table. So, the same value may be registered in multiple rows of a table.

Containers (tables) and partitioned tables in a GridDB cluster are automatically distributed to each node. By using operation management tools or SQL, it is possible to check which container (table) is placed on each node.

This function is used to:

• check containers placed on a node when the database size of each node is not balanced.
• find backup location of the node where specified container is placed on.

[Memo]

• See the "Data model" for the description of the container and partition.
• When the autonomous data placement is executed by a node down or a node failure, the placement of containers may be changed. The placement of containers is not persistent.

The placement information of containers (tables) is checked by the following methods.

When GridDB writes in-memory data to the database file residing on the disk, a database with larger capacity independent to the memory size can be obtained. However, as the size increases, so does the cost of the storage. To reduce the cost, the database file (checkpoint file) can be effectively compressed using GridDB's block data compression. In this case, flash memory with a higher price per unit of capacity can be utilized much more efficiently than HDD.

The deallocation of unused data blocks is the function that reduces the size (disk space) of database files by the Linux file block deallocation processing on unused block areas of database files (checkpoint files).

Use this function in the following cases.

• A large amount of data has been deleted
• There is no plan to update data and it is necessary to keep the DB for a long term.
• The disk becomes full when updating data and reducing the DB size is needed temporarily.

The processing for the deallocation of unused blocks, the support environment and the execution method are explained below.

When a row search or update etc. is carried out on a container, a new transaction is started and this transaction ends when the update results of the data are committed or aborted.

[Memo]

• A commit is a process to confirm transaction information under processing to perpetuate the data.
  • In GridDB, updated data of a transaction is stored as a transaction log by a commit process, and the lock that had been maintained will be released.
• An abort is a process to rollback (delete) all transaction data under processing.
  • In GridDB, all data under processing are discarded and retained locks will also be released.

The initial action of a transaction is set in autocommit.

In autocommit, a new transaction is started every time a container is updated (data addition, deletion or revision) by the application, and this is automatically committed at the end of the operation. A transaction can be committed or aborted at the requested timing by the application by turning off autocommit.

A transaction recycle may terminate in an error due to a timeout in addition to being completed through a commit or abort. If a transaction terminates in an error due to a timeout, the transaction is aborted. The transaction timeout is the elapsed time from the start of the transaction. Although the initial value of the transaction timeout time is set in the definition file (gs_node.json), it can also be specified as a parameter when connecting to GridDB on an application basis.

There are 2 types of transaction consistency levels, immediate consistency and eventual consistency. This can also be specified as a parameter when connecting to GridDB for each application. The default setting is immediate consistency.

• Immediate consistency: Container update results from other clients are reflected immediately at the end of the transaction concerned. As a result, the latest details can be referenced all the time.
• Eventual consistency: Container update results from other clients may not be reflected immediately at the end of the transaction concerned. As a result, there is a possibility that old details may be referred to.

Immediate consistency is valid in update operations and read operations. Eventual consistency is valid in read operations only. For applications which do not require the latest results to be read all the time, the reading performance improves when eventual consistency is specified.

Conformity of the database contents need to be maintained all the time. When executing multiple transaction simultaneously, the following events will generally surface as issues.

• Dirty read

  An event which involves uncommitted data written by a dirty read transaction being read by another transaction.

• Non-repeatable read

  An event which involves data read previously by a non-repeatable read transaction becoming unreadable. Even if you try to read the data read previously by a transaction again, the previous data can no longer be read as the data has already been updated and committed by another transaction (the new data after the update will be read instead).

• Phantom read

  An event in which the inquiry results obtained previously by a phantom read transaction can no longer be acquired. Even if you try to execute an inquiry executed previously in a transaction again in the same condition, the previous results can no longer be acquired as the data satisfying the inquiry condition has already been changed, added and committed by another transaction (new data after the update will be acquired instead).

In GridDB, “READ_COMMITTED” is supported as a transaction isolation level. In READ_COMMITTED, the latest data confirmed data will always be read.

When executing a transaction, this needs to be taken into consideration so that the results are not affected by other transactions. The isolation level is an indicator from 1 to 4 that shows how isolated the executed transaction is from other transactions (the extent that consistency can be maintained).

The 4 isolation levels and the corresponding possibility of an event raised as an issue occurring during simultaneous execution are as follows.

Isolation level	Dirty read	Non-repeatable read	Phantom read
READ_UNCOMMITTED	Possibility of occurrence	Possibility of occurrence	Possibility of occurrence
READ_COMMITTED	Safe	Possibility of occurrence	Possibility of occurrence
REPEATABLE_READ	Safe	Safe	Possibility of occurrence
SERIALIZABLE	Safe	Safe	Safe

In READ_COMMITED, if data read previously is read again, data that is different from the previous data may be acquired, and if an inquiry is executed again, different results may be acquired even if you execute the inquiry with the same search condition. This is because the data has already been updated and committed by another transaction after the previous read.

In GridDB, data that is being updated by MVCC is isolated.

In order to realize READ_COMMITTED, “MVCC (Multi-Version Concurrency Control)” has been adopted.

MVCC is a processing method that refers to the data prior to being updated instead of the latest data that is being updated by another transaction when a transaction sends an inquiry to the database. System throughput improves as the transaction can be executed concurrently by referring to the data prior to the update.

When the transaction process under execution is committed, other transactions can also refer to the latest data.

There is a data lock mechanism to maintain the consistency when there are competing container update requests from multiple transactions.

The lock granularity differs depending on the type of container. In addition, the lock range changes depending on the type of operation in the database.

Data registered or updated in a container or table is perpetuated in the disk or SSD, and protected from data loss when a node failure occurs. There are 2 types of transaction log process, one to synchronize data in a data update and write the updated data sequentially in a transaction log file, and the other is a checkpoint process to store updated data in the memory regularly in the database file on a block basis.

To write to a transaction log, either one of the following settings can be made in the node definition file.

• 0: SYNC
• An integer value of 1 or higher: DELAYED_SYNC

In the "SYNC" mode, log writing is carried out synchronously every time an update transaction is committed or aborted. In the "DELAYED_SYNC" mode, log writing during an update is carried out at a specified delay of several seconds regardless of the update timing. Default value is "1 (DELAYED_SYNC 1 sec)".

When "SYNC" is specified, although the possibility of losing the latest update details when a node failure occurs is lower, the performance is affected in systems that are updated frequently.

On the other hand, if “DELAYED_SYNC" is specified, although the update performance improves, any update details that have not been written in the disk when a node failure occurs will be lost.

If there are 2 or more replicas in a raster configuration, the possibility of losing the latest update details when a node failure occurs is lower even if the mode is set to "DELAYED_SYNC" as the other nodes contain replicas. Consider setting the mode to "DELAYED_SYNC" as well if the update frequency is high and performance is required.

In a checkpoint, the update block is updated in the database file. A checkpoint process operates at the cycle set on a node basis. A checkpoint cycle is set by the parameters in the node definition file. Initial value is 60 sec (1 minute).

By raising the checkpoint execution cycle figure, data perpetuation can be set to be carried out in a time band when there is relatively more time to do so e.g. by perpetuating data to a disk at night and so on. On the other hand, when the cycle is lengthened, the disadvantage is that the number of transaction log files that have to be rolled forward when a node is restarted outside the system process increases, thereby increasing the recovery time.

The timeout details that can be set differ between a NoSQL I/F and a NewSQL I/F.

Data replicas are created on a partition basis in accordance with the number of replications set by the user among multiple nodes constituting a cluster.

A process can be continued non-stop even when a node failure occurs by maintaining replicas of the data among scattered nodes. In the client API, when a node failure is detected, the client automatically switches access to another node where the replica is maintained.

The default number of replication is 2, allowing data to be replicated twice when operating in a cluster configuration with multiple nodes.

When there is an update in a container, the owner node (the node having the master replica) among the replicated partitions is updated.

There are 2 ways of subsequently reflecting the updated details from the owner node in the backup node.

• Asynchronous replication

  Replication is carried out without synchronizing with the timing of the asynchronous replication update process. Update performance is better for quasi-synchronous replication but the availability is worse.

• Quasi-synchronous replication

  Although replication is carried out synchronously at the quasi-synchronous replication update process timing, no appointment is made at the end of the replication. Availability is excellent but performance is inferior.

If performance is more important than availability, set the mode to asynchronous replication and if availability is more important, set it to quasi-synchronous replication.

[Memo]

The number of replications is set in the cluster definition file (gs_cluster.json) /cluster/replicationNum. Synchronous settings of the replication are set in the cluster definition file (gs_cluster.json) /transaction/replicationMode.

An overview of the failures which occur and the treatment method is shown in the table below.

A node failure refers to a situation in which a node has stopped due to a processor failure or an error in a GridDB server process, while a database failure refers to a situation in which an error has occurred in accessing a database placed in a disk.

Configuration of GridDB	Type of failure	Action and treatment
Single configuration	Node failure	Although access from the application is no longer possible, data in a transaction which has completed processing can be recovered simply by restarting the transaction, except when caused by a node failure. Recovery by another node is considered when the node failure is prolonged.
Single configuration	Database failure	The database file is recovered from the backup data in order to detect an error in the application. Recovered at the backup point.
Cluster configuration	Single node failure	The error is covered up in the application, and the process can continue in nodes with replicas. Recovery operation is not necessary in a node where a failure has occurred.
Cluster configuration	Multiple node failure	If both owner/backup partitions of a replica exist in a failure target node , the cluster will operate normally even though the subject partitions cannot be accessed. Except when caused by a node failure, data in a transaction which has completed processing can be recovered simply by restarting the transaction. Recovery by another node is considered when the node failure is prolonged.
Cluster configuration	Single database failure	Since data access will continue through another node constituting the cluster when there is a database failure in a single node, the data can be recovered simply by changing the database deployment location to a different disk, and then starting the node again.
Cluster configuration	Multiple database failure	A partition that cannot be recovered in a replica needs to be recovered at the point backup data is sampled from the latest backup data.

If a node failure occurs when operating in a cluster configuration, the partitions (containers) placed in the failure node cannot be accessed. At this point, a client failover function to automatically connect to the backup node again and continue the process is activated in the client API. To automatically perform a failover countermeasure in the client API, the application developer does not need to be aware of the error process in the node.

However, due to a network failure or simultaneous failure of multiple nodes, an error may also occur and access to the target application operations may not be possible.

Depending on the data to be accessed, the following points need to be considered in the recovery process after an error occurs.

• For a collection in which the timeseries container or row key is defined, the data can be recovered by executing the failed operation or transaction again.
• For a collection in which the row key is not defined, the failed operation or transaction needs to be executed again after checking the contents of the DB.

[Memo]

In order to simplify the error process in an application, it is recommended that the row key be defined when using a collection. If the data cannot be uniquely identified by a single column value but can be uniquely identified by multiple column values, a column having a value that links the values of the multiple columns is recommended to be set as the row key so that the data can be uniquely identified.

It is necessary to collect periodic backups to prepare for data corruption due to database failure or application malfunction. The backup operation method should be selected according to the service level requirements and system resources.

The types of online backup provided by GridDB are as follows. The recovery point varies depending on the type of backup.

Backup type	Backup actions	Recovery point
Full backup	A backup of the cluster database currently in use is stored online in node units in the backup directory specified in the node definition file.	Full backup collection point
Differential/incremental backup	A backup of the cluster database currently in use is stored online in node units in the backup directory specified in the node definition file. In subsequent backups, only the difference in the update block after the backup is backed up.	Differential/incremental backup collection point
Automatic log backup	In addition to backing up the cluster database currently in use which is stored online in node units in the backup directory specified in the node definition file, the transaction log is also automatically picked up at the same timing as the transaction log file writing. The write timing of the transaction log file follows the value of /dataStore/logWriteMode in the node definition file.	Latest transaction update point

See “GridDB backup guide” (GridDB_BackupGuide.html) for the backup details and the recovery processes.

An event log is a log to record system operating information and messages related to event information e.g. exceptions which occurred internally in a GridDB node etc.

An event log is created with the file name gridstore-%Y%m%d-n.log in the directory shown in the environmental variable GS_LOG (Example: gridstore-20150328-5.log). 22/5000 The file switches at the following timing:

• When the log is written first after the date changes
• When the node is restarted
• When the size of one file exceeds 1MB

The default value of the maximum number of event log files is 30. If it exceeds 30 files, it will be deleted from the old file. The maximum number can be changed with the node definition file.

Output format of event log is as follows.

• (Date and time) (host name) (thread no.) (log level) (category) [(error trace no.): (error trace no. and name)] (message) < (base64 detailed information: Detailed information for problem analysis in the support service)>

  An overview of the event which occurred can be found in the error trace no. and name. In addition, measures to deal with the problems can be searched using the error trace no. in the troubleshooting guide. A output example of an event log is shown below.

2014-11-12T10:35:29.746+0900 TSOL1234 8456 ERROR TRANSACTION_SERVICE [10008:TXN_CLUSTER_NOT_SERVICING] (nd={clientId=2, address=127.0.0.1:52719}, pId=0, eventType=CONNECT, stmtId=1) <Z3JpZF9zdG9yZS9zZXJ2ZXIvdHJhbnNhY3Rpb25fc2VydmljZS5jcHAgQ29ubmVjdEhhbmRsZXI6OmhhbmRsZUVycm9yIGxpbmU9MTg2MSA6IGJ5IERlbnlFeGNlcHRpb24gZ3JpZF9zdG9yZS9zZXJ2ZXIvdHJhbnNhY3Rpb25fc2VydmljZS5jcHAgU3RhdGVtZW50SGFuZGxlcjo6Y2hlY2tFeGVjdXRhYmxlIGxpbmU9NjExIGNvZGU9MTAwMDg=>

The event log output level can be changed online by using the gs_logconf command. When analyzing details of trouble information, change it online. However, online changes are temporary memory changes. Therefore, in order to make it permanent such as setting valid at restart of the node, it is necessary to change the trace item of the node definition file of each node constituting the cluster.

The current setting can be displayed with the gs_logconf command.

$ gs_logconf -u admin/admin
{
    "levels": {
        "CHECKPOINT_FILE": "ERROR",
        "CHECKPOINT_SERVICE": "INFO",
        "CHUNK_MANAGER": "ERROR",
        "CHUNK_MANAGER_IODETAIL": "ERROR",
        "CLUSTER_OPERATION": "INFO",
        "CLUSTER_SERVICE": "ERROR",
        "COLLECTION": "ERROR",
        "DATA_STORE": "ERROR",
        "DEFAULT": "ERROR",
        "EVENT_ENGINE": "WARNING",
        "IO_MONITOR": "WARNING",
        "LOG_MANAGER": "WARNING",
        "MAIN": "WARNING",
        "MESSAGE_LOG_TEST": "ERROR",
        "OBJECT_MANAGER": "ERROR",
        "RECOVERY_MANAGER": "INFO",
        "REPLICATION_TIMEOUT": "WARNING",
        "SESSION_TIMEOUT": "WARNING",
        "SYNC_SERVICE": "ERROR",
        "SYSTEM": "UNKNOWN",
        "SYSTEM_SERVICE": "INFO",
        "TIME_SERIES": "ERROR",
        "TRANSACTION_MANAGER": "ERROR",
        "TRANSACTION_SERVICE": "ERROR",
        "TRANSACTION_TIMEOUT": "WARNING",
        "TRIGGER_SERVICE": "ERROR"
    }
}

TQL and SQL-92 compliant SQL (GridDB AE/VE only) are supported as database access languages.

• What is TQL?

  A simplified SQL prepared for GridDB SE. The support range is limited to functions such as search, aggregation, etc., using a container as a unit. TQL is employed by using the client API (Java, C language) of GridDB SE.

  The TQL is adequate for the search in the case of a small container and a small number of hits. For that case, the response is faster than SQL. The number of hits can be suppressed by the LIMIT clause of TQL.

  For the search of a large amount of data, SQL is recommended.

  TQL is available for the containers and paritioned tables created by operations through the NewSQL interface. The followings are the limitations of TQL for the partitioned tables.

  • Filtering data by the WHERE clause is available. But aggregate functions, timeseries data selection or interpolation, min or max function and ORDER BY clause, etc. are not available.
  • It is not possible to apply the update lock.
• What is SQL?

  Standardization of the language specifications is carried out in ISO to support the interface for defining and performing data operations in conformance with SQL-92 in GridDB. SQL can be used in NewSQL I/F.

  SQL is also available for the containers created by operations through the NoSQL interface.

See “GridDB API Reference” (GridDB_API_Reference.html) for details on TQL, and “GridDB AE SQL Reference” (GridDB_AE_SQL_Reference.pdf) for details on SQL.

An interface to quickly process event information that occurs occasionally is available in NoSQL I/F.

When a large volume of events is sent to the database server every time an event occurs, the load on the network increases and system throughput does not increase. Significant impact will appear especially when the communication line bandwidth is narrow. Multi-processing is available in NoSQL I/F to process multiple row registrations for multiple containers and multiple inquiries (TQL) to multiple containers with a single request. The overall throughput of the system rises as the database server is not accessed frequently.

An example is given below.

• Multiput

  A container is prepared for each sensor name as a process to register event information from multiple sensors in the database. The sensor name and row array of the timeseries event of the sensor are created and a list (map) summarizing the data for multiple sensors is created. This list data is registered in the GridDB database each time the API is invoked.

  In the API of a multi- registration process, the communication process is optimized by consolidating requests for 1 or more containers to a node in GridDB formed by multiple clusters. In addition, multi-registrations are processed quickly without performing MVCC when executing a transaction.

  In a multiput, transactions are committed automatically. Data is confirmed on a single case basis.

  

  Multiput

• Multi-query (fetchAll)

  Instead of executing multiple queries to a sensor, these can be executed in a single query by consolidating event information of the sensor. For example, this is most suitable for acquiring aggregate results such as the daily maximum, minimum and average values of data acquired from a sensor, or data of a row set having the maximum or minimum value, or data of a row set meeting the specified condition.

  

  fetchAll

• Multiget

  Instead of executing multiple queries to a sensor, these can be executed in a single query by consolidating event information of the sensor. For example, this is most suitable for acquiring aggregate results such as the daily maximum, minimum and average values of data acquired from a sensor, or data of a row set having the maximum or minimum value, or data of a row set meeting the specified condition.

  In a RowKeyPredicate object, the acquisition condition is set in either one of the 2 formats below.

  • Specify the acquisition range
  • Specified individual value

  

  Multiget

GridDB service is automatically performed during OS start-up to start a node or cluster.

GridDB service is enabled after installing with RPM. Since the service is enabled, the GridDB server is started at the same time the OS starts up, and the server is stopped when the OS is stopped.

When you use an interface that integrates middleware and application operation including OS monitoring and database software operation, consideration of dependency with other middleware such as whether to use service or operating commands for GridDB operation is necessary.

See "GridDB operation control guide" (GridDB_OperationGuide.html) for parameters of the service.

If service is not used, disable the service as follows:

# /sbin/chkconfig gridstore off

The following commands are available in GridDB.

Type	Command	Function
Start/stop node	gs_startnode	Start a node
	gs_stopnode	Stop a node
Cluster management	gs_joincluster	Join to cluster configuration
	gs_leavecluster	Leave from cluster configuration
	gs_stopcluster	Stop a cluster
	gs_config	Get cluster configuration data
	gs_stat	Get cluster data
	gs_appendcluster	Add a node to a cluster
	gs_failovercluster	Do manual failover of a cluster
	gs_partition	Get partition data
	gs_loadbalance	Set autonomous data redistribution
User management	gs_adduser	Add an administrator user
	gs_deluser	Delete an administrator user
	gs_passwd	Change a password of an administrator user
Log data	gs_logs	Display recent event logs
	gs_logconf	Display and change the event log output level
Backup/restoration	gs_backup	Collect backup data
	gs_backuplist	Display backup data list
	gs_restore	Restore a backup data
Import/export	gs_import	Import exported containers and database on the disk
	gs_export	Export containers and database as CSV or ZIP format to the disk
Maintenance	gs_paramconf	Display and change parameters

The integrated operation control GUI (hereinafter referred to gs_admin) is a Web application that integrates GridDB cluster operation functions. Gs_admin has an intuitive UI, it is possible to grasp the operation information of the cluster on one screen (dashboard screen), start and stop operation to individual nodes constituting the cluster, check performance information, etc.

Gs_admin also supports the following functions to support development, so it can be used effectively in the development stage of the system.

• Create and drop database, manage general user
• Create, drop and search container
• Create and drop index and trigger
• Execute TQL/SQL statement to container

The cluster operation control command interpreter (hereinafter referred to gs_sh) is a command line interface tool to manage GridDB cluster operations and data operations. While operating commands provide operation on a per-node basis, gs_sh provides interfaces for processing on a per-cluster basis. In addition to user management operations, it also provides data manipulation such as creating databases, containers and tables, and searching by TQL or SQL.

There are two types of start modes in gs_sh.

• Interactive mode: specify subcommand interactively to execute processing
• Batch mode: Execute a script file containing a series of operations with subcommands

Use of batch script enables automation of operation verification at development and labor saving of system construction

Gs_sh has its own variable definition function, and operations on nodes and clusters can also be executed with instructions to the operation target whose variables are defined in advance.

// Interactive mode: start gs_sh and execute subcommand "version"
$ gs_sh
gs> version

// Batch mode: execute a script file specified as an argument
$ gs_sh test.gsh

The following subcommands are available in gs_sh.

Type	Subcommand	Function
Start/stop node	startnode	Start nodes. Starting all nodes of the cluster and waiting for startup completion are possible.
	stopnode	Stop nodes. Stopping all nodes of the cluster is possible.
Cluster management	joincluster	Attach a node individually to a cluster.
	leavecluster	Detach a node individually from a cluster.
	stopcluster	Detach all of the currently attached nodes from a cluster, together at once.
	config	Display the cluster configuration data.
	configcluster	Display the cluster status data.
	appendcluster	Add an undefined node to a pre-defined cluster.
Connection	connect	Connect to a GridDB cluster. For GridDB AE/VE, connect with both NoSQL I/F and NewSQL I/F, and can be used both TQL and SQL.
	disconnect	Disconnect from a GridDB cluster.
Database management	createdatabase	Create a database.
	dropdatabase	Delete a database.
	getcurrentdatabase	Display the current database name.
	showdatabase	Display the database list and access rights data.
General user management	createuser	Create a general user.
	dropuser	Delete a general user.
	setpassword	Change the password of a general user.
	showuser	Display the user data.
Access control	grant	Grant access rights for a user to a database.
	revoke	Revoke access rights for a user from a database.
Container operation	createcontainer	Create a container from the container data file.
	createcollection	Create a collection of simple schema.
	createtimeseries	Create a timeseries container of simple schema.
	dropcontainer	Delete a container.
	showcontainer	Display the container data.
Table operation (GridDB AE/VE)	showtable	Display the table data.
Index operation	createindex	Create an index in the specified column.
	dropindex	Delete an index in the specified column.
Trigger operation	droptrigger	Delete a trigger from a container.
	showtrigger	Show trigger data of a container.
TQL/SQL	tql	Execute a TQL and retain the search results from a container.
	get/getnoprint	Get the search results.
	getcsv	Get the search results and save them in a file in the CSV format.
	tqlexplain	Execute the specified TQL and display the execution plan and actual measurement values such as the number of cases processed etc.
	sql (GridDB AE/VE)	Execute an SQL and retain the search results from a table or container.
Log data	logs	The following command displays the log of the specified node.
	logconf	Display and change the log settings.

GridDB performance and statistical information can be checked in GridDB using the operating command gs_stat. gs_stat represents information common in the cluster and performance and statistical information unique to the nodes.

Among the outputs of the gs_stat command, the performance structure is an output that is related to the performance and statistical information.

-bash-4.1$ gs_stat -u admin/admin -s 192.168.0.1:10040
{
    ：
    "performance": {
        "batchFree": 0,
        "checkpointFileSize": 65536,
        "checkpointFileUsageRate": 0,
        "checkpointMemory": 2031616,
        "checkpointMemoryLimit": 1073741824,
        "checkpointWriteSize": 0,
        "checkpointWriteTime": 0,
        "currentTime": 1428024628904,
        "numConnection": 0,
        "numTxn": 0,
        "peakProcessMemory": 42270720,
        "processMemory": 42270720,
        "recoveryReadSize": 65536,
        "recoveryReadTime": 0,
        "storeDetail": {
            "batchFreeMapData": {
                "storeMemory": 0,
                "storeUse": 0,
                "swapRead": 0,
                "swapWrite": 0
            },
            "batchFreeRowData": {
                "storeMemory": 0,
                "storeUse": 0,
                "swapRead": 0,
                "swapWrite": 0
            },
            "mapData": {
                "storeMemory": 0,
                "storeUse": 0,
                "swapRead": 0,
                "swapWrite": 0
            },
            "metaData": {
                "storeMemory": 0,
                "storeUse": 0,
                "swapRead": 0,
                "swapWrite": 0
            },
            "rowData": {
                "storeMemory": 0,
                "storeUse": 0,
                "swapRead": 0,
                "swapWrite": 0
            }
        },
        "storeMemory": 0,
        "storeMemoryLimit": 1073741824,
        "storeTotalUse": 0,
        "swapRead": 0,
        "swapReadSize": 0,
        "swapReadTime": 0,
        "swapWrite": 0,
        "swapWriteSize": 0,
        "swapWriteTime": 0,
        "syncReadSize": 0,
        "syncReadTime": 0,
        "totalLockConflictCount": 0,
        "totalReadOperation": 0,
        "totalRowRead": 0,
        "totalRowWrite": 0,
        "totalWriteOperation": 0
    },
    ：
}

Information related to performance and statistical information is explained below. The description of the storeDetail structure is omitted as this is internal debugging information.

• The type is shown below.
  • CC: Current value of all cluster
  • c: Current value of specified node
  • CS: Cumulative value after service starts for all clusters
  • s: Cumulative value after service starts for all nodes
  • CP: Peak value after service starts for all clusters
  • p: Peak value after service starts for all nodes
• Check the event figure to be monitored, and show the items that ought to be reviewed in continuing with operations.

  Output parameters	Type	Description	Event to be monitored
  checkpointFileSize	c	Checkpoint file size (byte)
  checkpointFileUsageRate	c	Checkpoint file usage rate
  checkpointMemory	c	Checkpoint memory size for checkpoint use (byte)
  checkpointMemoryLimit	c	CheckpointMemoryLimit setting for checkpoint use (byte)
  checkpointWriteSize	s	Checkpoint process CP file write size (byte)
  checkpointWriteTime	s	Checkpoint process CP file write time
  checkpointFileAllocateSize	c	Total size of allocated blocks in checkpoint files (byte)
  currentTime	c	Current time
  numConnection	c	Current no. of connections. Number of connections used in the transaction process, not including the number of connections used in the cluster process. Value is equal to the no. of clients + no. of replicas \* no. of partitions retained.	If the no. of connections is insufficient in monitoring the log, review the connectionLimit value of the node configuration.
  numSession	c	Current no. of sessions
  numTxn	c	Current no. of transactions
  peakProcessMemory	p	Peak value of the memory used in the GridDB server, including the storememory value which is the maximum memory size (byte) used in the process	If the peakProcessMemory or processMemory is larger than the installed memory of the node and an OS Swap occurs, additional memory or a temporary drop in the value of the storeMemoryLimit needs to be considered.
  processMemory	c	Memory space used by a process (byte)
  recoveryReadSize	s	Checkpoint file size read by the recovery process (byte)
  recoveryReadTime	s	Checkpoint file time read by the recovery process (byte)
  storeMemory	c	Memory space used in an in-memory database (byte)
  storeMemoryLimit	c	Memory space limit used in an in-memory database (byte)
  storeTotalUse	c	Full data capacity (byte) retained by the nodes, including the data capacity in the database file
  swapRead	s	Swap read count
  swapReadSize	s	Swap process file read size (byte)
  swapReadTime	s	Swap process file read time
  swapWrite	s	Swap write count
  swapWriteSize	s	Swap process file write size (byte)
  swapWriteTime	s	Swap process file write time
  syncReadSize	s	Synchronization process CP file read size (byte)
  syncReadTime	s	Synchronization process CP file read time
  totalLockConflictCount	s	Row lock competing count
  totalReadOperation	s	Search process count
  totalRowRead	s	Insert and update process count
  totalRowWrite	s	Row reading count
  totalWriteOperation	s	Row writing count

The upgrade of nodes while the cluster is running is possible by the rolling upgrade. By operating one by one to leave a node from the cluster, upgrading GridDB on the node and join the node to the cluster again, GridDB on all nodes are replaced to a newer version.

1. Make a plan for the operations of rolling upgrade in advance

   Estimate the time of the operations. The operations for a node are as follows. Estimate the time of the following operations and calculate the time for all the nodes. The estimated time is a few minutes for the operations other than the start-up of a node (recovery).

   • Leave cluster
   • Stop node
   • Installation of GridDB
   • Start-up node (recovery)
   • Join cluster

   When there are many data updates before leaving the cluster or during the rolling upgrade, the recovery may take longer than usual.

2. Disable autonomous data redistribution

   In the rolling upgrade, after each node leaves the cluster, it rejoins the cluster soon. By disabling autonomous data redistribution, redundant redistributions can be eliminated and the load of the processing and network communication can be reduced.

   By executing the gs_loadbalance command with the --cluster option, the autonomous data redistribution on all the nodes of the cluster is disabled.

   Example)

   $ gs_loadbalance -u admin/admin --off --cluster
   

3. Upgrade the nodes other than the master node

   Perform the following operations on each node. Login the node and do the operations. From starting these operations and until finishing step 5, the operations of SQL cause an error.

   a. Check the role of each node

   Execute the gs_stat command. Do the step b. and the subsequence operations on the nodes which are display '"clusterStatus": "FOLLOWER"'.

   Example)

   $ gs_stat -u admin/admin
   ・・・
   "clusterStatus": "FOLLOWER"
   

   b. Leave the cluster (gs_leavecluster)

   Example)

   $ gs_leavecluster -u admin/admin --force
   

   c. Stop the node (gs_stopnode)

   Example)

   $ gs_stopnode -u admin/admin
   

   d. Upgrade GridDB

   The process of upgrading is different for each of the version. See the quick start guide about the operations.

   e. Start-up the node

   Example)

   $ gs_startnode -u admin/admin
   

   f. Disable autonomous data redistribution

   The --cluster option is not needed because of the operation on single node.

   Example)

   $ gs_loadbalance -u admin/admin --off
   

   g. Join the node to the cluster (gs_join)

   Example) Cluster name: mycluster, The number of nodes in the cluster: 5

   $ gs_joincluster -u admin/admin -c mycluster -n 5
   

4. Upgrade the master node

   a. Check the role of each node

   Execute the gs_stat command. Do the step b. and the subsequence operations on the nodes which are display '"clusterStatus": "MASTER"'.

   Example)

   $ gs_stat -u admin/admin
   ・・・
   "clusterStatus": "MASTER"
   

   b. Leave the cluster (gs_leavecluster) The cluster stops temporarily (for about 1 minutes) by this operation

   c. Stop the node (gs_stopnode)

   d. Upgrade GridDB

   e. Start-up the node

   f. Disable autonomous data redistribution

   g. Join the node to the cluster (gs_joincluster)

1. Check that all nodes are the new version (gs_stat)
2. Enable autonomous data redistribution on all nodes in the cluster (gs_loadbalance)

   $ gs_loadbalance -u admin/admin --on --cluster
   

[Memo]

• The rolling upgrade can be used for version 4.0 or later.
• The rolling upgrade can not be performed when the current major version and the replaced major version of the cluster are different.

  Example) When the current version is V4.0 and the version to be replaced is V5.0, the rolling upgrade cannot be performed because the major versions are different.

• The replaced version is required to be newer than the current version.

[Notes]

• When different versions are mixed in a cluster by the rolling upgrade, an error may occur in a SQL search. The limitations will be explained in each version's Readme file, so please see the Readme file for the limitations. (The SQL search is retried repeatedly in the failover time. In the meantime, if the rolling upgrade is finished, the error does not occur on the client.)
• When the master node is replaced, the cluster stops temporarily (about 1 minute). For the processing that can be failed over (NoSQL interface and SQL search), if the cluster restarts in the failover time, no error occurs on the client.
• When there are many data updates during a rolling upgrade, the synchronization of data takes time. It is recommended to perform the rolling upgrade during the hours when there are fewer data updates.