GridDB backup guide

In GridDB, backup of node units can be carried out online. A backup of the entire cluster can be carried out online while maintaining the services by performing a backup of all the nodes constituting the GridDB cluster in sequence. The types of online backup provided by GridDB are as follows.

The recovery point differs depending on the type of backup used.

The various backup operations and systems recommendation provided by GridDB are shown below.

• Full backup

  A full backup is acquired after an overnight batch process to update data in a reference system. A full backup will take a while to complete as data in all the database files will be copied. In addition, the data capacity at the backup collection destination needs to be the same as the database file.

  The backup disk capacity needs to be multiplied by the actual database size according to how many backup generations to retain.

• Differential/incremental backup

  In differential/incremental backup, once a full backup of the entire databases have been performed, only the differences between the current and the updated data will be backed up. This type of backup is suitable for systems which needs to be backed up quickly, and systems which perform systematic operations such as automatic backup of batch operations during the night, monthly full backup (baseline creation), differential (since) backup once a week, incremental backup (incremental) once a day, etc. As an incremental backup uses only updated data, it can be carried out faster compared to a full backup or differential backup. However, recovery when a failure occurs may take a while as the update block needs to be rolled forward for the data of the full backup. A differential backup using a regular Baseline or Since command is required.

• Automatic log backup

  After performing a full backup with an automatic log backup command (baseline creation), the updated log will be collected in the backup directory. Backup operation is not required as a transaction log will be taken automatically. This command is used when you want to simplify the operation or when you do not wish to impose a load on the system due to the backup. However, if the baseline is not updated regularly, recovery will take a while as the number of transaction log files used in recovery when a failure occurs increases. In a differential/incremental backup, data of the same block will be backed up as a single data when updated, but in an automatic log backup, a log is recorded every time there is an update, so recovery during a failure takes more time than a differential/incremental backup.

[Note]

• To shorten the recovery time when failure occurs during a differential/incremental backup or automatic log backup, a full backup that will serve as the baseline needs to be performed regularly.

The type of backup is specified in the command option.

Specify /dataStore/backupPath in the node definition file as the backup destination. Take into consideration physical failure of the disk, and be sure to set up the backup destination and database file (/dataStore/dbPath) so that the file is stored in a different physical disk.

There are 2 log persistency modes for transactions. Default is NORMAL.

• NORMAL: A transaction log file which is no longer required will be deleted by a checkpoint.
• KEEP_ALL_LOGS: All transaction log files are retained.

KEEP_ALL_LOGS is specified only for special operations. e.g. when issuing instructions to delete a log file in conjunction with the backup software of other companies, etc., but normally this is not used.

A specified example of a node definition file is shown below.

$ cat /var/lib/gridstore/conf/gs_node.json         # example of configuration check
{
        "dataStore":{
                "dbPath":"/var/lib/gridstore/data",
                "backupPath":"/mnt/gridstore/backup",      # backup directory
                "storeMemoryLimit":"1024",
                "concurrency":2,
                "logWriteMode":1,
                "persistencyMode":"NORMAL"                 # persistency mode
            :
            :
}

This section explains how to use a full backup, differential/incremental backup, and automatic log backup.

Specify the backup name (BACKUPNAME) when executing any type of backup. Data created in a backup will be stored in a directory created with the BACKUPNAME under the directory specified in the backupPath of the node definition file.

Up to 12 alphanumeric characters can be specified in the BACKUPNAME.

When a failure occurs, the system can be recovered up to the point where the full backup was completed. Implement a full backup of all the nodes constituting the cluster. Backup data is stored in the directory indicated by the BACKUPNAME of the command. It is recommended to specify the date in the BACKUPNAME in order to make it easier to understand and manage the backup data gathered.

Execute the following command on all the nodes inside the cluster.

$ gs_backup -u admin/admin 20141025

In this example,

1. a directory called “20141025” will be created under the backup directory,
2. and a checkpoint file (gs_cp_n_p.dat), transaction log file (gs_log_n_m.log), backup data file (gs_backup_info.json, gs_backup_info_digest.json), and LSN data file will be created in the backup directory.

Checkpoint files and transaction log files are created according to the concurrency (/dataStore/concurrency) figure in the node definition file.

$ cd /mnt/gridstore/backup
$ ls -l 20141025
-bash-4.1$ ls -l 20141025
Total 13502036
-rw-r--r-- 1 gsadm gridstore        731 Oct 25 06:39 2015 gs_backup_info.json
-rw-r--r-- 1 gsadm gridstore         82 Oct 25 06:39 2015 gs_backup_info_digest.json
-rw-r--r-- 1 gsadm gridstore 6759120896 Oct 25 06:37 2015 gs_cp_0_1.dat
-rw-r--r-- 1 gsadm gridstore 7064846336 Oct 25 06:38 2015 gs_cp_1_1.dat
-rw-r--r-- 1 gsadm gridstore    1048576 Oct 25 06:38 2015 gs_log_0_161.log
-rw-r--r-- 1 gsadm gridstore    1048576 Oct 25 06:39 2015 gs_log_1_161.log
-rw-r--r-- 1 gsadm gridstore        706 Oct 25 06:39 2015 gs_lsn_info.json

A backup command will only notify the server of the backup instructions and will not wait for the process to end.

Check the completion of the backup process by the status of the gs_stat command.

$ gs_backup -u admin/admin 20141025
$ gs_stat -u admin/admin --type backup
BackupStatus: Processing

• The backup status (BackupStatus) will be one of the following.
  • Processing: Under execution
  • -: Completed or not in operation

The status of the command below can be used to check whether the backup has been performed correctly.

$ gs_backuplist -u admin/admin

BackupName   Status   StartTime                EndTime
------------------------------------------------------------------------
 20141025NO2     P   2014-10-25T06:37:10+0900 -
 20141025        NG  2014-10-25T02:13:34+0900 -
 20140925        OK  2014-09-25T05:30:02+0900 2014-09-25T05:59:03+0900
 20140825        OK  2014-08-25T04:35:02+0900 2014-08-25T04:55:03+0900

The status symbol of the backup list indicates the following.

• P: Backup execution in progress
• NG: An error has occurred during backup execution, backup data is abnormal
• OK: Backup has been carried out normally

When a failure occurs, data can be recovered until the last differential/incremental backup was performed by using the full backup serving as the baseline (reference point) and the differential/incremental backup after the baseline. Get the full backup as a baseline for the differential/incremental backup and specify differential/incremental backup thereafter.

The backup interval needs to be studied in accordance with the service targets for the data update capacity and the time taken for recovery, but use the following as a guide.

• Full backup of baseline (Baseline): monthly
• Differential backup of updated blocks after creation of the baseline (Since): weekly
• Incremental backup (Incremental) of updated blocks after a baseline or differential backup: daily

Creation of baseline for full backup is specified below.

$ gs_backup -u admin/admin --mode baseline 201504
$ gs_stat -u admin/admin --type backup
BackupStatus: Processing (Baseline)

Database file in the data directory is copied under the backup directory as a baseline for the backup.

Specify incremental or since as the mode of the backup command when performing a regular backup of the differential/incremental block after creating a baseline (backup of data block updated after a full backup of the baseline). Specify the same BACKUPNAME as when the baseline was created.

# For incremental backup
$ gs_backup -u admin/admin --mode incremental 201504
$ gs_stat -u admin/admin --type backup
BackupStatus: Processing (Incremental)

# For differential backup
$ gs_backup -u admin/admin --mode since 201504
$ gs_stat -u admin/admin --type backup
BackupStatus: Processing (Since)

The command below can be used to check whether the backup has been performed correctly. As a differential/incremental backup will become a single recovery unit in a multiple backup, it will be treated as a single backup in the list of BACKUPNAME. Therefore, specify the backup name and check the details to see the detailed status.

A differential/incremental backup can be confirmed by checking that an asterisk "*" is appended at the beginning of the BACKUPNAME. The status of a differential/incremental backup is always "--".

The status of differential/incremental backup can be checked by specifying the BACKUPNAME in the argument of the gs_backuplist command.

# Display a list of BACKUPNAME
$ gs_backuplist -u admin/admin

BackupName   Status   StartTime                EndTime
------------------------------------------------------------------------
*201504          --  2015-04-01T05:20:00+0900 2015-04-24T06:10:55+0900
*201503          --  2015-03-01T05:20:00+0900 2015-04-24T06:05:32+0900
  :
 20141025NO2     OK   2014-10-25T06:37:10+0900 2014-10-25T06:37:10+0900
 
# Specify the individual BACKUPNAME and display the detailed information 
$ gs_backuplist -u admin/admin 201504
 
BackupName : 201504

BackupData            Status   StartTime                EndTime
--------------------------------------------------------------------------------
201504_lv0                OK  2015-04-01T05:20:00+0900 2015-04-01T06:10:55+0900
201504_lv1_000_001        OK  2015-04-02T05:20:00+0900 2015-04-01T05:20:52+0900
201504_lv1_000_002        OK  2015-04-03T05:20:00+0900 2015-04-01T05:20:25+0900
201504_lv1_000_003        OK  2015-04-04T05:20:00+0900 2015-04-01T05:20:33+0900
201504_lv1_000_004        OK  2015-04-05T05:20:00+0900 2015-04-01T05:21:15+0900
201504_lv1_000_005        OK  2015-04-06T05:20:00+0900 2015-04-01T05:21:05+0900
201504_lv1_001_000        OK  2015-04-07T05:20:00+0900 2015-04-01T05:22:11+0900
201504_lv1_001_001        OK  2015-04-07T05:20:00+0900 2015-04-01T05:20:55+0900

A directory will be created in the backup directory according to the following rules to store the differential/incremental backup data.

• BACKUPNAME_lv0: Baseline backup data of the differential/incremental backup is stored. lv0 fixed.
• BACKUPNAME_lv1_NNN_MMM: Differential (Since) and incremental (Incremental) backup data of the differential/incremental backup is stored.
  • NNN count is increased during a differential backup.
  • MMM count is cleared to 000 during a differential backup and increased during an incremental backup.

The status symbol of the backup list indicates the following.

• P: Backup execution in progress
• NG: An error has occurred during backup execution, backup data is abnormal
• OK: Backup has been carried out normally

A log of the updated block with the name gs_log_n_m_incremental.log is output under the directory shown in BackupData during a differential/incremental backup.

Differential/incremental backup can be compared to a full backup and backup time can be reduced. However, recovery when a failure occurs may take a while as the update block is rolled forward for the data of the full backup. Get the baseline regularly or execute a differential backup from the baseline by specifying since .

[Points to note]

• Partitions are automatically arranged (rebalanced) according to the cluster configuration changes and system load. When a differential log backup is specified after the arrangement of the partitions is changed, a message error “Log backup cannot be performed due to a change in the partition status” will appear. In this case, be sure to back up all the nodes constituting the cluster (baseline). Re-arrangement of partitions (rebalancing) occurs when the cluster configuration is changed as shown below.
  • Increase the number of nodes constituting a cluster by adding nodes
  • Decrease the number of nodes constituting a cluster by detaching nodes

GridDB automatically outputs a transaction log to the backup directory. Therefore, the system can always be recovered to the latest condition. As backup is carried out automatically, it is not possible to perform systematic backups according to the system operating state such as a “backup process scheduled in advance during low peak periods”. In addition, due to the automatic log backup, a system load will be imposed more or less during normal operation as well. Therefore, use of this indication is recommended only when there are surplus system resources.

Specify as follows when using an automatic log backup.

$ gs_backup -u admin/admin --mode auto 201411252100
$ gs_stat -u admin/admin --type backup

Execute the command to get the backup data in the directory indicated in BACKUPNAME.

• During an automatic log backup, operational settings for errors which occur during backup can be set in the option mode.
  • auto: When a backup error occurs, the node will become ABNORMAL and stop.
  • auto_nostop: Backup will be incomplete when a backup error occurs but the node will continue to operate.

In this example,

1. a directory with the name “201411252100” will be created under the backup directory.
2. A checkpoint file (gs_cp_n_p.dat), transaction log file (gs_log_n_m.log), backup data file (gs_backup_info.json, gs_backup_info_digest.json), and LSN data file will be created in the backup directory just like a full backup. Checkpoint files and transaction log files are created according to the concurrency (/dataStore/concurrency) figure in the node definition file.
3. A transaction log file (gs_log_n_m.log) is created under the “201411252100” directory when execution of the transaction is completed.

When operating with an automatic log backup, the transaction log file (gs_log_n_m.log) in 3) is rolled forward for the full backup data in 2) during recovery when a failure occurs. Therefore, specify the --mode auto to perform a full backup regularly as the recovery time will increase when the number of log files used during recovery increases.

The mode of the backup currently being executed and the detailed execution status can also be checked in data that can be obtained from the gs_stat command.

$ gs_stat -u admin/admin

    "checkpoint": {
        "archiveLog": 0,
        "backupOperation": 3,
        "duplicateLog": 0,
        "endTime": 0,
        "mode": "INCREMENTAL_BACKUP_LEVEL_0",
        "normalCheckpointOperation": 139,
        "pendingPartition": 1,
        "requestedCheckpointOperation": 0,
        "startTime": 1429756253260
    },
        :
        :

The meaning of each parameter related to the backup output in gs_stat is as follows.

• archiveLog: Cannot be used in the current version Item for compatibility purposes.
• backupOperation: Number of backups performed after system start-up.
• duplicateLog: Automatic log backup is performed to indicate whether a redundant log output has been carried out.
  • 0: Automatic log backup off
  • 1: Automatic log backup on
• endtime: "0" when backup or checkpoint is being performed. The time is set when the process ends.
• mode: Name of the backup or checkpoint process is displayed. Name of the backup process last performed or currently under execution is displayed.
  • BACKUP: Perform a full backup with an automatic log backup or full backup
  • INCREMENTAL_BACKUP_LEVEL_0: Create baseline of differential/incremental backup
  • INCREMENTAL_BACKUP_LEVEL_1_CUMULATIVE: Differential backup from the baseline
  • INCREMENTAL_BACKUP_LEVEL_1_DIFFERENTIAL: Incremental backup from the last backup

When a database failure occurs, it is necessary to understand which container needs to be recovered and how to contact the user of the container. To detect a container subject to recovery, the following data needs to be collected regularly.

• List of containers arranged in a partition
  • As containers are automatically created and arranged in the partitions according to the specifications of the application system, a list of containers and partition arrangements need to be output regularly using the gs_sh command.

Operating efforts can be cut down by creating a gs_sh command script to output the container list in advance.

In the example below, a gs_sh sub-command is created with the file name listContainer.gsh.

setnode node1 198.2.2.1  10040
setnode node2 198.2.2.2  10040
setnode node3 198.2.2.3  10040
setcluster cl1 clusterSeller 239.0.0.20 31999 $node1 $node2 $node3
setuser admin admin gstore
connect $cl1
showcontainer
connect $cl1 db0
showcontainer
 :
 :   (Repeat db for several minutes)
 :
quit

Change the node variables such as node 1, node 2, node 3 that constitute a cluster, and change the cluster variable such as cl1, user settings and database data where appropriate to suit the environment. See “GridDB operation control guide” (GridDB_OperationGuide.html) for the details of gs_sh.

Execute the gs_sh script file as shown below to collect a list of containers and partitions.

$ gs_sh listContainer.gsh>`date +%Y%m%d`Container.txt

Data is stored in 20141001Container.txt in the following format.

Database : public
Name                  Type         PartitionId
------------------------------------------------
container_7           TIME_SERIES            0
container_9           TIME_SERIES            7
container_2           TIME_SERIES           15
container_8           TIME_SERIES           17
container_6           TIME_SERIES           22
container_3           TIME_SERIES           25
container_0           TIME_SERIES           35
container_5           TIME_SERIES           44
container_1           TIME_SERIES           53
    :
 Total Count: 20
 
Database : db0
Name                  Type         PartitionId 
---------------------------------------------
CO_ALL1              COLLECTION           32
COL1                 COLLECTION          125
 Total Count: 2

When a failure occurs in GridDB, in addition to the cause of the failure being output to the event log file of the node in which the error occurred, if it is deemed that node operation cannot continue, the node status will become ABNORMAL and the node will be detached from the cluster service.

Cluster service will not stop even if the node status becomes ABNORMAL as operations are carried out with multiple replicas in a cluster configuration. Data recovery is necessary when all partitions including the replicas were to fail.

Use gs_stat to check the status of the master node to see whether data recovery is necessary or not. Recovery is necessary if the value of /cluster/partitionStatus is "OWNER_LOSS".

$ gs_stat -u admin/admin -p 10041
{
    "checkpoint": {
        :
    },
    "cluster": {
        "activeCount": 2,
        "clusterName": "clusterSeller",
        "clusterStatus": "MASTER",
        "designatedCount": 3,
        "loadBalancer": "ACTIVE",
        "master": {
            "address": "192.168.0.1",
            "port": 10011
        },
        "nodeList": [
            {
                "address": "192.168.0.2",
                "port": 10011
            },
            {
                "address": "192.168.0.3",
                "port": 10010
            }
        ],
        "nodeStatus": "ACTIVE",
        "partitionStatus": "OWNER_LOSS",     //(*)
        "startupTime": "2014-10-07T15:22:59+0900",
        "syncCount": 4
          :

Use the gs_partition command to check for data to be recovered. Partitions with problems can be checked by specifying the --loss option and executing the command.

In the example below, an error has occurred in Partition 68 due to a problem with node 192.168.0.3.

$ gs_partition -u admin/admin -p 10041 --loss
[
 {
        "all": [
            {
                "address": "192.168.0.1",
                "lsn": 0,
                "port": 10011,
                "status": "ACTIVE"
            },
                :
                :
            ,
            {
                "address": "192.168.0.3",
                "lsn": 2004,
                "port": 10012,
                "status": "INACTIVE"   // <--- The status of this node is not ACTIVE
            }
        ],
        "backup": [],
        "catchup": [],
        "maxLsn": 2004,
        "owner": null,           // Partition owner is not present in the cluster
        "pId": "68",             // ID of partition which needs to be recovered
        "status": "OFF"
   },
   {
        :
   }
  ]

Perform the following operation after starting a node.

1. Incorporation of node in cluster
2. Data consistency check and failover operations

After recovery ends, perform a full backup of all the nodes constituting the cluster.