Database Administrators
Q&A for database professionals who wish to improve their database skills
Latest Questions
1
votes
0
answers
12
views
GridDB How to properly delete a database with containers?
I'm working with GridDB CE in a Java application and trying to programmatically delete a database. However, I'm encountering that error. > 145045 JC_DATABASE_NOT_EMPTY ERROR Checking of client request failed. Number of DB containers may not be zero. Delete all containers in the DB targeted for delet...
I'm working with GridDB CE in a Java application and trying to programmatically delete a database. However, I'm encountering that error.
> 145045 JC_DATABASE_NOT_EMPTY ERROR Checking of client request failed. Number of DB containers may not be zero. Delete all containers in the DB targeted for deletion.
I want to delete an entire database (or simulate that by removing all its data). For example, before running an integration test or during cleanup, I want to ensure the DB is clean.
import com.toshiba.mwcloud.gs.*;
public class GridDBCleanup {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.setProperty("notificationAddress", "239.0.0.1");
props.setProperty("notificationPort", "31999");
props.setProperty("clusterName", "myCluster");
props.setProperty("user", "admin");
props.setProperty("password", "admin");
GridStore store = GridStoreFactory.getInstance().getGridStore(props);
// Attempt to drop database (simulated cleanup)
store.dropDatabase("testDB"); // This line throws the error
}
}
I've Tried
Ensured that testDB exists and is accessible.
Verified that containers are still present using getContainerNames.
Tried iterating and deleting containers manually before calling dropDatabase, but I want to know if there's a better/more idiomatic way to clean a DB.
What is the correct approach to fully delete a database or simulate that in GridDB?
Do I have to delete all containers manually before using dropDatabase()?
Is there a built-in way to list and delete all containers programmatically?
Any help or a working snippet would be appreciated!
omar esawy
(11 rep)
Jun 8, 2025, 09:06 AM
• Last activity: Jun 8, 2025, 01:03 PM
2
votes
0
answers
9
views
How Can I Optimize Partitioning and Replication Settings in GridDB for High-Volume Time-Series IoT Workloads?
I am designing an **IoT application** that ingests high volumes of time-series data in real time. I am evaluating **GridDB** as the backend due to its in-memory capabilities, hybrid storage model, and ACID compliance for container-level transactions. In particular, I am trying to determine the best...
I am designing an **IoT application** that ingests high volumes of time-series data in real time. I am evaluating **GridDB** as the backend due to its in-memory capabilities, hybrid storage model, and ACID compliance for container-level transactions.
In particular, I am trying to determine the best partitioning strategy to balance load and achieve efficient query performance for time-based queries. GridDB supports various partitioning methods (hash, interval, and interval-hash), each affecting data distribution and pruning efficiency differently. Also, I need to decide on the replication configuration—choosing between asynchronous and semi-synchronous modes—to balance data consistency, availability, and update overhead.
**Below is a sample Java snippet that demonstrates creating a time-series container with a basic schema.**
import com.toshiba.mwcloud.gs.*;
import com.toshiba.mwcloud.gs.common.*;
public class GridDBSetup {
public static void main(String[] args) throws Exception {
// Establish connection to the GridDB cluster
GridStore store = GridStoreFactory.getInstance().getGridStore("clusterName", "username", "password");
// Define container schema for a time-series data container
ContainerInfo containerInfo = new ContainerInfo("SensorData", ContainerType.TIME_SERIES);
containerInfo.setColumnNames(new String[]{
"timestamp", // primary key (time value)
"sensorID",
"value",
"status"
});
containerInfo.setColumnTypes(new DataType[]{
DataType.TIMESTAMP,
DataType.STRING,
DataType.DOUBLE,
DataType.STRING
});
// Specify the row key (primary key) column index
containerInfo.setRowKeyColumn(0);
// Optional: In some GridDB versions, you might include partitioning hints or tuning options.
// For example, if using a TQL-based method or custom API for partitioning:
// containerInfo.setPartitionOption("INTERVAL '1 DAY'");
// containerInfo.setReplicationNum(3); // for example, 3 replicas
// containerInfo.setReplicationMode(0); // 0: Asynchronous, 1: Semi-synchronous
// Create the container
store.putContainer(containerInfo);
// Sample query using TQL to filter by time range
String queryString = "SELECT * FROM SensorData WHERE timestamp >= ? AND timestamp rs = query.fetch();
while (rs.hasNext()) {
Row row = rs.next();
System.out.println(row.toString());
}
// Close connection
store.close();
}
}
Farwa Waheed
(21 rep)
Apr 8, 2025, 08:15 PM
1
votes
0
answers
7
views
Fail to be Cross Compiled in GridDB
I am trying to cross-compile GridDB. The configuration command I used is: ./configure --target=arm-buildroot-linux-gnueabihf --host=arm-buildroot-linux-gnueabihf --build=x86_64-pc-linux-gnu --prefix=/usr --exec-prefix=/usr --sysconfdir=/etc --localstatedir=/var --program-prefix= --disable-gtk-doc --...
I am trying to cross-compile GridDB. The configuration command I used is:
./configure --target=arm-buildroot-linux-gnueabihf --host=arm-buildroot-linux-gnueabihf --build=x86_64-pc-linux-gnu --prefix=/usr --exec-prefix=/usr --sysconfdir=/etc --localstatedir=/var --program-prefix= --disable-gtk-doc --disable-gtk-doc-html --disable-doc --disable-docs --disable-documentation --with-xmlto=no --with-fop=no --disable-dependency-tracking --enable-ipv6 --disable-nls --disable-static --enable-shared
However, after a while, the terminal outputs the following error:
./../3rd_party/sqlite_mod/build/work/libsqlite3.a: error adding symbols: File format not recognized
Upon checking the
configure
script of sqlite_mod
in the work folder, I found that the Makefile generated by build.sh
contains the following:
TCC = gcc -I. -I../../src -I../../gs_ext ...
I suspect that TCC
should be set to the cross-compiler (arm-buildroot-linux-gnueabihf-gcc
) like other Makefiles in GridDB. I modified TCC accordingly, but the cross-compilation still fails.
Shams
(111 rep)
Apr 3, 2025, 06:16 AM
0
votes
0
answers
10
views
GridDB Error 10085: TXN_CURRENT_DATABASE_REMOVED - Database Deleted During Login
I'm encountering an issue with GridDB where I receive the following error message: 10085 TXN_CURRENT_DATABASE_REMOVED ERROR User/DB administration operation failed. Current DB may have been deleted during login. Check whether the administrator has deleted the DB. I'm developing an application that c...
I'm encountering an issue with GridDB where I receive the following error message:
10085 TXN_CURRENT_DATABASE_REMOVED ERROR User/DB administration operation failed. Current DB may have been deleted during login. Check whether the administrator has deleted the DB.
I'm developing an application that connects to a GridDB instance to perform various data operations. The application is supposed to log in to a specific database and execute queries. However, I'm facing an issue where the connection fails intermittently with the error mentioned above.
GridDB Version: 4.5
Client Language: Java
Operating System: Ubuntu 20.04
Here's a simplified version of the Java code I'm using to connect to the database:
import com.toshiba.mwcloud.gs.*;
public class GridDBExample {
public static void main(String[] args) {
try {
// Set up GridDB connection
Properties props = new Properties();
props.setProperty("notificationAddress", "239.0.0.1");
props.setProperty("notificationPort", "31999");
props.setProperty("clusterName", "myCluster");
props.setProperty("user", "admin");
props.setProperty("password", "admin");
GridStoreFactory factory = GridStoreFactory.getInstance();
GridStore store = factory.getGridStore(props);
// Attempt to get a container
Container container = store.getContainer("myContainer");
if (container == null) {
System.out.println("Container not found.");
}
} catch (GSException e) {
e.printStackTrace();
}
}
}
The error occurs sporadically, and I'm unsure why the database might be getting deleted or how to handle this situation gracefully in my application.
Verified that the database exists before running the application.
Checked with the system administrator to ensure no manual deletions are occurring.
Reviewed the application logs for any operations that might inadvertently delete the database.
1. What could cause the database to be deleted during the login process?
2. How can I programmatically check for this condition and handle it in my application?
3. Are there any best practices for managing database connections in GridDB to avoid this issue?
Any help or guidance would be greatly appreciated. Thank you!
Abdelrahman Shehata
(1 rep)
Mar 31, 2025, 10:00 PM
0
votes
0
answers
15
views
Reconfiguring Data Partitioning on an Existing GridDB Container Without Downtime
I am managing a production GridDB environment where I initially created a container for storing sales data using the following SQL-like command: CREATE CONTAINER sales_data ( storeId INTEGER, saleDate TIMESTAMP, amount DOUBLE ) PARTITION BY HASH(storeId) REPLICATION 3; Over time, I have noticed that...
I am managing a production GridDB environment where I initially created a container for storing sales data using the following SQL-like command:
CREATE CONTAINER sales_data (
storeId INTEGER,
saleDate TIMESTAMP,
amount DOUBLE
)
PARTITION BY HASH(storeId)
REPLICATION 3;
Over time, I have noticed that the data distribution has become uneven due to evolving workload patterns. To address this, I attempted to modify the partitioning scheme by running:
ALTER CONTAINER sales_data
PARTITION BY HASH(storeId, saleDate);
However, this command fails with an error indicating that altering the partitioning key on an existing container isn’t supported.
**My Question:**
What are the recommended best practices for reconfiguring the partitioning scheme in GridDB when the current distribution is suboptimal?
Zaigham Ali Anjum
(9 rep)
Mar 30, 2025, 09:59 AM
• Last activity: Mar 30, 2025, 10:11 AM
2
votes
3
answers
226
views
When I combine the NOT and BETWEEN operators, the query unexpectedly retrieves additional null values
Query 1 and Query 2 share the same semantics, both involving the combination of the NOT and BETWEEN operators, with the evaluation of the BETWEEN expression being FALSE in both cases. However, Query 1 unexpectedly retrieves more null values than Query 2. Could this be a logical error? DROP TABLE tsq...
Query 1 and Query 2 share the same semantics, both involving the combination of the NOT and BETWEEN operators, with the evaluation of the BETWEEN expression being FALSE in both cases. However, Query 1 unexpectedly retrieves more null values than Query 2. Could this be a logical error?
DROP TABLE tsqsdb0_t0;
CREATE TABLE tsqsdb0_t0(
time TIMESTAMP PRIMARY KEY,
c0 INTEGER
) USING TIMESERIES ;
INSERT OR REPLACE INTO tsqsdb0_t0(time, c0)
VALUES (TIMESTAMP('2022-01-01T16:00:00Z'), 0),
(TIMESTAMP('2022-01-01T16:00:05Z'), null);
---------------------------
**Query 1**
SELECT c0,
time
FROM tsqsdb0_t0
WHERE NOT ( 2 BETWEEN c0 AND 1);
------------------------------------
**Query 2**
SELECT c0,
time
FROM tsqsdb0_t0
WHERE NOT ( -1 BETWEEN c0 AND 1);
Alice
(163 rep)
Jan 24, 2025, 12:21 PM
• Last activity: Jan 26, 2025, 03:37 AM
0
votes
0
answers
13
views
bin/gs_startnode fails to start on GridDB version 4.3.0
> System: ubuntu18.04 > > version:4.3.0 griddb-4.3.0.tar.gz I tried to compile and install the 4.3.0 version on ubuntu18.04. After compiling and installing, executing `bin/gs_ Startnode`, the program has an error, and the python 2.7 version is used. Is it the Python version? wdgk@ubuntu:~/applicatio...
> System: ubuntu18.04
>
> version:4.3.0 griddb-4.3.0.tar.gz
I tried to compile and install the 4.3.0 version on ubuntu18.04. After compiling and installing, executing
bin/gs_ Startnode
, the program has an error, and the python 2.7 version is used. Is it the Python version?
wdgk@ubuntu:~/application/griddb/griddb-4.3.0$ bin/gs_startnode
Traceback (most recent call last):
File "bin/gs_startnode", line 122, in
proc = subprocess.Popen(command, stdout=f, stderr=f)
File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Alice
(163 rep)
Nov 27, 2024, 03:51 AM
0
votes
1
answers
39
views
How can I optimize the integration between Oracle and GridDB when joining large tables
I’m currently working on integrating GridDB with Oracle 19c for a data-intensive application. The goal is to store some time-series data in GridDB while performing complex ETL processes and queries in Oracle. One of the Oracle tables is partitioned and contains over 500 million records. The query pe...
I’m currently working on integrating GridDB with Oracle 19c for a data-intensive application. The goal is to store some time-series data in GridDB while performing complex ETL processes and queries in Oracle. One of the Oracle tables is partitioned and contains over 500 million records. The query performance is significantly slower than expected, even with indexing and partitioning.
Here’s a simplified version of the query I’m running in Oracle 19c that also pulls data from GridDB via an API:
SELECT t1.customer_name, t2.product_name, SUM(t1.amount), gdb.stock_price FROM transactions t1 JOIN products t2 ON t1.product_id = t2.product_id JOIN (SELECT * FROM TABLE(griddb_stock_price_api())) gdb ON t1.product_id = gdb.product_id WHERE t1.transaction_date BETWEEN TO_DATE('2023-01-01', 'YYYY-MM-DD') AND TO_DATE('2023-12-31', 'YYYY-MM-DD') GROUP BY t1.customer_name, t2.product_name, gdb.stock_price;
**Data Details:**
**transactions (Oracle):**
**customer_name:** Name of the customer making the transaction.
**product_id:** The ID of the product being purchased.
**amount:** The transaction amount.
**transaction_date:** The date the transaction was made (partitioned by date).
**products (Oracle):Data Details:**
**transactions (Oracle):**
**customer_name**: Name of the customer making the transaction.
**product_id:** The ID of the product being purchased.
**amount**: The transaction amount.
**transaction_date:** The date the transaction was made (partitioned by date).
**products (Oracle):**
**product_id:** The ID of the product.
**product_name:** The name of the product.
**griddb_stock_price_api() (GridDB):** This API function retrieves real-time stock price data from GridDB.
**product_id:** The product ID linked to the stock price.
**stock_price:** The current stock price for the product.
**product_id:** The ID of the product.
**product_name:** The name of the product.
**griddb_stock_price_api() (GridDB):** This API function retrieves real-time stock price data from GridDB.
**product_id:** The product ID linked to the stock price.
**stock_price:** The current stock price for the product.
Zaigham Ali Anjum
(9 rep)
Oct 2, 2024, 10:20 AM
• Last activity: Oct 8, 2024, 06:31 AM
0
votes
0
answers
10
views
Why am I getting "SYNC_SERVICE_START_FAILED ERROR 20000" in GridDB when starting the synchronization service?
I'm using GridDB to manage a distributed database cluster, and I recently encountered the following error when attempting to start the synchronization service: ```python from griddb_python import StoreFactory, GSException def start_sync_service(): try: # Connect to GridDB cluster factory = StoreFact...
I'm using GridDB to manage a distributed database cluster, and I recently encountered the following error when attempting to start the synchronization service:
from griddb_python import StoreFactory, GSException
def start_sync_service():
try:
# Connect to GridDB cluster
factory = StoreFactory.get_default()
gridstore = factory.get_store({
"host": "239.0.0.1",
"port": 41999,
"cluster_name": "defaultCluster",
"username": "admin",
"password": "admin"
})
# Retrieve container
container = gridstore.get_container("containerName")
if container is None:
raise Exception("Container not found")
# Simulate data synchronization across nodes
row = container.get(1)
print(f"Data retrieved: {row}")
# Perform a write operation to trigger synchronization
container.put(2, {"id": 2, "value": "syncTest"})
print("Data synchronized across cluster")
except GSException as e:
print(f"Sync service failed to start: {e.what()}")
raise
if __name__ == "__main__":
start_sync_service()
I connect to the GridDB cluster and attempt to retrieve and update data on a container.
The goal is to start the synchronization service between nodes, but the process fails with the SYNC_SERVICE_START_FAILED ERROR 20000
.
Operating System: Ubuntu 20.04
Python Version: 3.8
GridDB Version: 4.5
Cluster Size: 5 nodes
System Memory: 8GB per node
I’m looking for guidance on:
How to check if the memory is indeed insufficient and causing the SYNC_SERVICE_START_FAILED ERROR 20000.
Best practices for ensuring sufficient memory for synchronization services in a GridDB cluster.
Steps I can take to adjust memory configurations or other system settings to avoid this issue.
Azza Galal
(9 rep)
Oct 2, 2024, 11:57 PM
0
votes
0
answers
18
views
How to obtain the names of only the affected columns after an update operation in GridDB?
I am performing an update operation on a GridDB container involving multiple columns. I update 10 columns, but only 2 of them are actually modified (the values of the other columns remain unchanged). For logging purposes, I need to identify which columns were updated. I am using GridDB’s APIs to exe...
I am performing an update operation on a GridDB container involving multiple columns. I update 10 columns, but only 2 of them are actually modified (the values of the other columns remain unchanged). For logging purposes, I need to identify which columns were updated.
I am using GridDB’s APIs to execute the update operation, similar to:
container.put(row);
After executing the update, I know that the number of affected rows can be tracked, but I need to log:
1. The actual data that was updated.
2. The names of the specific
columns that were modified.
Alice
(163 rep)
Sep 25, 2024, 06:26 AM
0
votes
0
answers
15
views
Creation of GridDB Time Series table not working (precision)
I'm testing out Kafka with GridDB and I wanted to create a time series table with a precision of nanoseconds (labeled as TIMESERIES(6)) but whenever I run the sql query into the CLI, i'm met with an error. I'm wondering if perhaps this is a CLI Error and if I wrote a short java method it'd work out?...
I'm testing out Kafka with GridDB and I wanted to create a time series table with a precision of nanoseconds (labeled as TIMESERIES(6)) but whenever I run the sql query into the CLI, i'm met with an error. I'm wondering if perhaps this is a CLI Error and if I wrote a short java method it'd work out?
Here is my SQL query:
create table meter_3 (timestamp TIMESTAMP(6) PRIMARY KEY, kwh DOUBLE, temp DOUBLE ) USING TIMESERIES;
USING TIMESERIES makes it a time_series container and the 6 in the col type of typestamp sets the precision level as indicated above. when I run this, I get the following error
D20332: An unexpected error occurred while executing a SQL. : msg=[[60015:DS_DS_SCHEMA_INVALID] Execute SQL failed (reason=CREATE TABLE failed (reason=Type of rowkey not supported)
so I removed the PRIMARY KEY indication because perhaps it's redundant with a time series container (it's already a feature)
but I get another error:
D20332: An unexpected error occurred while executing a SQL. : msg=[[60015:DS_DS_SCHEMA_INVALID] Execute SQL failed (reason=CREATE TABLE failed (reason=must define one rowkey)
I suppose worst case I can remove the time_series container designation and use this as a collection container? But does anybody know how to get around this?
L. Connell
(69 rep)
Sep 10, 2024, 05:41 PM
1
votes
0
answers
19
views
Unable to query after a successful connection for a period of time in GridDB
I’m experiencing an issue with GridDB where the program becomes blocked during a query and logs a `Sending heartbeat` It seems that the issue is related to the connection management or the GridDB cluster's responsiveness Following are the logs select * where (name='message-count') AND count > 0 AND...
I’m experiencing an issue with GridDB where the program becomes blocked during a query and logs a
Sending heartbeat
It seems that the issue is related to the connection management or the GridDB cluster's responsiveness
Following are the logs
select * where (name='message-count') AND count > 0 AND timestamp TIMESTAMP('2022-12-29T11:02:01.567+0800') order by timestamp
Sending heartbeat (statement=CREATE_SESSION, address=/192.168.1.134:10001, partition=7, statementId=15, elapsedMillis=10000)
Sending heartbeat (statement=CREATE_SESSION, address=/192.168.1.134:10001, partition=7, statementId=15, elapsedMillis=20004)
This may be a concurrency problem. I try the following code to reproduce this problem
package com.griddb.client;
import cn.hutool.core.date.DateField;
import cn.hutool.core.date.DateTime;
import cn.hutool.core.date.DateUtil;
import com.toshiba.mwcloud.gs.*;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.FutureTask;
import static cn.hutool.core.date.DatePattern.UTC_MS_WITH_ZONE_OFFSET_PATTERN;
public class GriddbClient {
static class Iot {
@RowKey
Date timestamp;
double data;
double temperature;
@Override
public String toString() {
return "Iot{" +
"timestamp=" + timestamp +
", data=" + data +
", temperature=" + temperature +
'}';
}
}
public static void main(String[] args) throws Exception{
//bulkInsertData();
concurrentQuery();
}
/**
* Bulk Insert Data
* @throws Exception
*/
public static void bulkInsertData() throws Exception{
Properties props = new Properties();
props.setProperty("host", "192.168.1.134");
props.setProperty("port", "10001");
props.setProperty("clusterName", "myCluster");
props.setProperty("user", "admin");
props.setProperty("password", "admin");
props.setProperty("database","public");
props.setProperty("transactionTimeout","5000");
props.setProperty("failoverTimeout","5000");
GridStore store = GridStoreFactory.getInstance().getGridStore(props);
TimeSeries lctime = store.putTimeSeries("lctime", test5.Iot.class);
lctime.createIndex("data");
lctime.setAutoCommit(false);
DateTime now = DateTime.now();
for(int i=0;i culume=new HashSet();
culume.add("timestamp");
culume.add("data");
culume.add("temperature");
Properties props = new Properties();
props.setProperty("host", "192.168.1.134");
props.setProperty("port", "10001");
props.setProperty("clusterName", "myCluster");
props.setProperty("user", "admin");
props.setProperty("password", "admin");
props.setProperty("database","public");
props.setProperty("transactionTimeout","5000");
props.setProperty("failoverTimeout","5000");
final GridStore store = GridStoreFactory.getInstance().getGridStore(props);
ExecutorService executorService = Executors.newFixedThreadPool(20);
List>> tasks=new ArrayList>>();
for(int i=0;i> task1=new FutureTask>(new Callable>() {
public List call() throws Exception {
return createTask(start,end,store);
}
});
tasks.add(task1);
executorService.submit(tasks.get(tasks.size()-1));
}
for(FutureTask> item:tasks){
List iots = item.get();
System.out.println(iots.size());
}
store.close();
}
private static List createTask(Date start, Date end, GridStore store) throws Exception{
TimeSeries lctime = store.getTimeSeries("lctime", test5.Iot.class);
String tql="select * where timestamp > TIMESTAMP('" + formatDate(start) + "') and timestamp query = lctime.query(tql, test5.Iot.class).fetch();
List iots=new ArrayList();
while (query.hasNext()){
test5.Iot next = query.next();
System.out.println(next.toString());
iots.add(next);
}
lctime.close();
return iots;
}
private static Date getDate(String dataStr) throws Exception{
DateFormat dateFormat1 = new SimpleDateFormat("yyyy-MM-dd");
return dateFormat1.parse(dataStr);
}
private static String formatDate(Date date){
return DateUtil.format(date,UTC_MS_WITH_ZONE_OFFSET_PATTERN);
}
}
Alice
(163 rep)
Sep 2, 2024, 04:54 AM
1
votes
1
answers
26
views
GridDB v5.6 Compression Type (ZSTD). Querying much faster?
GridDB 5.6 has a new compression method that I wanted to test. I made a simple test where I ingested X amount of rows and tested the compression method against the old compression available prior to 5.6 (ZLIB) and against no compression. The results were what you would expect: no compression had the...
GridDB 5.6 has a new compression method that I wanted to test. I made a simple test where I ingested X amount of rows and tested the compression method against the old compression available prior to 5.6 (ZLIB) and against no compression.
The results were what you would expect: no compression had the highest data footprint for 100,000,000 rows, ZLIB next, and finally, the new compression method ZSTD had the smallest footprint.
I also tested the query speed of these compression methods and to my surprise, the one with the smallest footprint (ZSTD) also had the quickest lookup times.
I am curious as to how this could be -- from my understanding, there must be some tradeoff when doing a more advanced method of compressing similar data. I'd at the very least expect that the newest compression method would be on par with ZLIB but with a smaller footprint.
And now for the results. As explained above, I inserted 100m rows of 'random' data and timed ingestion, the directory size of the data dir, and lookup times. Here are the results:
| | NO_COMPRESSION | COMPRESSION_ZLIB | COMPRESSION_ZSTD |
|---------------------|------------------|------------------|-----------------|
| Search (ms) | 32644 | 20666 | 11475 |
| Agreggation (ms) | 30261 | 13302 | 8402 |
| Storage (gridstore) | 11968312 (17GB) | 7162824 (6.9GB) | 6519520 (6.3GB) |
| Storage (/data) | 17568708 (12GB) | 1141152 (1.1GB) | 1140384 (1.1GB) |
| Insert (m:ss.mmm) | 14:42.452 | 15:02.748 | 15:05.404 |
If anybody has any insight into this perplexing compression issue, please share any expertise.
L. Connell
(69 rep)
Aug 28, 2024, 02:18 PM
• Last activity: Aug 28, 2024, 03:12 PM
1
votes
1
answers
24
views
Database memory limit
I want to create a test lab used by developers, one of the VMs will be running griddb as a database. I want to restrict the database engine memory usage by running "gs_paramconf -u admin/ --set storeMemoryLimit 4096MB" as described here https://griddb.net/en/docs/GridDB_OperationGuide.html. However,...
I want to create a test lab used by developers, one of the VMs will be running griddb as a database. I want to restrict the database engine memory usage by running "gs_paramconf -u admin/ --set storeMemoryLimit 4096MB" as described here https://griddb.net/en/docs/GridDB_OperationGuide.html .
However, the change doesn't persist, and after some time, when I run "gs_paramconf -u admin/ --show storeMemoryLimit", the value is reseted to the default one.
Could you please advise how should I proceed?
Thank you
Jacob_P
(29 rep)
Aug 22, 2024, 03:00 AM
• Last activity: Aug 23, 2024, 04:21 AM
1
votes
1
answers
25
views
GridDB via Kubernetes
I want to deploy application using Kubernetes, one of the used components will be a GridDb database, I used the following article to deploy it - https://griddb.net/en/blog/creating-a-kubernetes-application-using-griddb-and-go/, the deployment manifest is shown below. One thing that I want to change...
I want to deploy application using Kubernetes, one of the used components will be a GridDb database, I used the following article to deploy it - https://griddb.net/en/blog/creating-a-kubernetes-application-using-griddb-and-go/ , the deployment manifest is shown below. One thing that I want to change is the securityContext, I'd like to avoid running as a root user. However, in documentation it says "we need to run as root user to have the sufficient permissions to save the changes to the config file". Any advise how I should proceed?
apiVersion: apps/v1
kind: Deployment
metadata:
name: griddb-server-deployment
spec:
replicas: 3
selector:
matchLabels:
app: griddb-server
template:
metadata:
labels:
app: griddb-server
spec:
volumes:
- name: griddb-pv-storage
persistentVolumeClaim:
claimName: griddb-server-pvc
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: griddb-server
containers:
- name: griddbcontainer
image: localhost:5000/griddb-server:01
imagePullPolicy: IfNotPresent
ports:
- containerPort: 10001
volumeMounts:
- mountPath: "/var/lib/gridstore/data"
name: griddb-pv-storage
securityContext:
runAsUser: 0
runAsGroup: 0
env:
- name: NOTIFICATION_MEMBER
value: '1'
- name: GRIDDB_CLUSTER_NAME
value: "myCluster"
Jacob_P
(29 rep)
Aug 10, 2024, 05:21 PM
• Last activity: Aug 19, 2024, 10:42 PM
2
votes
0
answers
14
views
EXPLAIN SQL does not use the index in GridDB
I have created an index, but `explain` sql does not use the index. I have ensured that GridDB has up-to-date statistics by running the available commands and index on (DeviceId, Property) is in the correct order. griddb 5.1.0 gs[public]> explain analyze select * from properties_test where deviceId='...
I have created an index, but
explain
sql does not use the index. I have ensured that GridDB has up-to-date statistics by running the available commands and index on (DeviceId, Property) is in the correct order.
griddb 5.1.0
gs[public]> explain analyze select * from properties_test where deviceId='101' and property='0' limit 0,10;
The query had been executed. (60,662 ms)
gs[public]> getplantxt
Id Type Input Rows Lead time Actual time Node And more..
---------------------------------------------------------------------------------------------------
0 SCAN - - 60597 48739 192.168.1.62:10001 table: {properties_test} LIMIT: 10
1 RESULT 0 10 0 0 192.168.1.62:20001
gs[public]> showcontainer properties_test
Database : public
Name : properties_test
Type : COLLECTION
Partition ID: 13
DataAffinity: -
Columns:
No Name Type CSTR RowKey
------------------------------------------------------------------------------
0 Type STRING
1 DeviceId STRING
2 Property STRING
3 Id STRING
4 NumberValue DOUBLE
5 Value STRING
6 CreateTime TIMESTAMP
7 Timestamp TIMESTAMP
Indexes:
Name :
Type : TREE
Columns:
No Name
--------------------------
0 Property
Name :
Type : TREE
Columns:
No Name
--------------------------
0 DeviceId
Name :
Type : TREE
Columns:
No Name
--------------------------
0 Timestamp
Name :
Type : TREE
Columns:
No Name
--------------------------
0 DeviceId
1 Property
Arqish Mithani
(49 rep)
Aug 13, 2024, 06:35 AM
1
votes
1
answers
25
views
Being hit with memory limit restrictions in GridDB despite increasing the SQL and general storeMemoryLimit
I have a table with ~1 million rows that is being attempted to be read by a java program via the TQL interface (ie. just using the normal Java API, no SQL). But when I try to read the container, I'm met with a memory error: ` Exception in thread "main" com.toshiba.mwcloud.gs.common.GSStatementExcept...
I have a table with ~1 million rows that is being attempted to be read by a java program via the TQL interface (ie. just using the normal Java API, no SQL). But when I try to read the container, I'm met with a memory error:
`
Exception in thread "main" com.toshiba.mwcloud.gs.common.GSStatementException: [1043:CM_MEMORY_LIMIT_EXCEEDED] Memory limit exceeded (name=transactionWork.workerStack, requestedSize=67108880, totalSizeLimit=134217728, freeSizeLimit=1048576, totalSize=74448896, freeSize=0) (address=172.18.0.2:10001, partitionId=29)
`
the obvious solution is to increase the storeMemoryLimit, and so I have increased it from the default 1024MB to double that at 2048MB. Though I have increased the memory limit, I am getting the same error being read from the Java program, as in the values of the totalSizeLimit etc remain the same, meaning that increasing the storeMemoryLimit further would not help in this case (though I have tried).
I have also changed the compression method from NONE to the new ZSTD in the hopes that it could compress the dataset small enough to be read at once without hitting the memory limit but this has also failed.
My only other idea is to switch from TQL to SQL and increasing the SQL memory limit but I'd rather not switch the entirety of my code around.
Any ideas on getting this entire table read in one shot?
L. Connell
(69 rep)
Jul 9, 2024, 03:03 PM
• Last activity: Jul 15, 2024, 10:41 PM
1
votes
1
answers
23
views
Data seemingly not being saved into GridDB partitioned table (No Errors being thrown?)
I have a java program which writes rows into my GridDB table via the NoSQL API. I am properly catching errors and I see in the logs that the data is being stored into my container via multiput, but when I go into the CLI to view the contents of the data, the container is completely empty? First, her...
I have a java program which writes rows into my GridDB table via the NoSQL API. I am properly catching errors and I see in the logs that the data is being stored into my container via multiput, but when I go into the CLI to view the contents of the data, the container is completely empty?
First, here's what the container looks like:
gs[public]> showcontainer LOG_agent_intrusion_exploit
Database : public
Name : LOG_agent_intrusion_exploit
Type : COLLECTION
Partition ID: 22
DataAffinity: -
Partitioned : true
Partition Type : INTERVAL
Partition Column : timestamp
Partition Interval Value : 30
Partition Interval Unit : DAY
Expiration Type : PARTITION
Expiration Time : 30
Expiration Time Unit : DAY
Columns:
No Name Type CSTR RowKey
------------------------------------------------------------------------------
0 timestamp TIMESTAMP(3) NN
1 username STRING
2 incomingIP STRING
3 serverIP STRING
4 mtu INTEGER
5 statusCode INTEGER
6 cacheHit STRING
7 method STRING
8 url STRING
9 urlPrefix STRING
10 urlSuffix STRING
11 httpVersion STRING
12 service STRING
13 riskLevel STRING
14 headerContentType STRING
15 bytesReceived INTEGER
16 bytesSent INTEGER
17 headerAgent STRING
18 url2 STRING
19 url2Prefix STRING
20 url2Suffix STRING
21 meta1 STRING
22 meta2 STRING
23 meta3 STRING
24 meta4 STRING
My log parser is creating a hashmap with the container names as the key name, gathering up all rows, and then using store.multiPut to push to GridDB. for this function, I am doing a simple try/catch block which normally *does* catch the GridDB errors.
Quick snippet:
for (RawLog log : logs) {
try {
System.out.println(log.logtype + "~~~~~~");
System.out.println("configs.get(log.logtype)" + configs.get(log.logtype));
Row row = lp.patternMatcher(proc_container, log, configs.get(log.logtype));
if (row != null) {
proc_logs.add(row);
System.out.println("parsing this row: " + row);
} else
System.out.println("Could not parse " + log);
} catch (Exception e) {
e.printStackTrace();
System.out.println("Could not parse " + log);
}
}
containerRowsMap.put(proc_container, proc_logs);
}
try {
db.store.multiPut(containerRowsMap);
} catch (Exception e) {
System.out.println("Error with inserting data");
e.printStackTrace();
}
So, in this case, it seems as though data should be in my table, but when I run a simple query select * from LOG_agent_intrusion_exploit;
, i get zero rows back.
I tried foregoing multiput and inserting one row at a time but I get the exact same behavior.
L. Connell
(69 rep)
Jun 21, 2024, 08:28 PM
• Last activity: Jun 26, 2024, 05:23 PM
1
votes
0
answers
37
views
GridDB error cannot build on ubuntu 22.04
I'm getting the following error > 3rd_party/MessagePack/Makefile.am:8: warning: source file > '$(srcdir)/src/unpack.c' is in a subdirectory, > 3rd_party/MessagePack/Makefile.am:8: but option 'subdir-objects' is > disabled AM_INIT_AUTOMAKE add ([subdir-objects]) AM_INIT_AUTOMAKE([subdir-objects]) aft...
I'm getting the following error
> 3rd_party/MessagePack/Makefile.am:8: warning: source file
> '$(srcdir)/src/unpack.c' is in a subdirectory,
> 3rd_party/MessagePack/Makefile.am:8: but option 'subdir-objects' is
> disabled
AM_INIT_AUTOMAKE
add ([subdir-objects])
AM_INIT_AUTOMAKE([subdir-objects])
after modify file "configure.ac ", bootstrap.sh can run success.
./configure
mak
got errors like this
./util/code.h: In static member function ‘static void util::ObjectCoder::Impl::decodeInternal(C&, S&, T&, const Attribute&, const Traits&, const util::ObjectCoder::Impl::TypeTagutil::ObjectCoder::TYPE_LIST&)’:
./util/code.h:4771:57: error: expected primary-expression before ‘>’ token
4771 | value.push_back(coder.create());
| ^
./util/code.h:4771:59: error: expected primary-expression before ‘)’ token
4771 | value.push_back(coder.create());
| ^
./util/code.h: In static member function ‘static void util::ObjectCoder::Impl::decodeInternal(C&, S&, T&, const Attribute&, const Traits&, const util::ObjectCoder::Impl::TypeTagutil::ObjectCoder::TYPE_OPTIONAL_OBJECT&)’:
./util/code.h:4824:31: error: expected primary-expression before ‘>’ token
4824 | value = coder.create();
| ^
./util/code.h:4824:33: error: expected primary-expression before ‘)’ token
4824 | value = coder.create();
the gcc version here
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 11.3.0-1ubuntu1~22.04' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04)
Nooruddin Lakhani
(149 rep)
Jun 21, 2024, 11:24 AM
2
votes
1
answers
23
views
Potential issue with aggregation result (std deviation and variance)
I am running a query against the following dataset: [https://www.kaggle.com/datasets/census/population-time-series-data][1]. Here is the code: year_in_mili = 31536000000 ts = store.get_container("population") query = ts.query("select * from population where value > 280000") rs = query.fetch() data =...
I am running a query against the following dataset: https://www.kaggle.com/datasets/census/population-time-series-data .
Here is the code:
year_in_mili = 31536000000
ts = store.get_container("population")
query = ts.query("select * from population where value > 280000")
rs = query.fetch()
data = rs.next()
timestamp = calendar.timegm(data.timetuple())
gsTS = (griddb.TimestampUtils.get_time_millis(timestamp))
time = datetime.datetime.fromtimestamp(gsTS/1000.0)
added = gsTS + (year_in_mili * 7)
addedTime = datetime.datetime.fromtimestamp(added/1000.0)
variance = ts.aggregate_time_series(time, addedTime, griddb.Aggregation.VARIANCE, "value")
print("VARIANCE: ", variance.get(griddb.Type.DOUBLE))
stdDev = ts.aggregate_time_series(time, addedTime, griddb.Aggregation.STANDARD_DEVIATION, "value")
print("STANDARD DEVIATION: ", stdDev.get(griddb.Type.LONG))
The results for all are correct except for stdDev and variance
TOTAL: 48714984
AVERAGE: 289970
VARIANCE: -84078718183.5204
STANDARD DEVIATION: -9223372036854775808
COUNT 168
WEIGHTED AVERAGE: 289970
Obviously they should not be negative, but I did run the numbers in excel. These are the results:
var: 31045317.27
std dev: 5571.832488
Nooruddin Lakhani
(149 rep)
Mar 11, 2024, 10:52 AM
• Last activity: Mar 11, 2024, 12:30 PM
Showing page 1 of 20 total questions