wal | A write-ahead log for Go

When using Ignite 2.8.1-1 version and default configuration(1GB heap, 20% default region for off-heap storage and persistence enabled) on a Linux host with 16GB memory, I notice the ignite process could use up to 11GB of memory(verified by checking the resident size of memory used by the process in top, see attachment). When I check the metrics in the log, the consumed memory(heap+off-heap) doesn't add up to close to 7GB. One possibility is the extra memory could be used by the checkpoint buffer but that shall be by default 1/4 of the default region, that is, only about a quarter of 0.25 * 0.2 * 16GB.

Any hints on what the rest of the memory is used for?

Thanks!

I am trying to create an ignite service using aspnetcore background service functionality. When I start the service it works fine, but as soon as a client is attempting to connect I get the following exception:

[09:51:15,746][SEVERE][disco-notifier-worker-#75%ignite-instance-7b562b36-d203-4652-99ec-5fad30b09a3b%][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Invalid array specification: Npgsql.TypeHandlers.TextHandler+-Read>d__7]]

I am using the below configuration for Ignition.Start():

{"IgniteInstanceName":null,"AutoGenerateIgniteInstanceName":true,"GridName":null,"BinaryConfiguration":null,"CacheConfiguration":null,"SpringConfigUrl":null,"JvmDllPath":null,"IgniteHome":null,"JvmClasspath":null,"JvmOptions":["-Xms1024m","-Xmx1024m"],"Assemblies":null,"SuppressWarnings":false,"LifecycleHandlers":null,"JvmInitialMemoryMb":-1,"JvmMaxMemoryMb":-1,"DiscoverySpi":{"IpFinder":{"LocalAddress":null,"MulticastGroup":"228.1.2.4","MulticastPort":47400,"AddressRequestAttempts":2,"ResponseTimeout":"00:00:00.5000000","TimeToLive":null,"Endpoints":["127.0.0.1:47500..47509"]},"SocketTimeout":"00:00:05","AckTimeout":"00:00:05","MaxAckTimeout":"00:10:00","NetworkTimeout":"00:00:05","JoinTimeout":"00:00:00","ForceServerMode":false,"ClientReconnectDisabled":false,"LocalAddress":null,"ReconnectCount":10,"LocalPort":47500,"LocalPortRange":100,"StatisticsPrintFrequency":"00:00:00","IpFinderCleanFrequency":"00:01:00","ThreadPriority":10,"TopologyHistorySize":1000},"CommunicationSpi":null,"EncryptionSpi":null,"ClientMode":false,"IncludedEventTypes":[12,22,45],"LocalEventListeners":null,"MetricsExpireTime":"10675199.02:48:05.4775807","MetricsHistorySize":10000,"MetricsLogFrequency":"00:01:00","MetricsUpdateFrequency":"00:00:02","NetworkSendRetryCount":3,"NetworkSendRetryDelay":"00:00:01","NetworkTimeout":"00:00:05","WorkDirectory":"D:\ApacheIgniteDataJr\","Localhost":null,"IsDaemon":false,"UserAttributes":null,"AtomicConfiguration":null,"TransactionConfiguration":null,"IsLateAffinityAssignment":true,"Logger":null,"FailureDetectionTimeout":"00:00:10","SystemWorkerBlockedTimeout":null,"ClientFailureDetectionTimeout":"00:00:30","PluginConfigurations":null,"EventStorageSpi":null,"MemoryConfiguration":null,"DataStorageConfiguration":{"StoragePath":null,"CheckpointFrequency":"00:03:00","CheckpointThreads":4,"LockWaitTime":"00:00:10","WalHistorySize":20,"WalSegments":10,"WalSegmentSize":67108864,"WalPath":"db/wal","WalArchivePath":"db/wal/archive","WalMode":1,"WalThreadLocalBufferSize":131072,"WalFlushFrequency":"00:00:02","WalFsyncDelayNanos":1000,"WalRecordIteratorBufferSize":67108864,"AlwaysWriteFullPages":false,"MetricsEnabled":false,"MetricsRateTimeInterval":"00:01:00","MetricsSubIntervalCount":5,"CheckpointWriteOrder":1,"WriteThrottlingEnabled":false,"WalCompactionEnabled":false,"MaxWalArchiveSize":1073741824,"SystemRegionInitialSize":41943040,"SystemRegionMaxSize":104857600,"PageSize":4096,"ConcurrencyLevel":16,"WalAutoArchiveAfterInactivity":"-00:00:00.0010000","CheckpointReadLockTimeout":null,"WalPageCompression":0,"WalPageCompressionLevel":null,"DataRegionConfigurations":[{"Name":"ProductName_Region","PersistenceEnabled":true,"InitialSize":104857600,"MaxSize":429496729,"SwapPath":null,"PageEvictionMode":0,"EvictionThreshold":0.9,"EmptyPagesPoolSize":100,"MetricsEnabled":false,"MetricsRateTimeInterval":"00:01:00","MetricsSubIntervalCount":5,"CheckpointPageBufferSize":0,"LazyMemoryAllocation":true}],"DefaultDataRegionConfiguration":{"Name":"Default_Region","PersistenceEnabled":false,"InitialSize":104857600,"MaxSize":429496729,"SwapPath":null,"PageEvictionMode":0,"EvictionThreshold":0.9,"EmptyPagesPoolSize":100,"MetricsEnabled":false,"MetricsRateTimeInterval":"00:01:00","MetricsSubIntervalCount":5,"CheckpointPageBufferSize":0,"LazyMemoryAllocation":true}},"SslContextFactory":null,"PeerAssemblyLoadingMode":0,"PublicThreadPoolSize":16,"StripedThreadPoolSize":16,"ServiceThreadPoolSize":16,"SystemThreadPoolSize":16,"AsyncCallbackThreadPoolSize":16,"ManagementThreadPoolSize":4,"DataStreamerThreadPoolSize":16,"UtilityCacheThreadPoolSize":16,"QueryThreadPoolSize":16,"SqlConnectorConfiguration":null,"ClientConnectorConfiguration":null,"ClientConnectorConfigurationEnabled":true,"LongQueryWarningTimeout":"00:00:03","PersistentStoreConfiguration":null,"IsActiveOnStart":true,"ConsistentId":null,"RedirectJavaConsoleOutput":true,"AuthenticationEnabled":false,"MvccVacuumFrequency":5000,"MvccVacuumThreadCount":2,"SqlQueryHistorySize":1000,"FailureHandler":null,"SqlSchemas":["PRODUCTNAME_MODELS"],"ExecutorConfiguration":null,"JavaPeerClassLoadingEnabled":false,"AsyncContinuationExecutor":0}

After starting the node, I am activating and setting baseline auto-adjustment flag.

What am I doing wrong here?

We are using GridGain Community edition : 8.8.10 and have created Ignite Cluster in Kubernetes using the Apache Ignite operator. We have enabled native persistence also.

https://ignite.apache.org/docs/latest/installation/kubernetes/gke-deployment

In the development environment we shutdown our cluster during the night and bring it up during morning hours. When the cluster comes up it contains the data which was stored earlier. If we search the cache using key then it returns the result, but if we use the Query API for partial search then it is not returning results. We checked the cache size and it matches the datasource record size. Also after we search the Cache using the cache key, then that entry is available in the Query search results.

If we shut-down one of the nodes of the Ignite Cluster or client nodes. The TextSearch still works. TextSearch doesn't works only when all the nodes of the cluster are scaled-down and then scaled up using the existing disk.

Is there any configuration required to enable Query search after cold restart of the Ignite cluster ?

I have been working on getting my database backing up to work and I have reached a point where I am not sure what to do.

Basically at first the application opens a Login activity, the user logs in and their database file (if it exists) is downloaded from the Firebase Storage, and then the application navigates to the MainActivity.

In the MainActivity I call a method that sends the user's database file to Firebase Storage. I tried to manage the process by closing the database but since i couldn't fix an error of "E/ROOM: Invalidation tracker is initialized twice :/.", then I found an answer to use a checkpoint (Backup Room database). Now I implemented the forced checkpoint method.

I'm using Room version 2.2.6., Android Studio 4.2.2 and writing in Java.

I filled my app's database manually interacting with the app in an emulator, and it shows up both in the app and in the database inspector as one would expect. I built a certain initial example state that I would like my users to have upon first start. I then downloaded the .db .db-shm .db-wal files straight from the device file explorer and saved them in src/main/assets/databases/ as one does.

In my database initialization code I added the line .createFromAsset("databases/Example.db") between the .databaseBuilder and .build calls, to prepopulate my database from those files.

Yet after wiping my local data and reinstalling the app, the database stays empty.

Any clue what direction I could search in? All the posts here I found relating to createFromAsset only contain the same copied paragraph from the documentation.

To check my database I always open Device File Explorer and navigate to package.name> databases> here I find 3 files: dbName, dbName-shm, dbName-wal. The file that I occupy is dbName, so I right click and choose "Save as ..." after choosing the path the file is saved, later I check it with a software called "DB Browser for SQLite".

I had never had problems to see my database, but about 1 week ago the problems started, because the file called "dbName" is never updated, how do I know that? In the column "Date" the last modification date of each file, dbName.shm and dbName-wal change value in "Date" when I tap on "Synchronize" but "dbName" keeps the creation date and time, when opening the file with "DB Browser" there is nothing.

What is the problem? Has the path where my database is being saved changed?

I have converted my app to use Android Room for SQLite DB. There are some crashes on different devices with my implementation.

I'm using an SQLite database with peewee on multiple machines, and I'm encountering various OperationalError, DataBaseError. It's obviously a problem of multithreading, but I'm not at all an expert with this nor with SQL. Here's my setup and what I've tried.

I'm using peewee to log machine learning experiments. Basically, I have multiple nodes (like, different computers) which run a python file, and all write to the same base.db file in a shared location. On top of that, I need a single read access from my laptop, to see what's going on. There are at most ~50 different nodes which instantiate the database and write things on it.

At first, I used the SQLite object:

I have a pod running RabbitMQ. Below is the deployment manifest: