Greenplum | Some Greenplum SQL commands for DBA | Database library

Try properly escaping the identifiers:

You are missing a few things in your table definition. You forgot "external" and "table".

Apparently, you can use regex pattern for consuming the data from different kafka topics.

Lets say, you have topic names like "topic-ingestion1", "topic-ingestion2" - then you can create a regex pattern for consuming data from all topics ending with "*ingestion".

Once the new topic gets created in the format of your regex pattern - spark will automatically start streaming data from the newly created topic.

Reference: [https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#consumer-caching]

you can use this parameter to specify your cache timeout. "spark.kafka.consumer.cache.timeout".

From the spark documentation:

spark.kafka.consumer.cache.timeout - The minimum amount of time a consumer may sit idle in the pool before it is eligible for eviction by the evictor.

Lets say if you have multiple sinks where you are reading from kafka and you are writing it into two different locations like hdfs and hbase - then you can branch out your application logic into two writeStreams.

If the sink (Greenplum) supports batch mode of operations - then you can look at forEachBatch() function from spark structured streaming. It will allow us to reuse the same batchDF for both the operations.

Reference: [https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#consumer-caching]

PostgreSQL 12 and later now allow GSSAPI encrypted connections. This parameter controls whether to enforce using GSSAPI encryption or not.

If encryption is not configured, should be disabled by setting JDBC connection parameter
gssEncMode=disable

Ref: https://www.ibm.com/support/pages/node/6323207

Please try like this it works for me:

The shared_buffers configuration is a per-segment configuration. Setting a value of 125MB for 8 segments will allocate 1GB for all 8 segments. Here is a snippet from the documentation.

Sets the amount of memory a Greenplum Database segment instance uses for shared memory buffers. This setting must be at least 128KB and at least 16KB times max_connections.

...

Each Greenplum Database segment instance calculates and attempts to allocate certain amount of shared memory based on the segment configuration.

Here's what I get from @leskin-in in greenplum slack:

I suppose this is currently not possible. When Greenplum executes "normal" queries, each of Greenplum segments acts as an independent PostgreSQL instance which executes a plan created by GPDB master. pg_stat_statements tracks resources usage for a single PostgreSQL instance; as a result, in GPDB it is able to track the resources consumption for each segment independently. There are several complications PostgreSQL pg_stat_statements does not deal with. An example is that GPDB uses slices. On GPDB segments, these are independent parts of plan trees and are executed as independent queries. I suppose when a query to pg_stat_statemens is made on GPDB master in current version, the results retrieved are for master only. As "normal" queries are executed in most part by segments, the results are inconsistent with actual resources' consumption by the query. In open-source Greenplum 5 & 6 there is a Greenplum-specific utility gpperfmon. It provides some of the pg_stat_statements features, and is cluster-aware (shows actual resources consumption for the cluster as a whole, and also several cluster-specific metrics).

The old version of PXF with hawq actually resides with data nodes and utilizes short-circuit read. THe current PXF has changed to reside with Greenplum segment hosts and acts like a hdfs client. I think you can tweak pxf source codes and setup pxf on datanodes with short-circuit read. However, you speed up the hdfs<->pxf communication, but slow down pxf<->greenplum segment communication.