pinot | Apache Pinot - A realtime distributed OLAP datastore

You are using incorrect variable inside loop.

replace

unset($_products[$month]);

with

What you are trying to get is

Your issue is in the definition of the Vinicola.emailVinicola column

A naive way of doing it would use pandas ffill-method

Update: This is because the connector does not support mixed case table names. Mixed case column names are supported. There is a pull request to add support for mixed case table names: https://github.com/trinodb/trino/pull/7630

I don't think you understood Classes real well but still you need to use self.attribute to use any attributes inside class functions, here is a code that will give you the required output

When you try to access presto from superset, the network connection is between superset container to presto container, so localhost will not work.

You will need to get the real ip of prestodb container, either container ip or host ip. Can you try the following?

Apache Pinot and Elasticsearch solve distinct problems.

Elasticsearch is a search engine used for full-text searches, fuzzy queries, auto-completion of search terms, etc. It achieves this using something called an inverted index. Conventional indexing used sorted index where the document was stored as the key and the keywords as the value. In this case, the query latency would be very high since the entire document needs to be searched. But in an inverted index, the keyword is stored as the key and the document id's as the value. Here, since only the search keywords are needed to be searched, the query latency would be very low. Hence, Elasticsearch uses inverted indices to solve its core purpose, which is 'search'.

Apache Pinot was not built for 'search'. It was rather built for realtime analytics. It uses something called Star-Tree index, which is something like pre-aggregated value store of all combinations of all dimensions of the data. As you can see, Apache Pinot is interested in the aggregate derivations/reductions from the data rather than the data itself. It uses these pre-aggregated values to provide a very low latency, realtime analytics on the data.

A very important use case of Apache Pinot would be to compute realtime per-user-level analytics and render live per-user-facing dashboards. Elasticsearch too can render realtime dashboards using Kibana, but since it uses inverted index approach, it won't be suitable for per-user-level analytics as that will put a huge load on the server and will require a large number of elastic instances. Due to this upper bound, Elasticsearch would not be suited for per-user-level analytics.

So, if you want to have search functionality in your application and also per-user-level analytics, the best way would be to have both Elasticsearch and Pinot consumers ingest data from the same Kafka topic, through parallel pipelines. This way, while Elasticsearch indexes the data for search purposes, Pinot will process the data for per-user-level analytics.

Pinot has two ways to handle JSON records:

1. Flatten the record during ingestion time: In this case, we treat each nested field as a separated field, so need to:

• Define those fields in the table schema
• Define transform functions to flatten nested fields in table config

Please see how column subjects_name and subjects_grade is defined below. Since it's an array, so both fields are multi-value columns in Pinot.

2. Directly ingest JSON records

In this case, we treat each nested field as one single field, so need to:

• Define the JSON field in table schema as a string with maxLength value
• Put this field into noDictionaryColumns and jsonIndexColumns in table config
• Define transform functions jsonFormat to stringify the JSON field in table config

Please see how column subjects_str is defined below.

Below is the sample table schema/config/query:

Sample Pinot Schema: