HAWQ | Support | Compression library

You might find the following steps useful. Both databases connect to HBase from a Java client, although the Java client in OpenTSDB might be different.

maybe you can try install thrift using yum.

HAWQ is not, as of yet, supported on 7. It is in the backlog of items and should hopefully be done quickly - but if you're looking to test it's capabilities in the near term, I suggest you reinstall with < 7.

Assuming that you are trying to connect to HAWQ database from spark, you may use Postgres 8.4 JDBC driver (since HAWQ is based on Postgres).

Your table, retail_demo.order_lineitems_hawq, must be distributed with a hash. When you do this in HAWQ, the number of buckets is determined by default_hash_table_bucket_number which is set when the database is initialized. There will be a file in HDFS for each bucket and this is because there will be a fixed number of virtual segments, or vsegs, with hash distributed tables.

You can specify the number of buckets two ways. One is to do it when you create the table.

An External Table that uses gpfdist

• Data is in a posix filesystem, not HDFS
• No statistics
• Files could be on ETL nodes which aren't part of the cluster
• You could have multiple files across many servers too

• Ideal solution to load data in parallel to an Internal table

Honestly, there is no strictly constraints on how the hadoop installed and hawq installed if they are configured correctly.

For your concern, "I think the hawq-master should be built on top of hadoop, but there are no connection with hadoop-master". IMO, it should be "hawq should be built on top of hadoop". And we configured the hawq-master conf files(hawq-site.xml) to make hawq have connections with hadoop.

Usually, for the hawq master and hadoop master, we could install each component on one node, but we could install some of them on one node to save nodes. But for HDFS datanode and HAWQ segment, we often install them together. Taking the workload of each machine, we could install them as below:

Pivotal HDB (Commercial offering of HAWQ) is at various clients. Hawq is true 100% SQL compliant SQL engine based on MPP history. This is a unique product with state of art Query optimizer and dynamic partition elimination, very robust HDFS data federation features with Hbase, Hive, JSON, ORC(beta), and native hadoop file system. Hawq uses parquet storage format so tables created in hawq can be used in Hadoop eco-system.Hawq has ability to collect stats on external tables for faster data access. Support ACID transaction(Insert). On top of all these most compelling feature is doing data science using language extensions right in sql, supports R, Python, Java, Perl. I have seen implementations of HAWQ in Auto, oil and gas, IOT, healthcare industries. The typical use case i experienced is BI on top of hadoop, Data science model training and executing models, Interactive SQL on structured data. Since HAWQ is born out of Greenplum heritage, Some of the features that hawq are hard to find in competitive products. Hawq perfectly complements the Hadoop eco-system.

It looks like a issue with the PATH that the crontab cannot find the pg_dump binaries.Please try to run the script with the absolute path of pg_dump (usr/local/hawq/bin/pg_dump).

You can source the /usr/local/hawq/greenplum_path.sh before calling the pg_dump call too.