spark-llap | write DataFrames and Streaming DataFrames

by hortonworks-spark Java Version: v1.0.7-1.6 License: Apache-2.0

X-Ray Key Features Code Snippets Community Discussions(2)Vulnerabilities Install Support

kandi X-RAY | spark-llap Summary

spark-llap is a Java library typically used in Big Data, Spark applications. spark-llap has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. However spark-llap build file is not available. You can download it from GitHub.

A library to read/write DataFrames and Streaming DataFrames to/from Apache Hive using LLAP. With Apache Ranger, this library provides row/column level fine-grained access controls. Ensure the following Spark properties are set via spark-defaults.conf or using --conf or through other Spark configuration. | Property | Description | Example | | ------------- |:-------------:| -----:| | spark.sql.hive.hiveserver2.jdbc.url | ThriftJDBC URL for LLAP HiveServer2 | jdbc:hive2://localhost:10000 | | spark.datasource.hive.warehouse.load.staging.dir | Temp directory for batch writes to Hive | /tmp | | spark.hadoop.hive.llap.daemon.service.hosts | App name for LLAP service | @llap0 | | spark.hadoop.hive.zookeeper.quorum | Zookeeper hosts used by LLAP | host1:2181;host2:2181;host3:2181 |.

Support

Quality

Security

License

Reuse

Support

spark-llap has a low active ecosystem.

It has 97 star(s) with 55 fork(s). There are 31 watchers for this library.

It had no major release in the last 12 months.

There are 28 open issues and 116 have been closed. On average issues are closed in 45 days. There are 4 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of spark-llap is v1.0.7-1.6

Quality

spark-llap has no bugs reported.

Security

spark-llap has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

License

spark-llap is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

spark-llap releases are available to install and integrate.

spark-llap has no build file. You will be need to create the build yourself to build the component from source.

Installation instructions are not available. Examples and code snippets are available.

Top functions reviewed by kandi - BETA

kandi has reviewed spark-llap and discovered the below as its top functions. This is intended to give you an instant insight into spark-llap implemented functionality, and help decide if they suit your requirements.

Read the schema
Convert a schema to a StructType
Re - registers a Spark Hive Driver
Get table schema
Create data reader factories
Create the job configuration
Gets the count data reader factories
Creates a statement type from the given options
Commit tables
Extract the table name from Hive table
Convert a Spark type to a hive data type
Create data reader
Handles record
Commits the current transaction
Aborts the streaming connection
Create a data writer
Return a new ColumnarBatch
Create streaming connection
Return a ColumnarBatch of the wrapped RowVector
Get data writer
Returns a DataSourceReader
Returns a record reader for the given split
Extract the task ID from the task context
Drop the specified table

Get all kandi verified functions for this library.

spark-llap Key Features

No Key Features are available at this moment for spark-llap.

spark-llap Examples and Code Snippets

No Code Snippets are available at this moment for spark-llap.

Community Discussions

Trending Discussions on spark-llap

How can we safely reuse a dataframe reading a Hive table using HiveWarehouseConnector?

Standalone spark cluster Authorization with Ranger

QUESTION

How can we safely reuse a dataframe reading a Hive table using HiveWarehouseConnector?

Asked 2020-Jan-16 at 23:38

When using the same dataframe reading a Hive table using HiveWarehouseConnector multiple times an exception occurs during computation.

Example :

...

ANSWER

Answered 2020-Jan-16 at 23:38

Spark 2.4
DataSourceV2 has been improved, especially by SPARK-23203 DataSourceV2 should use immutable trees

Spark 2.3
Disable column pruning in the HiveWarehouseConnector datasource reader.

Hortonworks has already fixed this issue, as stated by the HDP 3.1.5 Release Notes.
We can find the correction in its HiveWarehouseConnector github repository :

Source https://stackoverflow.com/questions/59758860

QUESTION

Standalone spark cluster Authorization with Ranger

Asked 2019-Sep-17 at 00:50

I'm working on a setup of EC2 machines that has standalone Spark cluster, Hive, Apache Ranger. Hive is integrated to Ranger.

As Ranger doesn't have support for Spark-SQL JDBC (port 10015), i tried this open source project https://github.com/yaooqinn/spark-authorizer for Spark Authorization. But didn't work as it seems to rely on yarn resource manager.

I wanted to know any possible ways to acheive authorization on Spark-sql with Apache Ranger.

We are not using any distributions implemented, so features like SPARK-LLAP in hortonworks is not an option.

I have already tried what is explained in http://mail-archives.apache.org/mod_mbox/ranger-user/201601.mbox/%3CCAC1CY9P7iek6U6VDwLEXvLdCNRTcJzk5UWg3sei1MuUMCGrtWA@mail.gmail.com%3E , but that didn't work either.

Have raised a spark jira last year for this but doesnt seem to have picked up yet. https://issues.apache.org/jira/browse/SPARK-24503

We are using Spark 2.3, Hive 2.3, Ranger 1.0.

...

ANSWER

Answered 2019-Sep-17 at 00:50

Build a simple authentication java application to spark-sql port 10015.

Source https://stackoverflow.com/questions/55448331

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install spark-llap

You can download it from GitHub.
You can use spark-llap like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the spark-llap component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: