alluxio | data orchestration for analytics and machine learning

by Alluxio Java Version: v2.7.4 License: Apache-2.0

Download this library from

kandi X-RAY | alluxio Summary

alluxio is a Java library typically used in Big Data, Tensorflow, Spark, Hadoop applications. alluxio has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has high support. You can download it from GitHub.

Alluxio (formerly known as Tachyon) is a virtual distributed storage system. It bridges the gap between computation frameworks and storage systems, enabling computation applications to connect to numerous storage systems through a common interface. Read more about Alluxio Overview. The Alluxio project originated from a research project called Tachyon at AMPLab, UC Berkeley, which was the data layer of the Berkeley Data Analytics Stack (BDAS). For more details, please refer to Haoyuan Li's PhD dissertation Alluxio: A Virtual Distributed File System.

Support

Quality

Security

License

Reuse

kandi-support Support

alluxio has a highly active ecosystem.
It has 5591 star(s) with 2665 fork(s). There are 440 watchers for this library.
There were 1 major release(s) in the last 6 months.
There are 595 open issues and 1016 have been closed. On average issues are closed in 65 days. There are 168 open pull requests and 0 closed requests.
It has a negative sentiment in the developer community.
The latest version of alluxio is v2.7.4

alluxio Support

Best in #Java

Average in #Java

alluxio Support

Best in #Java

Average in #Java

quality kandi Quality

alluxio has 0 bugs and 0 code smells.

alluxio Quality

Best in #Java

Average in #Java

alluxio Quality

Best in #Java

Average in #Java

security Security

alluxio has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
alluxio code analysis shows 0 unresolved vulnerabilities.
There are 0 security hotspots that need review.

alluxio Security

Best in #Java

Average in #Java

alluxio Security

Best in #Java

Average in #Java

license License

alluxio is licensed under the Apache-2.0 License. This license is Permissive.
Permissive licenses have the least restrictions, and you can use them in most projects.

alluxio License

Best in #Java

Average in #Java

alluxio License

Best in #Java

Average in #Java

build Reuse

alluxio releases are available to install and integrate.
Build file is available. You can build the component from source.
Installation instructions, examples and code snippets are available.
alluxio saves you 235336 person hours of effort in developing the same functionality from scratch.
It has 271922 lines of code, 21215 functions and 2779 files.
It has medium code complexity. Code complexity directly impacts maintainability of the code.

alluxio Reuse

Best in #Java

Average in #Java

alluxio Reuse

Best in #Java

Average in #Java

Top functions reviewed by kandi - BETA

kandi has reviewed alluxio and discovered the below as its top functions. This is intended to give you an instant insight into alluxio implemented functionality, and help decide if they suit your requirements.

Creates an inode path .
Updates the metadata for the given inode .
Updates the state of the UDB .
Initializes the journal from backup .
Handles a persist job .
Executes collect info on all hosts .
Attempts to put a page .
Get the effective Memory layout specification .
Get configuration .
Initialize the journal .

alluxio Key Features

Alluxio, data orchestration for analytics and machine learning in the cloud

alluxio Examples and Code Snippets

See all related Code Snippets

Docker

Copy

Download

# Create a network for connecting Alluxio containers
$ docker network create alluxio_nw
# Create a volume for storing ufs data
$ docker volume create ufs
# Launch the Alluxio master
$ docker run -d --net=alluxio_nw \
    -p 19999:19999 \
    --name=alluxio-master \
    -v ufs:/opt/alluxio/underFSStorage \
    alluxio/alluxio master
# Launch the Alluxio worker
$ export ALLUXIO_WORKER_RAMDISK_SIZE=1G
$ docker run -d --net=alluxio_nw \
    --shm-size=${ALLUXIO_WORKER_RAMDISK_SIZE} \
    --name=alluxio-worker \
    -v ufs:/opt/alluxio/underFSStorage \
    -e ALLUXIO_JAVA_OPTS="-Dalluxio.worker.ramdisk.size=${ALLUXIO_WORKER_RAMDISK_SIZE} -Dalluxio.master.hostname=alluxio-master" \
    alluxio/alluxio worker

MacOS Homebrew

Copy

Download

$ brew install alluxio

Depend on Alluxio

Copy

Download

&lt;dependency&gt;
  &lt;groupId&gt;org.alluxio&lt;/groupId&gt;
  &lt;artifactId&gt;alluxio-shaded-client&lt;/artifactId&gt;
  &lt;version&gt;2.6.0&lt;/version&gt;
&lt;/dependency&gt;

convert text to number in case of an error in Bash

Copy

Download

if folder_get_time=$((time /opt/alluxio/bin/alluxio fs copyToLocal "$SRC" "$COPY_TO_LOCAL" )2>&1 >/dev/null )
then
    echo "success" >&2
else
    rc=$?
    echo "fail" >&2
    exit $rc
fi

case $folder_get_time in
  '' | *[!0-9]*) echo "fail" >&2; exit 127 ;;
esac

-----------------------

if folder_get_time=$((time /opt/alluxio/bin/alluxio fs copyToLocal "$SRC" "$COPY_TO_LOCAL" )2>&1 >/dev/null )
then
    echo "success" >&2
else
    rc=$?
    echo "fail" >&2
    exit $rc
fi

case $folder_get_time in
  '' | *[!0-9]*) echo "fail" >&2; exit 127 ;;
esac

See all related Code Snippets

Community Discussions

Trending Discussions on alluxio

convert text to number in case of an error in Bash
Spark kubernetes client mode (separate driver pod) setup

Trending Discussions on alluxio

QUESTION

convert text to number in case of an error in Bash

Asked 2022-Mar-28 at 09:09

I am running this command which usually returns a number. In cases of error, this parses into text which later on can cause error. How can I prevent this ?

folder_get_time=$((time  /opt/alluxio/bin/alluxio fs copyToLocal "$SRC" "$COPY_TO_LOCAL" )2>&1 >/dev/null )

ANSWER

Answered 2022-Mar-28 at 09:09

If you can't reliably trap the error

if folder_get_time=$((time /opt/alluxio/bin/alluxio fs copyToLocal "$SRC" "$COPY_TO_LOCAL" )2>&1 >/dev/null )
then
    echo "success" >&2
else
    rc=$?
    echo "fail" >&2
    exit $rc
fi

maybe check for a non-numeric output.

case $folder_get_time in
  '' | *[!0-9]*) echo "fail" >&2; exit 127 ;;
esac

Source https://stackoverflow.com/questions/71644102

QUESTION

Spark kubernetes client mode (separate driver pod) setup

Asked 2020-Apr-29 at 21:21

I'm trying to get a spark kubernetes install working where the spark driver node resides in its own separate pod (client mode), and uses the SparkSession.builder mechanism to bootstrap the cluster (not using spark-submit).

I'm working from this:

https://spark.apache.org/docs/latest/running-on-kubernetes.html

Here is the code used by the driver to bootstrap the cluster:

val sparkSession = SparkSession.builder
  .master("k8s://https://kubernetes.default.svc:32768")
  .appName("test")
  .config("spark.driver.host", "sparkrunner-0")
  .config("spark.driver.port", "7077")
  .config("spark.driver.blockManager.port", "7078")
  .config("spark.kubernetes.container.image","spark-alluxio")
  .config("fs.alluxio.impl", "alluxio.hadoop.FileSystem")
  .config("fs.alluxio-ft.impl", "alluxio.hadoop.FaultTolerantFileSystem")
  .getOrCreate

The container image (spark-alluxio) was built by adding the alluxio client library to a binary spark distribution (2.4.2).

Here is the kubernetes yaml used to deploy the driver program, which sits inside a StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: sparkrunner
  labels:
    app: sparkrunner
spec:
  selector:
    matchLabels:
      app: sparkrunner
  serviceName: sparkrunner
  replicas: 1
  template:
    metadata:
      labels:
        app: sparkrunner
    spec:
      containers:
        - name: sparkrunner
          image: "rb/sparkrunner:latest"
          imagePullPolicy: Never
          ports:
            - name: application 
              containerPort: 9100
            - name: driver-rpc-port
              containerPort: 7077
            - name: blockmanager
              containerPort: 7078

and here is the kubernetes yaml to deploy the services which sit on top of the driver program:

# Headless service for stable DNS entries of StatefulSet members.
apiVersion: v1
kind: Service
metadata:
  name: sparkrunner
spec:
  ports:
  - name: driver-rpc-port
    protocol: TCP 
    port: 7077
    targetPort: 7077
  - name: blockmanager
    protocol: TCP 
    port: 7078
    targetPort: 7078
  clusterIP: None
  selector:
    app: sparkrunner

---

# Client service for connecting to any spark instance.
apiVersion: v1
kind: Service
metadata:
  name: sparkdriver
spec:
  type: NodePort
  ports:
  - name: sparkdriver
    port: 9100
  selector:
    app: sparkrunner

When I deploy this to the cluster the driver will start, but when it attempts to find executors things will fail with a socket exception, presumably because the workers can't connect back to the driver, or vice-versa?

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/04/26 20:24:39 INFO SparkContext: Running Spark version 2.4.2
20/04/26 20:24:40 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/04/26 20:24:40 INFO SparkContext: Submitted application: test
20/04/26 20:24:40 INFO SecurityManager: Changing view acls to: root
20/04/26 20:24:40 INFO SecurityManager: Changing modify acls to: root
20/04/26 20:24:40 INFO SecurityManager: Changing view acls groups to: 
20/04/26 20:24:40 INFO SecurityManager: Changing modify acls groups to: 
20/04/26 20:24:40 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
20/04/26 20:24:41 INFO Utils: Successfully started service 'sparkDriver' on port 7077.
20/04/26 20:24:41 INFO SparkEnv: Registering MapOutputTracker
20/04/26 20:24:41 INFO SparkEnv: Registering BlockManagerMaster
20/04/26 20:24:41 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/04/26 20:24:41 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/04/26 20:24:41 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-e8aa33ba-26d2-421d-9957-9cba1c9a3b9f
20/04/26 20:24:41 INFO MemoryStore: MemoryStore started with capacity 1150.2 MB
20/04/26 20:24:41 INFO SparkEnv: Registering OutputCommitCoordinator
20/04/26 20:24:41 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/04/26 20:24:41 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://sparkrunner-0:4040
20/04/26 20:24:53 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 7078.
20/04/26 20:24:53 INFO NettyBlockTransferService: Server created on sparkrunner-0:7078
20/04/26 20:24:53 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/04/26 20:24:53 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, sparkrunner-0, 7078, None)
20/04/26 20:24:53 INFO BlockManagerMasterEndpoint: Registering block manager sparkrunner-0:7078 with 1150.2 MB RAM, BlockManagerId(driver, sparkrunner-0, 7078, None)
20/04/26 20:24:53 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, sparkrunner-0, 7078, None)
20/04/26 20:24:53 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, sparkrunner-0, 7078, None)
20/04/26 20:24:53 WARN WatchConnectionManager: Exec Failure
java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at okhttp3.internal.platform.Platform.connectSocket(Platform.java:129)
    at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:246)
    at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:166)
    at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257)
    at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
    at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:107)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:254)
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:200)
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

From this I can't really tell what's failing - is the issue with the service definition or the driver itself? I've tried fiddling with selectors and hostnames but nothing seems to work.

ANSWER

Answered 2020-Apr-29 at 21:21

After more poking and prodding, I found that the address I was using for the k8s service was incorrect:

k8s://https://kubernetes.default.svc:32768

I got this from a kubectl cluster-info, but my minikube instance may be reporting that incorrectly (or proxying for external perhaps). When I replaced with this:

k8s://https://10.96.0.1:443

which is the internal address of the api, things started to work.

Source https://stackoverflow.com/questions/61449718

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install alluxio

Prebuilt binaries are available to download at https://www.alluxio.io/download .
Please follow the Guide to Get Started to run a simple example with Alluxio.