elephas | Distributed Deep learning with Keras & Spark
kandi X-RAY | elephas Summary
kandi X-RAY | elephas Summary
Elephas brings deep learning with Keras to Spark. Elephas intends to keep the simplicity and high usability of Keras, thereby allowing for fast prototyping of distributed models, which can be run on massive data sets. For an introductory example, see the following iPython notebook. ἐλέφας is Greek for ivory and an accompanying project to κέρας, meaning horn. If this seems weird mentioning, like a bad dream, you should confirm it actually is at the Keras documentation. Elephas also means elephant, as in stuffed yellow elephant. Elephas implements a class of data-parallel algorithms on top of Keras, using Spark's RDDs and data frames. Keras Models are initialized on the driver, then serialized and shipped to workers, alongside with data and broadcasted model parameters. Spark workers deserialize the model, train their chunk of data and send their gradients back to the driver. The "master" model on the driver is updated by an optimizer, which takes gradients either synchronously or asynchronously.
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Process a docstring
- Count the number of leading spaces in a string
- Process a list block
- Start the Flask service
- Release the lock
- Acquire a read lock
- Render a function
- Process docstring
- Compute the keras model
- Fit the model
- Transform a Pandas DataFrame into a numpy array
- Convert a vector
- Train the model
- Subtract two parameters
- Return the parameters
- Listen for incoming messages
- Convert a class to a source link
- Convert features to a pandas dataframe
- Update parameters
- Return the signature of a class
- Estimates the model from labeled points
- Load data from a CSV file
- Fit the model
- Convert features and labels to LabeledPoints
- Collect method methods
- Read page data
- Create a simple RDD of features and labels
- Start the server
elephas Key Features
elephas Examples and Code Snippets
Lines of Code : 23
License : No License
Copy./efs-client.exe
______ ______ ______
/\ ___\ /\ ___\ /\ ___\
\ \ __\ \ \ __\ \ \___ \
\ \_____\ \ \_\ \/\_____\
\/_____/ \/_/ \/_____/
---------- account ----------
namenode_addr: 192.168.0.179
namenode_port: ./efs-server datanode DataNodeConfig01.yaml
./efs-server datanode DataNodeConfig02.yaml
./efs-server namenode NameNodeConfig.yaml
class MyGenerator(object):
def __init__(self, spark_df, buffer_size, feature_col='features', label_col='labels'):
w = Window().partitionBy(sf.lit('a')).orderBy(sf.lit('a'))
self.df = (
spark_df.withColumn('i!pip install q keras==2.2.4
!pip install q tensorflow==1.14.0
gradients = rdd.mapPartitions(worker.train).collect()
# "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py"
# line 1152
def __reduce__(self):
# The implementation mirrors thaSPARK_CLASSPATH=./path/to/mongo-hadoop-core.jar pyspark
sc = SparkContext(conf=sparkConf)
mongo_conf = {
"mongo.input.uri": "mongodb://..."
"mongo.input.query": s"...mongo query here..."
}
rdd = sc.newAPIvirtualenv venv --relocatable
cd venv
zip -qr ../venv.zip *
PYSPARK_PYTHON=./SP/bin/python spark-submit --master yarn --deploy-mode cluster --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./SP/bin/python --driver-memory 4G --archives venv.Community Discussions
Trending Discussions on elephas
QUESTION
I use PySpark and Elephas but it's not working at the moment. I tried the example given on Elephas' doc Github. Please note, in the PySpark console, my code with Keras and Pandas works (but without using the PySpark library). But the example given on https://github.com/maxpumperla/elephas to interface Keras and the PySpark library with Elephas, doesn't work, and I don't know how to fix this problem at all. All my PySpark configuration is using Python 3.7
Here is the content of my script and the error message:
...ANSWER
Answered 2020-May-03 at 10:32After some research, I switched to java 8 and deleted my java 11 installation. Then, I manually rewrote all my installation under python2.7. Now I think it works. I also had to adapt the script a bit better to fit my x_train and y_train. I used the predict() function of keras to get an array that I think is consistent.
Java 11 doesn't work with Spark 2.4, apparently it works fine with PySpark 3, check it out.
QUESTION
When I launch my ANN script, everything works fine at console level, but nothing changes on the Spark web interface: The application is not displayed in Running Applications or Completed Applications. I created a config file spark-defaults.conf in which I put:
...ANSWER
Answered 2020-May-03 at 10:26I found a solution: In fact spark takes into account first the script config, then the command config, then the general config. However, in my script, I had put
QUESTION
I am working on the following dataset which is a Churn prediction problem: https://www.kaggle.com/jpacse/telecom-churn-new-cell2cell-dataset
I am using pyspark, keras & Elephas to build a distributed neural network model using pyspark pipeline.
When I fit the dataset in the pipeline I get the pickling error. I am following this link to build a model: https://github.com/aviolante/pyspark_dl_pipeline/blob/master/pyspark_dl_pipeline.ipynb
The line on which I am getting the error in my code is:
...ANSWER
Answered 2020-Apr-23 at 01:21The solution which worked for me is found here:
https://github.com/maxpumperla/elephas/issues/151
I downgraded my keras and tensorflow version using the following commands:
QUESTION
Suppose that i have the following data:
...ANSWER
Answered 2020-Feb-01 at 16:55Keep this as your base.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install elephas
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page
