SparkNet | Distributed Neural Networks for Spark

You will probably have to change several fields in this command. For example, the flags --key-pair and --identity-file specify the key pair you will use to connect to the cluster. The flag --slaves specifies the number of Spark workers.
Create an AWS secret key and access key. Instructions here.
Run export AWS_SECRET_ACCESS_KEY= and export AWS_ACCESS_KEY_ID= with the relevant values.
Clone our repository locally.
Start a 5-worker Spark cluster on EC2 by running SparkNet/ec2/spark-ec2 --key-pair=key \ --identity-file=key.pem \ --region=eu-west-1 \ --zone=eu-west-1c \ --instance-type=g2.8xlarge \ --ami=ami-d0833da3 \ --copy-aws-credentials \ --spark-version=1.5.0 \ --spot-price=1.5 \ --no-ganglia \ --user-data SparkNet/ec2/cloud-config.txt \ --slaves=5 \ launch sparknet
SSH to the Spark master as root.
Run bash /root/SparkNet/data/cifar10/get_cifar10.sh to get the Cifar data
Train Cifar on 5 workers using /root/spark/bin/spark-submit --class apps.CifarApp /root/SparkNet/target/scala-2.10/sparknet-assembly-0.1-SNAPSHOT.jar 5
That's all! Information is logged on the master in /root/SparkNet/training_log*.txt.
Obtain the ImageNet data by following the instructions here with wget http://.../ILSVRC2012_img_train.tar wget http://.../ILSVRC2012_img_val.tar This involves creating an account and submitting a request.
On the Spark master, create ~/.aws/credentials with the following content: [default] aws_access_key_id= aws_secret_access_key= and fill in the two fields.
Copy this to the workers with ~/spark-ec2/copy-dir ~/.aws (copy this command exactly because it is somewhat sensitive to the trailing backslashes and that kind of thing).
Create an Amazon S3 bucket with name S3_BUCKET.
Upload the ImageNet data in the appropriate format to S3 with the command python $SPARKNET_HOME/scripts/put_imagenet_on_s3.py $S3_BUCKET \ --train_tar_file=/path/to/ILSVRC2012_img_train.tar \ --val_tar_file=/path/to/ILSVRC2012_img_val.tar \ --new_width=256 \ --new_height=256 This command resizes the images to 256x256, shuffles the training data, and tars the validation files into chunks.
Train ImageNet on 5 workers using /root/spark/bin/spark-submit --class apps.ImageNetApp /root/SparkNet/target/scala-2.10/sparknet-assembly-0.1-SNAPSHOT.jar 5 $S3_BUCKET
We have built the JavaCPP binaries for a couple platforms. They are stored at the following locations:.
Ubuntu with GPUs: http://www.eecs.berkeley.edu/~rkn/snapshot-2016-03-05/
Ubuntu with CPUs: http://www.eecs.berkeley.edu/~rkn/snapshot-2016-03-16-CPU/
CentOS 6 with CPUs: http://www.eecs.berkeley.edu/~rkn/snapshot-2016-03-23-CENTOS6-CPU/