dsub | source command-line tool | BPM library

You can install dsub from PyPI, or you can clone and install from github.
Choose one of the following:.
If necessary, install pip.
Install dsub pip install dsub
Be sure you have git installed Instructions for your environment can be found on the git website.
Clone this repository. git clone https://github.com/DataBiosphere/dsub cd dsub
Install dsub (this will also install the dependencies) python setup.py install
Set up Bash tab completion (optional). source bash_tab_complete
We think you'll find the local provider to be very helpful when building your dsub tasks. Instead of submitting a request to run your command on a cloud VM, the local provider runs your dsub tasks on your local machine. The local provider is not designed for running at scale. It is designed to emulate running on a cloud VM such that you can rapidly iterate. You'll get quicker turnaround times and won't incur cloud charges using it.
Run a dsub job and wait for completion. Here is a very simple "Hello World" test: dsub \ --provider local \ --logging "${TMPDIR:-/tmp}/dsub-test/logging/" \ --output OUT="${TMPDIR:-/tmp}/dsub-test/output/out.txt" \ --command 'echo "Hello World" > "${OUT}"' \ --wait Note: TMPDIR is commonly set to /tmp by default on most Unix systems, although it is also often left unset. On some versions of MacOS TMPDIR is set to a location under /var/folders. Note: The above syntax ${TMPDIR:-/tmp} is known to be supported by Bash, zsh, ksh. The shell will expand TMPDIR, but if it is unset, /tmp will be used.
View the output file. cat "${TMPDIR:-/tmp}/dsub-test/output/out.txt"
dsub supports the use of two different APIs from Google Cloud for running tasks. Google Cloud is transitioning from Genomics v2alpha1 to Cloud Life Sciences v2beta. dsub supports both APIs with the (old) google-v2 and (new) google-cls-v2 providers respectively. google-v2 is the current default provider. dsub will be transitioning to make google-cls-v2 the default in coming releases.
Sign up for a Google account and create a project.
Enable the APIs: For the v2alpha1 API (provider: google-v2): Enable the Genomics, Storage, and Compute APIs. For the v2beta API (provider: google-cls-v2): Enable the Cloud Life Sciences, Storage, and Compute APIs
Install the Google Cloud SDK and run gcloud init This will set up your default project and grant credentials to the Google Cloud SDK. Now provide credentials so dsub can call Google APIs: gcloud auth application-default login
Create a Google Cloud Storage bucket. The dsub logs and output files will be written to a bucket. Create a bucket using the storage browser or run the command-line utility gsutil, included in the Cloud SDK. gsutil mb gs://my-bucket Change my-bucket to a unique name that follows the bucket-naming conventions. (By default, the bucket will be in the US, but you can change or refine the location setting with the -l option.)
Run a very simple "Hello World" dsub job and wait for completion. For the v2alpha1 API (provider: google-v2): dsub \ --provider google-v2 \ --project my-cloud-project \ --regions us-central1 \ --logging gs://my-bucket/logging/ \ --output OUT=gs://my-bucket/output/out.txt \ --command 'echo "Hello World" > "${OUT}"' \ --wait Change my-cloud-project to your Google Cloud project, and my-bucket to the bucket you created above. For the v2beta API (provider: google-cls-v2): dsub \ --provider google-cls-v2 \ --project my-cloud-project \ --regions us-central1 \ --logging gs://my-bucket/logging/ \ --output OUT=gs://my-bucket/output/out.txt \ --command 'echo "Hello World" > "${OUT}"' \ --wait Change my-cloud-project to your Google Cloud project, and my-bucket to the bucket you created above. The output of the script command will be written to the OUT file in Cloud Storage that you specify.
View the output file. gsutil cat gs://my-bucket/output/out.txt