Support
Quality
Security
License
Reuse
kandi has reviewed prediction-try-java-python and discovered the below as its top functions. This is intended to give you an instant insight into prediction-try-java-python implemented functionality, and help decide if they suit your requirements.
Python 2.5 or later
Google App Engine
Google Python API Client
Command line flags modules for Python
HTTP Client Library for Python
Google OAuth 2.0 Client Library for Python
URI Templates for Python
Java 5 (or higher) standard (SE) and enterprise (EE)
Google App Engine
Maven
Maven Plugin for App Engine
Clone this repo into a new directory.
Customize the following files: In shared/rc/client_secrets.json, replace the placeholder strings with your actual client id and client secret from the Google APIs console. In shared/rc/models.json, enter information about the model(s) you would like to use, following the format shown for the two sample models. Java only: edit the file gae-java/src/main/java/com/google/tryPredictionJava/web/IndexServlet.java to specify your redirect URI, which should be your app’s base URI /auth_return, e.g. http://your-app-name.appspot.com/auth_return. Add your redirect URI (defined in previous step) to the list of valid redirect URIs in the "API Access" tab of the APIs Console. If you miss this step, you’ll get a redirect_uri_mismatch error during initial authorization of the shared server credentials.
Build and deploy your app: For Python: Modify the "application:" line in your app.yaml file to reflect your chosen app name and use the Google App Engine tools to deploy your app. Install google-api-python-client library to your project vendor dir via pip: pip install -t vendor -r requirements.txt Note: this is only required for dev. On production this is done upon deployment For Java: modify the contents of the "application" XML element in your gae-java/src/main/webapp/WEB-INF/appengine-web.xml file to reflect your chosen app name and use the Maven plugin for Google App Engine to deploy your app (you need to run "mvn gae:unpack" once and then you can subsequently deploy your app repeatedly with "mvn gae:deploy").
The first time you access your app, it will step you through the login and OAuth 2.0 sequence, however, all access thereafter, by you or anyone else, will reuse your initially established security credentials. If you ever wish to change or re-establish the shared server credentials, simply visit your service’s URI with the "/reset" suffix (note that the reset service can only be invoked by the application administrator).
No Code Snippets are available at this moment for prediction-try-java-python.
QUESTION
Submit command line arguments to a pyspark job on airflow
Asked 2022-Mar-29 at 10:37I have a pyspark job available on GCP Dataproc to be triggered on airflow as shown below:
config = help.loadJSON("batch/config_file")
MY_PYSPARK_JOB = {
"reference": {"project_id": "my_project_id"},
"placement": {"cluster_name": "my_cluster_name"},
"pyspark_job": {
"main_python_file_uri": "gs://file/loc/my_spark_file.py"]
"properties": config["spark_properties"]
"args": <TO_BE_ADDED>
},
}
I need to supply command line arguments to this pyspark job as show below [this is how I am running my pyspark job from command line]:
spark-submit gs://file/loc/my_spark_file.py --arg1 val1 --arg2 val2
I am providing the arguments to my pyspark job using "configparser". Therefore, arg1 is the key and val1 is the value from my spark-submit commant above.
How do I define the "args" param in the "MY_PYSPARK_JOB" defined above [equivalent to my command line arguments]?
ANSWER
Answered 2022-Mar-28 at 08:18You have to pass a Sequence[str]
. If you check DataprocSubmitJobOperator you will see that the params job
implements a class google.cloud.dataproc_v1.types.Job.
class DataprocSubmitJobOperator(BaseOperator):
...
:param job: Required. The job resource. If a dict is provided, it must be of the same form as the protobuf message.
:class:`~google.cloud.dataproc_v1.types.Job`
So, on the section about job type pySpark
which is google.cloud.dataproc_v1.types.PySparkJob:
args Sequence[str] Optional. The arguments to pass to the driver. Do not include arguments, such as
--conf
, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
No vulnerabilities reported
Save this library and start creating your kit
Save this library and start creating your kit