prediction-try-java-python | Sample application illustrating use of the Google | GCP library

 by   GoogleCloudPlatform Java Version: Current License: Apache-2.0

kandi X-RAY | prediction-try-java-python Summary

prediction-try-java-python is a Java library typically used in Cloud, GCP applications. prediction-try-java-python has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. However prediction-try-java-python build file is not available. You can download it from GitHub.
sample application illustrating use of the google prediction api within the google app engine environmenttry prediction (v1.0). this project provides a complete application illustrating use of the google prediction api within the google app engine environment. sample code is provided for both the java and python app engine runtimes, along with resources for css, javascript, images and config data files, all of which are shared across the two runtime environments. the application presents a simple interactive user experience: select a prediction model, enter a corresponding set of input text and submit your prediction request. for classification models, a graphical response is provided showing the confidence level for each category in the selected model. for regression models, a numerical result is presented. the set of models supported and the corresponding input fields are entirely dynamic and controlled by a
    Support
      Quality
        Security
          License
            Reuse
            Support
              Quality
                Security
                  License
                    Reuse

                      kandi-support Support

                        summary
                        prediction-try-java-python has a low active ecosystem.
                        summary
                        It has 59 star(s) with 36 fork(s). There are 86 watchers for this library.
                        summary
                        It had no major release in the last 6 months.
                        summary
                        There are 6 open issues and 2 have been closed. On average issues are closed in 49 days. There are 1 open pull requests and 0 closed requests.
                        summary
                        It has a neutral sentiment in the developer community.
                        summary
                        The latest version of prediction-try-java-python is current.
                        This Library - Support
                          Best in #GCP
                            Average in #GCP
                            This Library - Support
                              Best in #GCP
                                Average in #GCP

                                  kandi-Quality Quality

                                    summary
                                    prediction-try-java-python has 0 bugs and 0 code smells.
                                    This Library - Quality
                                      Best in #GCP
                                        Average in #GCP
                                        This Library - Quality
                                          Best in #GCP
                                            Average in #GCP

                                              kandi-Security Security

                                                summary
                                                prediction-try-java-python has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
                                                summary
                                                prediction-try-java-python code analysis shows 0 unresolved vulnerabilities.
                                                summary
                                                There are 0 security hotspots that need review.
                                                This Library - Security
                                                  Best in #GCP
                                                    Average in #GCP
                                                    This Library - Security
                                                      Best in #GCP
                                                        Average in #GCP

                                                          kandi-License License

                                                            summary
                                                            prediction-try-java-python is licensed under the Apache-2.0 License. This license is Permissive.
                                                            summary
                                                            Permissive licenses have the least restrictions, and you can use them in most projects.
                                                            This Library - License
                                                              Best in #GCP
                                                                Average in #GCP
                                                                This Library - License
                                                                  Best in #GCP
                                                                    Average in #GCP

                                                                      kandi-Reuse Reuse

                                                                        summary
                                                                        prediction-try-java-python releases are not available. You will need to build from source code and install.
                                                                        summary
                                                                        prediction-try-java-python has no build file. You will be need to create the build yourself to build the component from source.
                                                                        summary
                                                                        It has 2811 lines of code, 15 functions and 22 files.
                                                                        summary
                                                                        It has low code complexity. Code complexity directly impacts maintainability of the code.
                                                                        This Library - Reuse
                                                                          Best in #GCP
                                                                            Average in #GCP
                                                                            This Library - Reuse
                                                                              Best in #GCP
                                                                                Average in #GCP
                                                                                  Top functions reviewed by kandi - BETA
                                                                                  kandi has reviewed prediction-try-java-python and discovered the below as its top functions. This is intended to give you an instant insight into prediction-try-java-python implemented functionality, and help decide if they suit your requirements.
                                                                                  • Handle POST
                                                                                    • Get user credentials
                                                                                    • Parses the given json file
                                                                                    • Forward a request to the given path
                                                                                  • Get server credentials
                                                                                  • Invoked to retrieve the OAuth code
                                                                                  • Handle an HTTP GET request
                                                                                  Get all kandi verified functions for this library.
                                                                                  Get all kandi verified functions for this library.

                                                                                  prediction-try-java-python Key Features

                                                                                  Python 2.5 or later
                                                                                  Google App Engine
                                                                                  Google Python API Client
                                                                                  Command line flags modules for Python
                                                                                  HTTP Client Library for Python
                                                                                  Google OAuth 2.0 Client Library for Python
                                                                                  URI Templates for Python
                                                                                  Java 5 (or higher) standard (SE) and enterprise (EE)
                                                                                  Google App Engine
                                                                                  Maven
                                                                                  Maven Plugin for App Engine
                                                                                  Clone this repo into a new directory.
                                                                                  Customize the following files: In shared/rc/client_secrets.json, replace the placeholder strings with your actual client id and client secret from the Google APIs console. In shared/rc/models.json, enter information about the model(s) you would like to use, following the format shown for the two sample models. Java only: edit the file gae-java/src/main/java/com/google/tryPredictionJava/web/IndexServlet.java to specify your redirect URI, which should be your app’s base URI /auth_return, e.g. http://your-app-name.appspot.com/auth_return. Add your redirect URI (defined in previous step) to the list of valid redirect URIs in the "API Access" tab of the APIs Console. If you miss this step, you’ll get a redirect_uri_mismatch error during initial authorization of the shared server credentials.
                                                                                  Build and deploy your app: For Python: Modify the "application:" line in your app.yaml file to reflect your chosen app name and use the Google App Engine tools to deploy your app. Install google-api-python-client library to your project vendor dir via pip: pip install -t vendor -r requirements.txt Note: this is only required for dev. On production this is done upon deployment For Java: modify the contents of the "application" XML element in your gae-java/src/main/webapp/WEB-INF/appengine-web.xml file to reflect your chosen app name and use the Maven plugin for Google App Engine to deploy your app (you need to run "mvn gae:unpack" once and then you can subsequently deploy your app repeatedly with "mvn gae:deploy").
                                                                                  The first time you access your app, it will step you through the login and OAuth 2.0 sequence, however, all access thereafter, by you or anyone else, will reuse your initially established security credentials. If you ever wish to change or re-establish the shared server credentials, simply visit your service’s URI with the "/reset" suffix (note that the reset service can only be invoked by the application administrator).

                                                                                  prediction-try-java-python Examples and Code Snippets

                                                                                  No Code Snippets are available at this moment for prediction-try-java-python.
                                                                                  Community Discussions

                                                                                  Trending Discussions on GCP

                                                                                  Submit command line arguments to a pyspark job on airflow
                                                                                  chevron right
                                                                                  Skip first line in import statement using gc.open_by_url from gspread (i.e. add header=0)
                                                                                  chevron right
                                                                                  Automatically Grab Latest Google Cloud Platform Secret Version
                                                                                  chevron right
                                                                                  Programmatically Connecting a GitHub repo to a Google Cloud Project
                                                                                  chevron right
                                                                                  Unable to create a new Cloud Function - cloud-client-api-gae
                                                                                  chevron right
                                                                                  TypeScript project failing to deploy to App Engine targeting Node 12 or 14, but works with Node 10
                                                                                  chevron right
                                                                                  Dataproc Java client throws NoSuchMethodError setUseJwtAccessWithScope
                                                                                  chevron right
                                                                                  Apache Beam Cloud Dataflow Streaming Stuck Side Input
                                                                                  chevron right
                                                                                  BIG Query command using BAT file
                                                                                  chevron right
                                                                                  Vertex AI Model Batch prediction, issue with referencing existing model and input file on Cloud Storage
                                                                                  chevron right

                                                                                  QUESTION

                                                                                  Submit command line arguments to a pyspark job on airflow
                                                                                  Asked 2022-Mar-29 at 10:37

                                                                                  I have a pyspark job available on GCP Dataproc to be triggered on airflow as shown below:

                                                                                  config = help.loadJSON("batch/config_file")
                                                                                  
                                                                                  MY_PYSPARK_JOB = {
                                                                                      "reference": {"project_id": "my_project_id"},
                                                                                      "placement": {"cluster_name": "my_cluster_name"},
                                                                                      "pyspark_job": {
                                                                                          "main_python_file_uri": "gs://file/loc/my_spark_file.py"]
                                                                                          "properties": config["spark_properties"]
                                                                                          "args": 
                                                                                      },
                                                                                  }
                                                                                  
                                                                                  

                                                                                  I need to supply command line arguments to this pyspark job as show below [this is how I am running my pyspark job from command line]:

                                                                                  spark-submit gs://file/loc/my_spark_file.py --arg1 val1 --arg2 val2
                                                                                  

                                                                                  I am providing the arguments to my pyspark job using "configparser". Therefore, arg1 is the key and val1 is the value from my spark-submit commant above.

                                                                                  How do I define the "args" param in the "MY_PYSPARK_JOB" defined above [equivalent to my command line arguments]?

                                                                                  ANSWER

                                                                                  Answered 2022-Mar-28 at 08:18

                                                                                  You have to pass a Sequence[str]. If you check DataprocSubmitJobOperator you will see that the params job implements a class google.cloud.dataproc_v1.types.Job.

                                                                                  class DataprocSubmitJobOperator(BaseOperator):
                                                                                  ...
                                                                                      :param job: Required. The job resource. If a dict is provided, it must be of the same form as the protobuf message.
                                                                                      :class:`~google.cloud.dataproc_v1.types.Job` 
                                                                                  

                                                                                  So, on the section about job type pySpark which is google.cloud.dataproc_v1.types.PySparkJob:

                                                                                  args Sequence[str] Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.

                                                                                  Source https://stackoverflow.com/questions/71616491

                                                                                  QUESTION

                                                                                  Skip first line in import statement using gc.open_by_url from gspread (i.e. add header=0)
                                                                                  Asked 2022-Mar-16 at 08:12

                                                                                  What is the equivalent of header=0 in pandas, which recognises the first line as a heading in gspread?

                                                                                  pandas import statement (correct)

                                                                                  import pandas as pd
                                                                                  
                                                                                  # gcp / google sheets URL
                                                                                  df_URL = "https://docs.google.com/spreadsheets/d/1wKtvNfWSjPNC1fNmTfUHm7sXiaPyOZMchjzQBt1y_f8/edit?usp=sharing"
                                                                                  
                                                                                  raw_dataset = pd.read_csv(df_URL, na_values='?',sep=';'
                                                                                                            , skipinitialspace=True, header=0, index_col=None)
                                                                                  

                                                                                  Using the gspread function, so far I import the data, change the first line to the heading then delete the first line after but this recognises everything in the DataFrame as a string. I would like to recognise the first line as a heading right away in the import statement.

                                                                                  gspread import statement that needs header=True equivalent

                                                                                  import pandas as pd
                                                                                  from google.colab import auth
                                                                                  auth.authenticate_user()
                                                                                  import gspread
                                                                                  from oauth2client.client import GoogleCredentials
                                                                                  
                                                                                  
                                                                                  # gcp / google sheets url
                                                                                  df_URL = "https://docs.google.com/spreadsheets/d/1wKtvNfWSjPNC1fNmTfUHm7sXiaPyOZMchjzQBt1y_f8/edit?usp=sharing"
                                                                                  
                                                                                  # importing the data from Google Drive  setup
                                                                                  gc = gspread.authorize(GoogleCredentials.get_application_default())
                                                                                  
                                                                                  # read data and put it in dataframe
                                                                                  g_sheets = gc.open_by_url(df_URL) 
                                                                                  
                                                                                  df = pd.DataFrame(g_sheets.get_worksheet(0).get_all_values())
                                                                                  
                                                                                    
                                                                                  # change first row to header
                                                                                  df = df.rename(columns=df.iloc[0]) 
                                                                                  
                                                                                  # drop first row
                                                                                  df.drop(index=df.index[0], axis=0, inplace=True) 
                                                                                  

                                                                                  ANSWER

                                                                                  Answered 2022-Mar-16 at 08:12

                                                                                  Looking at the API documentation, you probably want to use:

                                                                                  df = pd.DataFrame(g_sheets.get_worksheet(0).get_all_records(head=1))
                                                                                  

                                                                                  The .get_all_records method returns a dictionary of with the column headers as the keys and a list of column values as the dictionary values. The argument head= determines which row to use as keys; rows start from 1 and follow the numeration of the spreadsheet.

                                                                                  Since the values returned by .get_all_records() are lists of strings, the data frame constructor, pd.DataFrame, will return a data frame that is all strings. To convert it to floats, we need to replace the empty strings, and the the dash-only strings ('-') with NA-type values, then convert to float.

                                                                                  Luckily pandas DataFrame has a convenient method for replacing values .replace. We can pass it mapping from the string we want as NAs to None, which gets converted to NaN.

                                                                                  import pandas as pd
                                                                                  
                                                                                  data = g_sheets.get_worksheet(0).get_all_records(head=1)
                                                                                  
                                                                                  na_strings_map= {
                                                                                      '-': None, 
                                                                                      '': None
                                                                                  }
                                                                                  
                                                                                  df = pd.DataFrame(data).replace(na_strings_map).astype(float)
                                                                                  

                                                                                  Source https://stackoverflow.com/questions/71418682

                                                                                  QUESTION

                                                                                  Automatically Grab Latest Google Cloud Platform Secret Version
                                                                                  Asked 2022-Mar-01 at 03:01

                                                                                  I'm trying to grab the latest secret version. Is there a way to do that without specifying the version number? Such as using the keyword "latest". I'm trying to avoid having to iterate through all the secret versions with a for loop as GCP documentation shows:

                                                                                  try (SecretManagerServiceClient client = SecretManagerServiceClient.create()) {
                                                                                    // Build the parent name.
                                                                                    SecretName projectName = SecretName.of(projectId, secretId);
                                                                                  
                                                                                    // Get all versions.
                                                                                    ListSecretVersionsPagedResponse pagedResponse = client.listSecretVersions(projectName);
                                                                                  
                                                                                    // List all versions and their state.
                                                                                    pagedResponse
                                                                                        .iterateAll()
                                                                                        .forEach(
                                                                                            version -> {
                                                                                              System.out.printf("Secret version %s, %s\n", version.getName(), version.getState());
                                                                                            });
                                                                                  }
                                                                                  

                                                                                  ANSWER

                                                                                  Answered 2021-Sep-12 at 18:54
                                                                                  import com.google.cloud.secretmanager.v1.AccessSecretVersionResponse;
                                                                                  import com.google.cloud.secretmanager.v1.SecretManagerServiceClient;
                                                                                  import com.google.cloud.secretmanager.v1.SecretVersionName;
                                                                                  import java.io.IOException;
                                                                                  
                                                                                  public class AccessSecretVersion {
                                                                                  
                                                                                    public static void accessSecretVersion() throws IOException {
                                                                                      // TODO(developer): Replace these variables before running the sample.
                                                                                      String projectId = "your-project-id";
                                                                                      String secretId = "your-secret-id";
                                                                                      String versionId = "latest"; //<-- specify version
                                                                                      accessSecretVersion(projectId, secretId, versionId);
                                                                                    }
                                                                                  
                                                                                    // Access the payload for the given secret version if one exists. The version
                                                                                    // can be a version number as a string (e.g. "5") or an alias (e.g. "latest").
                                                                                    public static void accessSecretVersion(String projectId, String secretId, String versionId)
                                                                                        throws IOException {
                                                                                      // Initialize client that will be used to send requests. This client only needs to be created
                                                                                      // once, and can be reused for multiple requests. After completing all of your requests, call
                                                                                      // the "close" method on the client to safely clean up any remaining background resources.
                                                                                      try (SecretManagerServiceClient client = SecretManagerServiceClient.create()) {
                                                                                        SecretVersionName secretVersionName = SecretVersionName.of(projectId, secretId, versionId);
                                                                                  
                                                                                        // Access the secret version.
                                                                                        AccessSecretVersionResponse response = client.accessSecretVersion(secretVersionName);
                                                                                  
                                                                                        // Print the secret payload.
                                                                                        //
                                                                                        // WARNING: Do not print the secret in a production environment - this
                                                                                        // snippet is showing how to access the secret material.
                                                                                        String payload = response.getPayload().getData().toStringUtf8();
                                                                                        System.out.printf("Plaintext: %s\n", payload);
                                                                                      }
                                                                                    }
                                                                                  }
                                                                                  

                                                                                  Source https://stackoverflow.com/questions/68805240

                                                                                  QUESTION

                                                                                  Programmatically Connecting a GitHub repo to a Google Cloud Project
                                                                                  Asked 2022-Feb-12 at 16:16

                                                                                  I'm working on a Terraform project that will set up all the GCP resources needed for a large project spanning multiple GitHub repos. My goal is to be able to recreate the cloud infrastructure from scratch completely with Terraform.

                                                                                  The issue I'm running into is in order to setup build triggers with Terraform within GCP, the GitHub repo that is setting off the trigger first needs to be connected. Currently, I've only been able to do that manually via the Google Cloud Build dashboard. I'm not sure if this is possible via Terraform or with a script but I'm looking for any solution I can automate this with. Once the projects are connected updating everything with Terraform is working fine.

                                                                                  TLDR; How can I programmatically connect a GitHub project with a GCP project instead of using the dashboard?

                                                                                  ANSWER

                                                                                  Answered 2022-Feb-12 at 16:16

                                                                                  Currently there is no way to programmatically connect a GitHub repo to a Google Cloud Project. This must be done manually via Google Cloud.

                                                                                  My workaround is to manually connect an "admin" project, build containers and save them to that project's artifact registry, and then deploy the containers from the registry in the programmatically generated project.

                                                                                  Source https://stackoverflow.com/questions/69834735

                                                                                  QUESTION

                                                                                  Unable to create a new Cloud Function - cloud-client-api-gae
                                                                                  Asked 2022-Feb-11 at 18:49

                                                                                  I'm unable to create a Cloud Function in my GCP project using GUI, but have admin roles for GCF, SA and IAM.

                                                                                  Here is the error message:

                                                                                  Missing necessary permission iam.serviceAccounts.actAs for cloud-client-api-gae on the service account serviceaccountname@DOMAIN.iam.gserviceaccount.com. Grant the role 'roles/iam.serviceAccountUser' to cloud-client-api-gae on the service account serviceaccountname@DOMAIN.iam.gserviceaccount.com.

                                                                                  cloud-client-api-gae is not an SA nor User on my IAM list. It must be a creature living underneath Graphical User Interfrace.

                                                                                  I have Enabled API for GCF, AppEngine and I have Service Account Admin role.

                                                                                  I had literally 0 search results when googling for cloud-client-api-gae.

                                                                                  ANSWER

                                                                                  Answered 2022-Jan-18 at 13:53

                                                                                  I contacted GCP support and it seems I was missing a single permission for my user: Service Account User - that's it.

                                                                                  PS: Person from support didn't know what this thing called "cloud-client-api-gae" is.

                                                                                  Source https://stackoverflow.com/questions/70756708

                                                                                  QUESTION

                                                                                  TypeScript project failing to deploy to App Engine targeting Node 12 or 14, but works with Node 10
                                                                                  Asked 2022-Jan-16 at 14:32

                                                                                  I have a TypeScript project that has been deployed several times without any problems to Google App Engine, Standard environment, running Node 10. However, when I try to update the App Engine project to either Node 12 or 14 (by editing the engines.node value in package.json and the runtime value in app.yaml), the deploy fails, printing the following to the console:

                                                                                  > ####@1.0.1 prepare /workspace
                                                                                  > npm run gcp-build
                                                                                  
                                                                                  
                                                                                  > ####@1.0.1 gcp-build /workspace
                                                                                  > tsc -p .
                                                                                  
                                                                                  sh: 1: tsc: not found
                                                                                  npm ERR! code ELIFECYCLE
                                                                                  npm ERR! syscall spawn
                                                                                  npm ERR! file sh
                                                                                  npm ERR! errno ENOENT
                                                                                  npm ERR! ####@1.0.1 gcp-build: `tsc -p .`
                                                                                  npm ERR! spawn ENOENT
                                                                                  npm ERR!
                                                                                  npm ERR! Failed at the ####@1.0.1 gcp-build script.
                                                                                  npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
                                                                                  

                                                                                  According to the following documentation, App Engine should be installing the modules listed in devDependencies in package.json prior to running the gcp-build script (which it has been doing, as expected, when deploying with the Node version set to 10).

                                                                                  https://cloud.google.com/appengine/docs/standard/nodejs/running-custom-build-step

                                                                                  I can't find any documentation regarding how this App Engine deployment behavior has changed when targeting Node 12 or 14 rather than Node 10. Is there some configuration I am missing? Or does Google no longer install devDependencies prior to running the gcp-build script?

                                                                                  Here is the devDependencies section of my package.json (TypeScript is there):

                                                                                      "devDependencies": {
                                                                                          "@google-cloud/nodejs-repo-tools": "^3.3.0",
                                                                                          "@types/cors": "^2.8.12",
                                                                                          "@types/express": "^4.17.13",
                                                                                          "@types/pg": "^8.6.1",
                                                                                          "@types/request": "^2.48.7",
                                                                                          "tsc-watch": "^4.4.0",
                                                                                          "tslint": "^6.1.3",
                                                                                          "typescript": "^4.3.5"
                                                                                      },
                                                                                  

                                                                                  ANSWER

                                                                                  Answered 2022-Jan-16 at 14:32

                                                                                  I encountered the exact same problem and just put typescript in dependencies, not devDependencies.

                                                                                  It worked after that, but cannot assure that it is due to this change (since I have no proof of that).

                                                                                  Source https://stackoverflow.com/questions/69153522

                                                                                  QUESTION

                                                                                  Dataproc Java client throws NoSuchMethodError setUseJwtAccessWithScope
                                                                                  Asked 2022-Jan-14 at 19:24

                                                                                  I am following this article ,for submit a job to an existing Dataproc cluster via a Dataproc API

                                                                                  For the following line of code :

                                                                                   // Configure the settings for the job controller client.
                                                                                      JobControllerSettings jobControllerSettings =
                                                                                          JobControllerSettings.newBuilder().setEndpoint(myEndpoint).build();
                                                                                  

                                                                                  I am getting the following errors :

                                                                                  SEVERE: Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.NoSuchMethodError: 'com.google.api.gax.core.GoogleCredentialsProvider$Builder com.google.api.gax.core.GoogleCredentialsProvider$Builder.setUseJwtAccessWithScope(boolean)'] with root cause
                                                                                  java.lang.NoSuchMethodError: 'com.google.api.gax.core.GoogleCredentialsProvider$Builder com.google.api.gax.core.GoogleCredentialsProvider$Builder.setUseJwtAccessWithScope(boolean)'
                                                                                  

                                                                                  In my pom I used the following dependencies :

                                                                                  
                                                                                          
                                                                                              
                                                                                                  com.google.cloud
                                                                                                  libraries-bom
                                                                                                  24.1.2
                                                                                                  pom
                                                                                                  import
                                                                                              
                                                                                  
                                                                                              
                                                                                                  org.springframework.cloud
                                                                                                  spring-cloud-gcp-dependencies
                                                                                                  1.2.8.RELEASE
                                                                                                  pom
                                                                                                  import
                                                                                              
                                                                                  
                                                                                          
                                                                                      
                                                                                  

                                                                                  And added the dataproc

                                                                                  
                                                                                              com.google.cloud
                                                                                              google-cloud-dataproc
                                                                                          
                                                                                  

                                                                                  Any help what I am missing here?

                                                                                  ANSWER

                                                                                  Answered 2022-Jan-14 at 19:22

                                                                                  The method com.google.api.gax.core.GoogleCredentialsProvider$Builder com.google.api.gax.core.GoogleCredentialsProvider$Builder.setUseJwtAccessWithScope(boolean) was introduced in com.google.api:gax in version 2.3.0.

                                                                                  Can you

                                                                                  1. run mvn dependency:tree and confirm that your version of com.google.api:gax is above version 2.3.0?

                                                                                  2. upgrade all Google libraries to the latest version?

                                                                                  Here is a similar issue found on the internet.

                                                                                  Source https://stackoverflow.com/questions/70640346

                                                                                  QUESTION

                                                                                  Apache Beam Cloud Dataflow Streaming Stuck Side Input
                                                                                  Asked 2022-Jan-12 at 13:12

                                                                                  I'm currently building PoC Apache Beam pipeline in GCP Dataflow. In this case, I want to create streaming pipeline with main input from PubSub and side input from BigQuery and store processed data back to BigQuery.

                                                                                  Side pipeline code

                                                                                  side_pipeline = (
                                                                                      p
                                                                                      | "periodic" >> PeriodicImpulse(fire_interval=3600, apply_windowing=True)
                                                                                      | "map to read request" >>
                                                                                          beam.Map(lambda x:beam.io.gcp.bigquery.ReadFromBigQueryRequest(table=side_table))
                                                                                      | beam.io.ReadAllFromBigQuery()
                                                                                  )
                                                                                  

                                                                                  Function with side input code

                                                                                  def enrich_payload(payload, equipments):
                                                                                      id = payload["id"]
                                                                                      for equipment in equipments:
                                                                                          if id == equipment["id"]:
                                                                                              payload["type"] = equipment["type"]
                                                                                              payload["brand"] = equipment["brand"]
                                                                                              payload["year"] = equipment["year"]
                                                                                  
                                                                                              break
                                                                                  
                                                                                      return payload
                                                                                  

                                                                                  Main pipeline code

                                                                                  main_pipeline = (
                                                                                      p
                                                                                      | "read" >> beam.io.ReadFromPubSub(topic="projects/my-project/topics/topiq")
                                                                                      | "bytes to dict" >> beam.Map(lambda x: json.loads(x.decode("utf-8")))
                                                                                      | "transform" >> beam.Map(transform_function)
                                                                                      | "timestamping" >> beam.Map(lambda src: window.TimestampedValue(
                                                                                          src,
                                                                                          dt.datetime.fromisoformat(src["timestamp"]).timestamp()
                                                                                      ))
                                                                                      | "windowing" >> beam.WindowInto(window.FixedWindows(30))
                                                                                  )
                                                                                  
                                                                                  final_pipeline = (
                                                                                      main_pipeline
                                                                                      | "enrich data" >> beam.Map(enrich_payload, equipments=beam.pvalue.AsIter(side_pipeline))
                                                                                      | "store" >> beam.io.WriteToBigQuery(bq_table)
                                                                                  )
                                                                                  
                                                                                  result = p.run()
                                                                                  result.wait_until_finish()
                                                                                  

                                                                                  After deploy it to Dataflow, everything looks fine and no error. But then I noticed that enrich data step has two nodes instead of one.

                                                                                  And also, the side input stuck as you can see it has Elements Added with 21 counts in Input Collections and - value in Elements Added in Output Collections.

                                                                                  You can find the full pipeline code here and mock pubsub publisher here

                                                                                  I already follow all instruction in these documentations:

                                                                                  Yet still found this error. Please help me. Thanks!

                                                                                  ANSWER

                                                                                  Answered 2022-Jan-12 at 13:12

                                                                                  Here you have a working example:

                                                                                  mytopic = ""
                                                                                  sql = "SELECT station_id, CURRENT_TIMESTAMP() timestamp FROM `bigquery-public-data.austin_bikeshare.bikeshare_stations` LIMIT 10"
                                                                                  
                                                                                  def to_bqrequest(e, sql):
                                                                                      from apache_beam.io import ReadFromBigQueryRequest
                                                                                      yield ReadFromBigQueryRequest(query=sql)
                                                                                       
                                                                                  
                                                                                  def merge(e, side):
                                                                                      for i in side:
                                                                                          yield f"Main {e.decode('utf-8')} Side {i}"
                                                                                  
                                                                                  pubsub = p | "Read PubSub topic" >> ReadFromPubSub(topic=mytopic)
                                                                                  
                                                                                  side_pcol = (p | PeriodicImpulse(fire_interval=300, apply_windowing=False)
                                                                                                 | "ApplyGlobalWindow" >> WindowInto(window.GlobalWindows(),
                                                                                                                             trigger=trigger.Repeatedly(trigger.AfterProcessingTime(5)),
                                                                                                                             accumulation_mode=trigger.AccumulationMode.DISCARDING)
                                                                                                 | "To BQ Request" >> ParDo(to_bqrequest, sql=sql)
                                                                                                 | ReadAllFromBigQuery()
                                                                                              )
                                                                                  
                                                                                  final = (pubsub | "Merge" >> ParDo(merge, side=beam.pvalue.AsList(side_pcol))
                                                                                                  | Map(logging.info)
                                                                                          )                    
                                                                                      
                                                                                  p.run()
                                                                                  

                                                                                  Note this uses a GlobalWindow (so that both inputs have the same window). I used a processing time trigger so that the pane contains multiple rows. 5 was chosen arbitrarily, using 1 would work too.

                                                                                  Please note matching the data between side and main inputs is non deterministic, and you may see fluctuating values from older fired panes.

                                                                                  In theory, using FixedWindows should fix this, but I cannot get the FixedWindows to work.

                                                                                  Source https://stackoverflow.com/questions/70561769

                                                                                  QUESTION

                                                                                  BIG Query command using BAT file
                                                                                  Asked 2022-Jan-09 at 15:24
                                                                                  echo Give yearmonth "yyyyMM"
                                                                                  setlocal enabledelayedexpansion
                                                                                  SET /p yearmonth= 
                                                                                  SET ClientName[0]=abc
                                                                                  SET ClientName[1]=def
                                                                                  
                                                                                  SET i = 0
                                                                                  
                                                                                  :myLoop
                                                                                  if defined ClientName[%i%] (
                                                                                      call bq query --use_legacy_sql=false "CREATE EXTERNAL TABLE `test.!ClientName[%%i]!.%yearmonth%` OPTIONS (format = 'CSV',skip_leading_rows = 1 uris = ['gs://test/!ClientName[%%i]!/AWS/%yearmonth%/Metrics/data/*.csv'])"
                                                                                      set /a "i+=1"
                                                                                      GOTO :myLoop
                                                                                  
                                                                                  )
                                                                                  

                                                                                  Hi, I am trying to create a batch so that i can run Multiple BIG QUERY at once. Above i tried to write a batch script putting command in a loop .

                                                                                  I am trying to create a table by using yearmonth as user input and then create array to create a table with different client name .

                                                                                  1. But I am unable to print if i =0 ClientName[i] = abc in a call query i am using !ClientName[%%i]! to print but its not working.

                                                                                  2. Call query inside loop is not running in GCP console, when i executed the bat file .

                                                                                  Can you please help me resolve this

                                                                                  ANSWER

                                                                                  Answered 2022-Jan-09 at 11:04
                                                                                  1. It is bad practice to set variables as standalone alphabetical characters like i. One reason is exactly as you have experienced, you have confused for metavariable %%i with a set variable %i%.

                                                                                  2. You are expanding in the loop, but have not enabledelayedexpansion so there are 2 ways, which we will get to in a second.

                                                                                  3. setting variables should not have spaces before or after = excluding the likes of set /a

                                                                                  So, Method 1, without delayedexpansion (note how the variables are used with double %% in the loop with the call command).

                                                                                  @echo off
                                                                                  echo Give yearmonth "yyyyMM"
                                                                                  SET /p yearmonth= 
                                                                                  SET ClientName[0]=abc
                                                                                  SET ClientName[1]=def
                                                                                  
                                                                                  SET num=0
                                                                                  
                                                                                  :myLoop
                                                                                  if defined ClientName[%num%] (
                                                                                      call bq query --use_legacy_sql=false "CREATE EXTERNAL TABLE `test.%%ClientName[%num%]%%.%yearmonth%` OPTIONS (format = 'CSV',skip_leading_rows = 1 uris = ['gs://test/%%ClientName[%num%]%%/AWS/%yearmonth%/Metrics/data/*.csv'])"
                                                                                      set /a num+=1
                                                                                      GOTO :myLoop
                                                                                  
                                                                                  )
                                                                                  

                                                                                  Method 2: (better method using delayedexpansion)

                                                                                  @echo off
                                                                                  setlocal enabledelayedexpansion
                                                                                  echo Give yearmonth "yyyyMM"
                                                                                  SET /p yearmonth= 
                                                                                  SET ClientName[0]=abc
                                                                                  SET ClientName[1]=def
                                                                                  
                                                                                  SET num=0
                                                                                  
                                                                                  :myLoop
                                                                                  if defined ClientName[%num%] (
                                                                                      call bq query --use_legacy_sql=false "CREATE EXTERNAL TABLE `test.!ClientName[%num%]!.%yearmonth%` OPTIONS (format = 'CSV',skip_leading_rows = 1 uris = ['gs://test/!ClientName[%num%]!/AWS/%yearmonth%/Metrics/data/*.csv'])"
                                                                                      set /a num+=1
                                                                                      GOTO :myLoop
                                                                                  
                                                                                  )
                                                                                  

                                                                                  Source https://stackoverflow.com/questions/70640411

                                                                                  QUESTION

                                                                                  Vertex AI Model Batch prediction, issue with referencing existing model and input file on Cloud Storage
                                                                                  Asked 2021-Dec-21 at 14:35

                                                                                  I'm struggling to correctly set Vertex AI pipeline which does the following:

                                                                                  1. read data from API and store to GCS and as as input for batch prediction.
                                                                                  2. get an existing model (Video classification on Vertex AI)
                                                                                  3. create Batch prediction job with input from point 1.
                                                                                    As it will be seen, I don't have much experience with Vertex Pipelines/Kubeflow thus I'm asking for help/advice, hope it's just some beginner mistake. this is the gist of the code I'm using as pipeline
                                                                                  from google_cloud_pipeline_components import aiplatform as gcc_aip
                                                                                  from kfp.v2 import dsl
                                                                                  
                                                                                  from kfp.v2.dsl import component
                                                                                  from kfp.v2.dsl import (
                                                                                      Output,
                                                                                      Artifact,
                                                                                      Model,
                                                                                  )
                                                                                  
                                                                                  PROJECT_ID = 'my-gcp-project'
                                                                                  BUCKET_NAME = "mybucket"
                                                                                  PIPELINE_ROOT = "{}/pipeline_root".format(BUCKET_NAME)
                                                                                  
                                                                                  
                                                                                  @component
                                                                                  def get_input_data() -> str:
                                                                                      # getting data from API, save to Cloud Storage
                                                                                      # return GS URI
                                                                                      gcs_batch_input_path = 'gs://somebucket/file'
                                                                                      return gcs_batch_input_path
                                                                                  
                                                                                  
                                                                                  @component(
                                                                                      base_image="python:3.9",
                                                                                      packages_to_install=['google-cloud-aiplatform==1.8.0']
                                                                                  )
                                                                                  def load_ml_model(project_id: str, model: Output[Artifact]):
                                                                                      """Load existing Vertex model"""
                                                                                      import google.cloud.aiplatform as aip
                                                                                  
                                                                                      model_id = '1234'
                                                                                      model = aip.Model(model_name=model_id, project=project_id, location='us-central1')
                                                                                  
                                                                                  
                                                                                  
                                                                                  @dsl.pipeline(
                                                                                      name="batch-pipeline", pipeline_root=PIPELINE_ROOT,
                                                                                  )
                                                                                  def pipeline(gcp_project: str):
                                                                                      input_data = get_input_data()
                                                                                      ml_model = load_ml_model(gcp_project)
                                                                                  
                                                                                      gcc_aip.ModelBatchPredictOp(
                                                                                          project=PROJECT_ID,
                                                                                          job_display_name=f'test-prediction',
                                                                                          model=ml_model.output,
                                                                                          gcs_source_uris=[input_data.output],  # this doesn't work
                                                                                          # gcs_source_uris=['gs://mybucket/output/'],  # hardcoded gs uri works
                                                                                          gcs_destination_output_uri_prefix=f'gs://{PIPELINE_ROOT}/prediction_output/'
                                                                                      )
                                                                                  
                                                                                  
                                                                                  if __name__ == '__main__':
                                                                                      from kfp.v2 import compiler
                                                                                      import google.cloud.aiplatform as aip
                                                                                      pipeline_export_filepath = 'test-pipeline.json'
                                                                                      compiler.Compiler().compile(pipeline_func=pipeline,
                                                                                                                  package_path=pipeline_export_filepath)
                                                                                      # pipeline_params = {
                                                                                      #     'gcp_project': PROJECT_ID,
                                                                                      # }
                                                                                      # job = aip.PipelineJob(
                                                                                      #     display_name='test-pipeline',
                                                                                      #     template_path=pipeline_export_filepath,
                                                                                      #     pipeline_root=f'gs://{PIPELINE_ROOT}',
                                                                                      #     project=PROJECT_ID,
                                                                                      #     parameter_values=pipeline_params,
                                                                                      # )
                                                                                  
                                                                                      # job.run()
                                                                                  

                                                                                  When running the pipeline it throws this exception when running Batch prediction:
                                                                                  details = "List of found errors: 1.Field: batch_prediction_job.model; Message: Invalid Model resource name. so I'm not sure what could be wrong. I tried to load model in the notebook (outside of component) and it correctly returns.

                                                                                  Second issue I'm having is referencing GCS URI as output from component to batch job input.

                                                                                     input_data = get_input_data2()
                                                                                     gcc_aip.ModelBatchPredictOp(
                                                                                          project=PROJECT_ID,
                                                                                          job_display_name=f'test-prediction',
                                                                                          model=ml_model.output,
                                                                                          gcs_source_uris=[input_data.output],  # this doesn't work
                                                                                          # gcs_source_uris=['gs://mybucket/output/'],  # hardcoded gs uri works
                                                                                          gcs_destination_output_uri_prefix=f'gs://{PIPELINE_ROOT}/prediction_output/'
                                                                                      )
                                                                                  

                                                                                  During compilation, I get following exception TypeError: Object of type PipelineParam is not JSON serializable, though I think this could be issue of ModelBatchPredictOp component.

                                                                                  Again any help/advice appreciated, I'm dealing with this from yesterday, so maybe I missed something obvious.

                                                                                  libraries I'm using:

                                                                                  google-cloud-aiplatform==1.8.0  
                                                                                  google-cloud-pipeline-components==0.2.0  
                                                                                  kfp==1.8.10  
                                                                                  kfp-pipeline-spec==0.1.13  
                                                                                  kfp-server-api==1.7.1
                                                                                  

                                                                                  UPDATE After comments, some research and tuning, for referencing model this works:

                                                                                  @component
                                                                                  def load_ml_model(project_id: str, model: Output[Artifact]):
                                                                                      region = 'us-central1'
                                                                                      model_id = '1234'
                                                                                      model_uid = f'projects/{project_id}/locations/{region}/models/{model_id}'
                                                                                      model.uri = model_uid
                                                                                      model.metadata['resourceName'] = model_uid
                                                                                  

                                                                                  and then I can use it as intended:

                                                                                  batch_predict_op = gcc_aip.ModelBatchPredictOp(
                                                                                          project=gcp_project,
                                                                                          job_display_name=f'batch-prediction-test',
                                                                                          model=ml_model.outputs['model'],
                                                                                          gcs_source_uris=[input_batch_gcs_path],
                                                                                  gcs_destination_output_uri_prefix=f'gs://{BUCKET_NAME}/prediction_output/test'
                                                                                      )
                                                                                  

                                                                                  UPDATE 2 regarding GCS path, a workaround is to define path outside of the component and pass it as an input parameter, for example (abbreviated):

                                                                                  @dsl.pipeline(
                                                                                      name="my-pipeline",
                                                                                      pipeline_root=PIPELINE_ROOT,
                                                                                  )
                                                                                  def pipeline(
                                                                                          gcp_project: str,
                                                                                          region: str,
                                                                                          bucket: str
                                                                                  ):
                                                                                      ts = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
                                                                                      
                                                                                      gcs_prediction_input_path = f'gs://{BUCKET_NAME}/prediction_input/video_batch_prediction_input_{ts}.jsonl'
                                                                                      batch_input_data_op = get_input_data(gcs_prediction_input_path)  # this loads input data to GCS path
                                                                                  
                                                                                      batch_predict_op = gcc_aip.ModelBatchPredictOp(
                                                                                          project=gcp_project,
                                                                                          model=training_job_run_op.outputs["model"],
                                                                                          job_display_name='batch-prediction',
                                                                                          # gcs_source_uris=[batch_input_data_op.output],
                                                                                          gcs_source_uris=[gcs_prediction_input_path],
                                                                                          gcs_destination_output_uri_prefix=f'gs://{BUCKET_NAME}/prediction_output/',
                                                                                      ).after(batch_input_data_op)  # we need to add 'after' so it runs after input data is prepared since get_input_data doesn't returns anything
                                                                                  
                                                                                  

                                                                                  still not sure, why it doesn't work/compile when I return GCS path from get_input_data component

                                                                                  ANSWER

                                                                                  Answered 2021-Dec-21 at 14:35

                                                                                  I'm glad you solved most of your main issues and found a workaround for model declaration.

                                                                                  For your input.output observation on gcs_source_uris, the reason behind it is because the way the function/class returns the value. If you dig inside the class/methods of google_cloud_pipeline_components you will find that it implements a structure that will allow you to use .outputs from the returned value of the function called.

                                                                                  If you go to the implementation of one of the components of the pipeline you will find that it returns an output array from convert_method_to_component function. So, in order to have that implemented in your custom class/function your function should return a value which can be called as an attribute. Below is a basic implementation of it.

                                                                                  class CustomClass():
                                                                                       def __init__(self):
                                                                                         self.return_val = {'path':'custompath','desc':'a desc'}
                                                                                        
                                                                                       @property
                                                                                       def output(self):
                                                                                         return self.return_val 
                                                                                  
                                                                                  hello = CustomClass()
                                                                                  print(hello.output['path'])
                                                                                  

                                                                                  If you want to dig more about it you can go to the following pages:

                                                                                  Source https://stackoverflow.com/questions/70356856

                                                                                  Community Discussions, Code Snippets contain sources that include Stack Exchange Network

                                                                                  Vulnerabilities

                                                                                  No vulnerabilities reported

                                                                                  Install prediction-try-java-python

                                                                                  You can download it from GitHub.
                                                                                  You can use prediction-try-java-python like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the prediction-try-java-python component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

                                                                                  Support

                                                                                  For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .
                                                                                  Find more information at:
                                                                                  Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
                                                                                  Find more libraries
                                                                                  Explore Kits - Develop, implement, customize Projects, Custom Functions and Applications with kandi kits​
                                                                                  Save this library and start creating your kit
                                                                                  CLONE
                                                                                • HTTPS

                                                                                  https://github.com/GoogleCloudPlatform/prediction-try-java-python.git

                                                                                • CLI

                                                                                  gh repo clone GoogleCloudPlatform/prediction-try-java-python

                                                                                • sshUrl

                                                                                  git@github.com:GoogleCloudPlatform/prediction-try-java-python.git

                                                                                • Share this Page

                                                                                  share link

                                                                                  Explore Related Topics

                                                                                  Consider Popular GCP Libraries

                                                                                  microservices-demo

                                                                                  by GoogleCloudPlatform

                                                                                  awesome-kubernetes

                                                                                  by ramitsurana

                                                                                  go-cloud

                                                                                  by google

                                                                                  infracost

                                                                                  by infracost

                                                                                  python-docs-samples

                                                                                  by GoogleCloudPlatform

                                                                                  Try Top Libraries by GoogleCloudPlatform

                                                                                  microservices-demo

                                                                                  by GoogleCloudPlatformPython

                                                                                  terraformer

                                                                                  by GoogleCloudPlatformGo

                                                                                  training-data-analyst

                                                                                  by GoogleCloudPlatformJupyter Notebook

                                                                                  python-docs-samples

                                                                                  by GoogleCloudPlatformJupyter Notebook

                                                                                  golang-samples

                                                                                  by GoogleCloudPlatformGo

                                                                                  Compare GCP Libraries with Highest Support

                                                                                  Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
                                                                                  Find more libraries
                                                                                  Explore Kits - Develop, implement, customize Projects, Custom Functions and Applications with kandi kits​
                                                                                  Save this library and start creating your kit