prediction-try-java-python | Sample application illustrating use | GCP library

by GoogleCloudPlatform Java Version: Current License: Apache-2.0

Download this library from

kandi X-RAY | prediction-try-java-python Summary

prediction-try-java-python is a Java library typically used in Cloud, GCP applications. prediction-try-java-python has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. However prediction-try-java-python build file is not available. You can download it from GitHub.

sample application illustrating use of the google prediction api within the google app engine environmenttry prediction (v1.0). this project provides a complete application illustrating use of the google prediction api within the google app engine environment. sample code is provided for both the java and python app engine runtimes, along with resources for css, javascript, images and config data files, all of which are shared across the two runtime environments. the application presents a simple interactive user experience: select a prediction model, enter a corresponding set of input text and submit your prediction request. for classification models, a graphical response is provided showing the confidence level for each category in the selected model. for regression models, a numerical result is presented. the set of models supported and the corresponding input fields are entirely dynamic and controlled by a

Support

Quality

Security

License

Reuse

kandi-support Support

prediction-try-java-python has a low active ecosystem.
It has 59 star(s) with 36 fork(s). There are 86 watchers for this library.
It had no major release in the last 12 months.
There are 6 open issues and 2 have been closed. On average issues are closed in 49 days. There are 1 open pull requests and 0 closed requests.
It has a neutral sentiment in the developer community.
The latest version of prediction-try-java-python is current.

This Library - Support

Best in #GCP

Average in #GCP

This Library - Support

Best in #GCP

Average in #GCP

quality kandi Quality

prediction-try-java-python has 0 bugs and 0 code smells.

This Library - Quality

Best in #GCP

Average in #GCP

This Library - Quality

Best in #GCP

Average in #GCP

security Security

prediction-try-java-python has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
prediction-try-java-python code analysis shows 0 unresolved vulnerabilities.
There are 0 security hotspots that need review.

This Library - Security

Best in #GCP

Average in #GCP

This Library - Security

Best in #GCP

Average in #GCP

license License

prediction-try-java-python is licensed under the Apache-2.0 License. This license is Permissive.
Permissive licenses have the least restrictions, and you can use them in most projects.

This Library - License

Best in #GCP

Average in #GCP

This Library - License

Best in #GCP

Average in #GCP

build Reuse

prediction-try-java-python releases are not available. You will need to build from source code and install.
prediction-try-java-python has no build file. You will be need to create the build yourself to build the component from source.
It has 2811 lines of code, 15 functions and 22 files.
It has low code complexity. Code complexity directly impacts maintainability of the code.

This Library - Reuse

Best in #GCP

Average in #GCP

This Library - Reuse

Best in #GCP

Average in #GCP

Top functions reviewed by kandi - BETA

kandi has reviewed prediction-try-java-python and discovered the below as its top functions. This is intended to give you an instant insight into prediction-try-java-python implemented functionality, and help decide if they suit your requirements.

HTTP GET request .
Parses a JsonFile .
Forward to the request .
Handle GET operation .

prediction-try-java-python Key Features

Python 2.5 or later

Google App Engine

Google Python API Client

Command line flags modules for Python

HTTP Client Library for Python

Google OAuth 2.0 Client Library for Python

URI Templates for Python

Java 5 (or higher) standard (SE) and enterprise (EE)

Google App Engine

Maven

Maven Plugin for App Engine

Clone this repo into a new directory.

Customize the following files: In shared/rc/client_secrets.json, replace the placeholder strings with your actual client id and client secret from the Google APIs console. In shared/rc/models.json, enter information about the model(s) you would like to use, following the format shown for the two sample models. Java only: edit the file gae-java/src/main/java/com/google/tryPredictionJava/web/IndexServlet.java to specify your redirect URI, which should be your app’s base URI /auth_return, e.g. http://your-app-name.appspot.com/auth_return. Add your redirect URI (defined in previous step) to the list of valid redirect URIs in the "API Access" tab of the APIs Console. If you miss this step, you’ll get a redirect_uri_mismatch error during initial authorization of the shared server credentials.

Build and deploy your app: For Python: Modify the "application:" line in your app.yaml file to reflect your chosen app name and use the Google App Engine tools to deploy your app. Install google-api-python-client library to your project vendor dir via pip: pip install -t vendor -r requirements.txt Note: this is only required for dev. On production this is done upon deployment For Java: modify the contents of the "application" XML element in your gae-java/src/main/webapp/WEB-INF/appengine-web.xml file to reflect your chosen app name and use the Maven plugin for Google App Engine to deploy your app (you need to run "mvn gae:unpack" once and then you can subsequently deploy your app repeatedly with "mvn gae:deploy").

The first time you access your app, it will step you through the login and OAuth 2.0 sequence, however, all access thereafter, by you or anyone else, will reuse your initially established security credentials. If you ever wish to change or re-establish the shared server credentials, simply visit your service’s URI with the "/reset" suffix (note that the reset service can only be invoked by the application administrator).

prediction-try-java-python Examples and Code Snippets

No Code Snippets are available at this moment for prediction-try-java-python.

See all Code Snippets related to GCP

Community Discussions

Trending Discussions on GCP

Submit command line arguments to a pyspark job on airflow
Skip first line in import statement using gc.open_by_url from gspread (i.e. add header=0)
Automatically Grab Latest Google Cloud Platform Secret Version
Programmatically Connecting a GitHub repo to a Google Cloud Project
Unable to create a new Cloud Function - cloud-client-api-gae
TypeScript project failing to deploy to App Engine targeting Node 12 or 14, but works with Node 10
Dataproc Java client throws NoSuchMethodError setUseJwtAccessWithScope
Apache Beam Cloud Dataflow Streaming Stuck Side Input
BIG Query command using BAT file
Vertex AI Model Batch prediction, issue with referencing existing model and input file on Cloud Storage

Trending Discussions on GCP

QUESTION

Submit command line arguments to a pyspark job on airflow

Asked 2022-Mar-29 at 10:37

I have a pyspark job available on GCP Dataproc to be triggered on airflow as shown below:

config = help.loadJSON("batch/config_file")

MY_PYSPARK_JOB = {
    "reference": {"project_id": "my_project_id"},
    "placement": {"cluster_name": "my_cluster_name"},
    "pyspark_job": {
        "main_python_file_uri": "gs://file/loc/my_spark_file.py"]
        "properties": config["spark_properties"]
        "args": <TO_BE_ADDED>
    },
}

I need to supply command line arguments to this pyspark job as show below [this is how I am running my pyspark job from command line]:

spark-submit gs://file/loc/my_spark_file.py --arg1 val1 --arg2 val2

I am providing the arguments to my pyspark job using "configparser". Therefore, arg1 is the key and val1 is the value from my spark-submit commant above.

How do I define the "args" param in the "MY_PYSPARK_JOB" defined above [equivalent to my command line arguments]?

ANSWER

Answered 2022-Mar-28 at 08:18

You have to pass a Sequence[str]. If you check DataprocSubmitJobOperator you will see that the params job implements a class google.cloud.dataproc_v1.types.Job.

class DataprocSubmitJobOperator(BaseOperator):
...
    :param job: Required. The job resource. If a dict is provided, it must be of the same form as the protobuf message.
    :class:`~google.cloud.dataproc_v1.types.Job`

So, on the section about job type pySpark which is google.cloud.dataproc_v1.types.PySparkJob:

args Sequence[str] Optional. The arguments to pass to the driver. Do not include arguments, such as --conf, that can be set as job properties, since a collision may occur that causes an incorrect job submission.

Source https://stackoverflow.com/questions/71616491

QUESTION

Skip first line in import statement using gc.open_by_url from gspread (i.e. add header=0)

Asked 2022-Mar-16 at 08:12

What is the equivalent of header=0 in pandas, which recognises the first line as a heading in gspread?

pandas import statement (correct)

import pandas as pd

# gcp / google sheets URL
df_URL = "https://docs.google.com/spreadsheets/d/1wKtvNfWSjPNC1fNmTfUHm7sXiaPyOZMchjzQBt1y_f8/edit?usp=sharing"

raw_dataset = pd.read_csv(df_URL, na_values='?',sep=';'
                          , skipinitialspace=True, header=0, index_col=None)

Using the gspread function, so far I import the data, change the first line to the heading then delete the first line after but this recognises everything in the DataFrame as a string. I would like to recognise the first line as a heading right away in the import statement.

gspread import statement that needs header=True equivalent

import pandas as pd
from google.colab import auth
auth.authenticate_user()
import gspread
from oauth2client.client import GoogleCredentials


# gcp / google sheets url
df_URL = "https://docs.google.com/spreadsheets/d/1wKtvNfWSjPNC1fNmTfUHm7sXiaPyOZMchjzQBt1y_f8/edit?usp=sharing"

# importing the data from Google Drive  setup
gc = gspread.authorize(GoogleCredentials.get_application_default())

# read data and put it in dataframe
g_sheets = gc.open_by_url(df_URL) 

df = pd.DataFrame(g_sheets.get_worksheet(0).get_all_values())

  
# change first row to header
df = df.rename(columns=df.iloc[0]) 

# drop first row
df.drop(index=df.index[0], axis=0, inplace=True)

ANSWER

Answered 2022-Mar-16 at 08:12

Looking at the API documentation, you probably want to use:

df = pd.DataFrame(g_sheets.get_worksheet(0).get_all_records(head=1))

The .get_all_records method returns a dictionary of with the column headers as the keys and a list of column values as the dictionary values. The argument head=<int> determines which row to use as keys; rows start from 1 and follow the numeration of the spreadsheet.

Since the values returned by .get_all_records() are lists of strings, the data frame constructor, pd.DataFrame, will return a data frame that is all strings. To convert it to floats, we need to replace the empty strings, and the the dash-only strings ('-') with NA-type values, then convert to float.

Luckily pandas DataFrame has a convenient method for replacing values .replace. We can pass it mapping from the string we want as NAs to None, which gets converted to NaN.

import pandas as pd

data = g_sheets.get_worksheet(0).get_all_records(head=1)

na_strings_map= {
    '-': None, 
    '': None
}

df = pd.DataFrame(data).replace(na_strings_map).astype(float)

Source https://stackoverflow.com/questions/71418682

QUESTION

Automatically Grab Latest Google Cloud Platform Secret Version

Asked 2022-Mar-01 at 03:01

I'm trying to grab the latest secret version. Is there a way to do that without specifying the version number? Such as using the keyword "latest". I'm trying to avoid having to iterate through all the secret versions with a for loop as GCP documentation shows:

try (SecretManagerServiceClient client = SecretManagerServiceClient.create()) {
  // Build the parent name.
  SecretName projectName = SecretName.of(projectId, secretId);

  // Get all versions.
  ListSecretVersionsPagedResponse pagedResponse = client.listSecretVersions(projectName);

  // List all versions and their state.
  pagedResponse
      .iterateAll()
      .forEach(
          version -> {
            System.out.printf("Secret version %s, %s\n", version.getName(), version.getState());
          });
}

ANSWER

Answered 2021-Sep-12 at 18:54

import com.google.cloud.secretmanager.v1.AccessSecretVersionResponse;
import com.google.cloud.secretmanager.v1.SecretManagerServiceClient;
import com.google.cloud.secretmanager.v1.SecretVersionName;
import java.io.IOException;

public class AccessSecretVersion {

  public static void accessSecretVersion() throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "your-project-id";
    String secretId = "your-secret-id";
    String versionId = "latest"; //<-- specify version
    accessSecretVersion(projectId, secretId, versionId);
  }

  // Access the payload for the given secret version if one exists. The version
  // can be a version number as a string (e.g. "5") or an alias (e.g. "latest").
  public static void accessSecretVersion(String projectId, String secretId, String versionId)
      throws IOException {
    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (SecretManagerServiceClient client = SecretManagerServiceClient.create()) {
      SecretVersionName secretVersionName = SecretVersionName.of(projectId, secretId, versionId);

      // Access the secret version.
      AccessSecretVersionResponse response = client.accessSecretVersion(secretVersionName);

      // Print the secret payload.
      //
      // WARNING: Do not print the secret in a production environment - this
      // snippet is showing how to access the secret material.
      String payload = response.getPayload().getData().toStringUtf8();
      System.out.printf("Plaintext: %s\n", payload);
    }
  }
}

source: https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets#secretmanager-access-secret-version-java

Source https://stackoverflow.com/questions/68805240

QUESTION

TypeScript project failing to deploy to App Engine targeting Node 12 or 14, but works with Node 10

Asked 2022-Jan-16 at 14:32

I have a TypeScript project that has been deployed several times without any problems to Google App Engine, Standard environment, running Node 10. However, when I try to update the App Engine project to either Node 12 or 14 (by editing the engines.node value in package.json and the runtime value in app.yaml), the deploy fails, printing the following to the console:

> ####@1.0.1 prepare /workspace
> npm run gcp-build


> ####@1.0.1 gcp-build /workspace
> tsc -p .

sh: 1: tsc: not found
npm ERR! code ELIFECYCLE
npm ERR! syscall spawn
npm ERR! file sh
npm ERR! errno ENOENT
npm ERR! ####@1.0.1 gcp-build: `tsc -p .`
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the ####@1.0.1 gcp-build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

According to the following documentation, App Engine should be installing the modules listed in devDependencies in package.json prior to running the gcp-build script (which it has been doing, as expected, when deploying with the Node version set to 10).

https://cloud.google.com/appengine/docs/standard/nodejs/running-custom-build-step

I can't find any documentation regarding how this App Engine deployment behavior has changed when targeting Node 12 or 14 rather than Node 10. Is there some configuration I am missing? Or does Google no longer install devDependencies prior to running the gcp-build script?

Here is the devDependencies section of my package.json (TypeScript is there):

    "devDependencies": {
        "@google-cloud/nodejs-repo-tools": "^3.3.0",
        "@types/cors": "^2.8.12",
        "@types/express": "^4.17.13",
        "@types/pg": "^8.6.1",
        "@types/request": "^2.48.7",
        "tsc-watch": "^4.4.0",
        "tslint": "^6.1.3",
        "typescript": "^4.3.5"
    },

ANSWER

Answered 2022-Jan-16 at 14:32

I encountered the exact same problem and just put typescript in dependencies, not devDependencies.

It worked after that, but cannot assure that it is due to this change (since I have no proof of that).

Source https://stackoverflow.com/questions/69153522

QUESTION

Dataproc Java client throws NoSuchMethodError setUseJwtAccessWithScope

Asked 2022-Jan-14 at 19:24

I am following this article ,for submit a job to an existing Dataproc cluster via a Dataproc API

For the following line of code :

 // Configure the settings for the job controller client.
    JobControllerSettings jobControllerSettings =
        JobControllerSettings.newBuilder().setEndpoint(myEndpoint).build();

I am getting the following errors :

SEVERE: Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.NoSuchMethodError: 'com.google.api.gax.core.GoogleCredentialsProvider$Builder com.google.api.gax.core.GoogleCredentialsProvider$Builder.setUseJwtAccessWithScope(boolean)'] with root cause
java.lang.NoSuchMethodError: 'com.google.api.gax.core.GoogleCredentialsProvider$Builder com.google.api.gax.core.GoogleCredentialsProvider$Builder.setUseJwtAccessWithScope(boolean)'

In my pom I used the following dependencies :

<dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>com.google.cloud</groupId>
                <artifactId>libraries-bom</artifactId>
                <version>24.1.2</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>

            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-gcp-dependencies</artifactId>
                <version>1.2.8.RELEASE</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>

        </dependencies>
    </dependencyManagement>

And added the dataproc

<dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-dataproc</artifactId>
        </dependency>

Any help what I am missing here?

ANSWER

Answered 2022-Jan-14 at 19:22

The method com.google.api.gax.core.GoogleCredentialsProvider$Builder com.google.api.gax.core.GoogleCredentialsProvider$Builder.setUseJwtAccessWithScope(boolean) was introduced in com.google.api:gax in version 2.3.0.

Can you

run mvn dependency:tree and confirm that your version of com.google.api:gax is above version 2.3.0?
upgrade all Google libraries to the latest version?

Here is a similar issue found on the internet.

Source https://stackoverflow.com/questions/70640346

QUESTION

Apache Beam Cloud Dataflow Streaming Stuck Side Input

Asked 2022-Jan-12 at 13:12

I'm currently building PoC Apache Beam pipeline in GCP Dataflow. In this case, I want to create streaming pipeline with main input from PubSub and side input from BigQuery and store processed data back to BigQuery.

Side pipeline code

side_pipeline = (
    p
    | "periodic" >> PeriodicImpulse(fire_interval=3600, apply_windowing=True)
    | "map to read request" >>
        beam.Map(lambda x:beam.io.gcp.bigquery.ReadFromBigQueryRequest(table=side_table))
    | beam.io.ReadAllFromBigQuery()
)

Function with side input code

def enrich_payload(payload, equipments):
    id = payload["id"]
    for equipment in equipments:
        if id == equipment["id"]:
            payload["type"] = equipment["type"]
            payload["brand"] = equipment["brand"]
            payload["year"] = equipment["year"]

            break

    return payload

Main pipeline code

main_pipeline = (
    p
    | "read" >> beam.io.ReadFromPubSub(topic="projects/my-project/topics/topiq")
    | "bytes to dict" >> beam.Map(lambda x: json.loads(x.decode("utf-8")))
    | "transform" >> beam.Map(transform_function)
    | "timestamping" >> beam.Map(lambda src: window.TimestampedValue(
        src,
        dt.datetime.fromisoformat(src["timestamp"]).timestamp()
    ))
    | "windowing" >> beam.WindowInto(window.FixedWindows(30))
)

final_pipeline = (
    main_pipeline
    | "enrich data" >> beam.Map(enrich_payload, equipments=beam.pvalue.AsIter(side_pipeline))
    | "store" >> beam.io.WriteToBigQuery(bq_table)
)

result = p.run()
result.wait_until_finish()

After deploy it to Dataflow, everything looks fine and no error. But then I noticed that enrich data step has two nodes instead of one.

And also, the side input stuck as you can see it has Elements Added with 21 counts in Input Collections and - value in Elements Added in Output Collections.

You can find the full pipeline code here and mock pubsub publisher here

I already follow all instruction in these documentations:

Yet still found this error. Please help me. Thanks!

ANSWER

Answered 2022-Jan-12 at 13:12

Here you have a working example:

mytopic = ""
sql = "SELECT station_id, CURRENT_TIMESTAMP() timestamp FROM `bigquery-public-data.austin_bikeshare.bikeshare_stations` LIMIT 10"

def to_bqrequest(e, sql):
    from apache_beam.io import ReadFromBigQueryRequest
    yield ReadFromBigQueryRequest(query=sql)
     

def merge(e, side):
    for i in side:
        yield f"Main {e.decode('utf-8')} Side {i}"

pubsub = p | "Read PubSub topic" >> ReadFromPubSub(topic=mytopic)

side_pcol = (p | PeriodicImpulse(fire_interval=300, apply_windowing=False)
               | "ApplyGlobalWindow" >> WindowInto(window.GlobalWindows(),
                                           trigger=trigger.Repeatedly(trigger.AfterProcessingTime(5)),
                                           accumulation_mode=trigger.AccumulationMode.DISCARDING)
               | "To BQ Request" >> ParDo(to_bqrequest, sql=sql)
               | ReadAllFromBigQuery()
            )

final = (pubsub | "Merge" >> ParDo(merge, side=beam.pvalue.AsList(side_pcol))
                | Map(logging.info)
        )                    
    
p.run()

Note this uses a GlobalWindow (so that both inputs have the same window). I used a processing time trigger so that the pane contains multiple rows. 5 was chosen arbitrarily, using 1 would work too.

Please note matching the data between side and main inputs is non deterministic, and you may see fluctuating values from older fired panes.

In theory, using FixedWindows should fix this, but I cannot get the FixedWindows to work.

Source https://stackoverflow.com/questions/70561769

QUESTION

BIG Query command using BAT file

Asked 2022-Jan-09 at 15:24

echo Give yearmonth "yyyyMM"
setlocal enabledelayedexpansion
SET /p yearmonth= 
SET ClientName[0]=abc
SET ClientName[1]=def

SET i = 0

:myLoop
if defined ClientName[%i%] (
    call bq query --use_legacy_sql=false "CREATE EXTERNAL TABLE `test.!ClientName[%%i]!.%yearmonth%` OPTIONS (format = 'CSV',skip_leading_rows = 1 uris = ['gs://test/!ClientName[%%i]!/AWS/%yearmonth%/Metrics/data/*.csv'])"
    set /a "i+=1"
    GOTO :myLoop

)

Hi, I am trying to create a batch so that i can run Multiple BIG QUERY at once. Above i tried to write a batch script putting command in a loop .

I am trying to create a table by using yearmonth as user input and then create array to create a table with different client name .

But I am unable to print if i =0 ClientName[i] = abc in a call query i am using !ClientName[%%i]! to print but its not working.
Call query inside loop is not running in GCP console, when i executed the bat file .

Can you please help me resolve this

ANSWER

Answered 2022-Jan-09 at 11:04

It is bad practice to set variables as standalone alphabetical characters like i. One reason is exactly as you have experienced, you have confused for metavariable %%i with a set variable %i%.
You are expanding in the loop, but have not enabledelayedexpansion so there are 2 ways, which we will get to in a second.
setting variables should not have spaces before or after = excluding the likes of set /a

So, Method 1, without delayedexpansion (note how the variables are used with double %% in the loop with the call command).

@echo off
echo Give yearmonth "yyyyMM"
SET /p yearmonth= 
SET ClientName[0]=abc
SET ClientName[1]=def

SET num=0

:myLoop
if defined ClientName[%num%] (
    call bq query --use_legacy_sql=false "CREATE EXTERNAL TABLE `test.%%ClientName[%num%]%%.%yearmonth%` OPTIONS (format = 'CSV',skip_leading_rows = 1 uris = ['gs://test/%%ClientName[%num%]%%/AWS/%yearmonth%/Metrics/data/*.csv'])"
    set /a num+=1
    GOTO :myLoop

)

Method 2: (better method using delayedexpansion)

@echo off
setlocal enabledelayedexpansion
echo Give yearmonth "yyyyMM"
SET /p yearmonth= 
SET ClientName[0]=abc
SET ClientName[1]=def

SET num=0

:myLoop
if defined ClientName[%num%] (
    call bq query --use_legacy_sql=false "CREATE EXTERNAL TABLE `test.!ClientName[%num%]!.%yearmonth%` OPTIONS (format = 'CSV',skip_leading_rows = 1 uris = ['gs://test/!ClientName[%num%]!/AWS/%yearmonth%/Metrics/data/*.csv'])"
    set /a num+=1
    GOTO :myLoop

)

Source https://stackoverflow.com/questions/70640411

QUESTION

Vertex AI Model Batch prediction, issue with referencing existing model and input file on Cloud Storage

Asked 2021-Dec-21 at 14:35

I'm struggling to correctly set Vertex AI pipeline which does the following:

read data from API and store to GCS and as as input for batch prediction.
get an existing model (Video classification on Vertex AI)
create Batch prediction job with input from point 1.
As it will be seen, I don't have much experience with Vertex Pipelines/Kubeflow thus I'm asking for help/advice, hope it's just some beginner mistake. this is the gist of the code I'm using as pipeline

from google_cloud_pipeline_components import aiplatform as gcc_aip
from kfp.v2 import dsl

from kfp.v2.dsl import component
from kfp.v2.dsl import (
    Output,
    Artifact,
    Model,
)

PROJECT_ID = 'my-gcp-project'
BUCKET_NAME = "mybucket"
PIPELINE_ROOT = "{}/pipeline_root".format(BUCKET_NAME)


@component
def get_input_data() -> str:
    # getting data from API, save to Cloud Storage
    # return GS URI
    gcs_batch_input_path = 'gs://somebucket/file'
    return gcs_batch_input_path


@component(
    base_image="python:3.9",
    packages_to_install=['google-cloud-aiplatform==1.8.0']
)
def load_ml_model(project_id: str, model: Output[Artifact]):
    """Load existing Vertex model"""
    import google.cloud.aiplatform as aip

    model_id = '1234'
    model = aip.Model(model_name=model_id, project=project_id, location='us-central1')



@dsl.pipeline(
    name="batch-pipeline", pipeline_root=PIPELINE_ROOT,
)
def pipeline(gcp_project: str):
    input_data = get_input_data()
    ml_model = load_ml_model(gcp_project)

    gcc_aip.ModelBatchPredictOp(
        project=PROJECT_ID,
        job_display_name=f'test-prediction',
        model=ml_model.output,
        gcs_source_uris=[input_data.output],  # this doesn't work
        # gcs_source_uris=['gs://mybucket/output/'],  # hardcoded gs uri works
        gcs_destination_output_uri_prefix=f'gs://{PIPELINE_ROOT}/prediction_output/'
    )


if __name__ == '__main__':
    from kfp.v2 import compiler
    import google.cloud.aiplatform as aip
    pipeline_export_filepath = 'test-pipeline.json'
    compiler.Compiler().compile(pipeline_func=pipeline,
                                package_path=pipeline_export_filepath)
    # pipeline_params = {
    #     'gcp_project': PROJECT_ID,
    # }
    # job = aip.PipelineJob(
    #     display_name='test-pipeline',
    #     template_path=pipeline_export_filepath,
    #     pipeline_root=f'gs://{PIPELINE_ROOT}',
    #     project=PROJECT_ID,
    #     parameter_values=pipeline_params,
    # )

    # job.run()

When running the pipeline it throws this exception when running Batch prediction:
details = "List of found errors: 1.Field: batch_prediction_job.model; Message: Invalid Model resource name. so I'm not sure what could be wrong. I tried to load model in the notebook (outside of component) and it correctly returns.

Second issue I'm having is referencing GCS URI as output from component to batch job input.

   input_data = get_input_data2()
   gcc_aip.ModelBatchPredictOp(
        project=PROJECT_ID,
        job_display_name=f'test-prediction',
        model=ml_model.output,
        gcs_source_uris=[input_data.output],  # this doesn't work
        # gcs_source_uris=['gs://mybucket/output/'],  # hardcoded gs uri works
        gcs_destination_output_uri_prefix=f'gs://{PIPELINE_ROOT}/prediction_output/'
    )

During compilation, I get following exception TypeError: Object of type PipelineParam is not JSON serializable, though I think this could be issue of ModelBatchPredictOp component.

Again any help/advice appreciated, I'm dealing with this from yesterday, so maybe I missed something obvious.

libraries I'm using:

google-cloud-aiplatform==1.8.0  
google-cloud-pipeline-components==0.2.0  
kfp==1.8.10  
kfp-pipeline-spec==0.1.13  
kfp-server-api==1.7.1

UPDATE After comments, some research and tuning, for referencing model this works:

@component
def load_ml_model(project_id: str, model: Output[Artifact]):
    region = 'us-central1'
    model_id = '1234'
    model_uid = f'projects/{project_id}/locations/{region}/models/{model_id}'
    model.uri = model_uid
    model.metadata['resourceName'] = model_uid

and then I can use it as intended:

batch_predict_op = gcc_aip.ModelBatchPredictOp(
        project=gcp_project,
        job_display_name=f'batch-prediction-test',
        model=ml_model.outputs['model'],
        gcs_source_uris=[input_batch_gcs_path],
gcs_destination_output_uri_prefix=f'gs://{BUCKET_NAME}/prediction_output/test'
    )

UPDATE 2 regarding GCS path, a workaround is to define path outside of the component and pass it as an input parameter, for example (abbreviated):

@dsl.pipeline(
    name="my-pipeline",
    pipeline_root=PIPELINE_ROOT,
)
def pipeline(
        gcp_project: str,
        region: str,
        bucket: str
):
    ts = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
    
    gcs_prediction_input_path = f'gs://{BUCKET_NAME}/prediction_input/video_batch_prediction_input_{ts}.jsonl'
    batch_input_data_op = get_input_data(gcs_prediction_input_path)  # this loads input data to GCS path

    batch_predict_op = gcc_aip.ModelBatchPredictOp(
        project=gcp_project,
        model=training_job_run_op.outputs["model"],
        job_display_name='batch-prediction',
        # gcs_source_uris=[batch_input_data_op.output],
        gcs_source_uris=[gcs_prediction_input_path],
        gcs_destination_output_uri_prefix=f'gs://{BUCKET_NAME}/prediction_output/',
    ).after(batch_input_data_op)  # we need to add 'after' so it runs after input data is prepared since get_input_data doesn't returns anything

still not sure, why it doesn't work/compile when I return GCS path from get_input_data component

ANSWER

Answered 2021-Dec-21 at 14:35

I'm glad you solved most of your main issues and found a workaround for model declaration.

For your input.output observation on gcs_source_uris, the reason behind it is because the way the function/class returns the value. If you dig inside the class/methods of google_cloud_pipeline_components you will find that it implements a structure that will allow you to use .outputs from the returned value of the function called.

If you go to the implementation of one of the components of the pipeline you will find that it returns an output array from convert_method_to_component function. So, in order to have that implemented in your custom class/function your function should return a value which can be called as an attribute. Below is a basic implementation of it.

class CustomClass():
     def __init__(self):
       self.return_val = {'path':'custompath','desc':'a desc'}
      
     @property
     def output(self):
       return self.return_val 

hello = CustomClass()
print(hello.output['path'])

If you want to dig more about it you can go to the following pages:

convert_method_to_component, which is the implementation of convert_method_to_component
Properties, basics of property in python.

Source https://stackoverflow.com/questions/70356856

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install prediction-try-java-python

You can download it from GitHub.
You can use prediction-try-java-python like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the prediction-try-java-python component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .