training-data-analyst | Labs and demos for courses for GCP Training ( http | Learning library

by GoogleCloudPlatform Jupyter Notebook Version: Current License: Apache-2.0

kandi X-RAY | training-data-analyst Summary

training-data-analyst is a Jupyter Notebook library typically used in Tutorial, Learning, Tensorflow, Docker applications. training-data-analyst has no bugs, it has no vulnerabilities, it has a Permissive License and it has medium support. You can download it from GitHub.

Labs and demos for courses for GCP Training (http://cloud.google.com/training).

Support

Quality

Security

License

Reuse

kandi-support Support

training-data-analyst has a medium active ecosystem.
It has 6542 star(s) with 5221 fork(s). There are 266 watchers for this library.
It had no major release in the last 6 months.
There are 103 open issues and 69 have been closed. On average issues are closed in 100 days. There are 182 open pull requests and 0 closed requests.
It has a neutral sentiment in the developer community.
The latest version of training-data-analyst is current.

This Library - Support

Best in #Learning

Average in #Learning

This Library - Support

Best in #Learning

Average in #Learning

quality kandi Quality

training-data-analyst has no bugs reported.

This Library - Quality

Best in #Learning

Average in #Learning

This Library - Quality

Best in #Learning

Average in #Learning

security Security

training-data-analyst has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

This Library - Security

Best in #Learning

Average in #Learning

This Library - Security

Best in #Learning

Average in #Learning

license License

training-data-analyst is licensed under the Apache-2.0 License. This license is Permissive.
Permissive licenses have the least restrictions, and you can use them in most projects.

This Library - License

Best in #Learning

Average in #Learning

This Library - License

Best in #Learning

Average in #Learning

build Reuse

training-data-analyst releases are not available. You will need to build from source code and install.

This Library - Reuse

Best in #Learning

Average in #Learning

This Library - Reuse

Best in #Learning

Average in #Learning

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample Here

Get all kandi verified functions for this library.

training-data-analyst Key Features

Labs and demos for courses for GCP Training (http://cloud.google.com/training).

training-data-analyst Examples and Code Snippets

Community Discussions

Trending Discussions on training-data-analyst

chown command result in user invalid error in google colab
''ls' is not recognized as an internal' in python
Hybrid recommendation system with matrix factorization and linear regression
How to use tensorflow to ingest sharded CSVs

Trending Discussions on training-data-analyst

QUESTION

chown command result in user invalid error in google colab

Asked 2020-Nov-23 at 00:58

I try to run code !sudo chown -R jupyter:jupyter /home/jupyter/training-data-analyst from https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/ai-for-finance/solution/aapl_regression_scikit_learn.ipynb in Google Colab jupyter notebook.

Although my account authenticated with authentication_user() method:

from google.colab import auth
auth.authenticate_user()

I receive the error:

chown: invalid user: ‘jupyter:jupyter’

I appreciate if you let me know how can I solve the problem, with a little bit description about what exactly happens when I run this code. All I have understood is that chown let me change the user and/or group ownership of a given file.

When I run !ls -la on jupyter cell(in current working directory of /content) the output is:

total 28
drwxr-xr-x 1 root root 4096 Nov 22 19:27 .
drwxr-xr-x 1 root root 4096 Nov 22 18:34 ..
drwxr-xr-x 1 root root 4096 Nov 22 18:47 .config
drwxr-xr-x 2 root root 4096 Nov 22 19:27 .ipynb_checkpoints
drwxr-xr-x 1 root root 4096 Nov 13 17:33 sample_data

the same result for /home directory:

total 8
drwxr-xr-x 2 root root 4096 Apr 24  2018 .
drwxr-xr-x 1 root root 4096 Nov 22 18:34 ..

I don't know even if I clone training-data-analyst folder from google github, the code work or not! cause it states the user is invalid!

ANSWER

Answered 2020-Nov-23 at 00:58

It is an error because the user jupyter does not exist.

You need to create it first with

!groupadd jupyter
!useradd -g jupyter jupyter

Source https://stackoverflow.com/questions/64958745

QUESTION

Hybrid recommendation system with matrix factorization and linear regression

Asked 2020-May-14 at 18:47

I'm following a tutorial that for creating a recommendation system in BigQueryML. The tutorial uses matrix factorization first to calculate user and item factors. In the end I have a model that can be queried with user ids or item ids to get recommendations.

The next step is feeding the factors and additional item + user features into a linear regression model to incorporate more context.

"Essentially, we have a couple of attributes about the movie, the product factors array corresponding to the movie, a couple of attributes about the user, and the user factors array corresponding to the user. These form the inputs to our “hybrid” recommendations model that builds off the matrix factorization model and adds in metadata about users and movies."

I just don't understand why the dataset for linear regression excludes the user and item ids:

 SELECT
        p.* EXCEPT(movieId),
        u.* EXCEPT(userId),
        rating 
    FROM productFeatures p, userFeatures u
    JOIN movielens.ratings r
    ON r.movieId = p.movieId AND r.userId = u.userId

My question is: How will I be able to get recommendations for a user from the linear model, when I don't have the user or item ids in the model?

Here you can find the full code:

https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/deepdive2/recommendation_systems/solutions/als_bqml_hybrid.ipynb

ANSWER

Answered 2020-May-14 at 18:47

In the example you have shared, the goal is to fit a linear regression to the discovered factor values so that a novel set of factor values can be used to predict the rating. In this kind of setup, you don't want information about which samples are being used; the only crucial information is the training features (the factor scores) and the rating (the training/test label). For more on this topic, take a look at "Dimensionality reduction using non-negative matrix factorization for information retrieval."

If you included the movie ids and user ids in as features, your regression would try to learn on those, which would either add noise to the model or learn that low ids = lower score etc. This is possible, especially if this ids are in some kind of order you're not aware of, such as chronological or by genre.

Note: You could use movie-specific or user-specific information to build a model, but you would have many, many dimensions of data, and that tends to create poorly performing models. The idea here is to avoid the problem of dimensionality by first reducing the dimensionality of the problem space. Matrix factorization is just one method among many to do this. See, for example, PCA, LDA, and word2vec.

Source https://stackoverflow.com/questions/61637034

QUESTION

How to use tensorflow to ingest sharded CSVs

Asked 2020-Jan-04 at 20:00

This is a problem I am working on in google cloud platform with tensorflow v1.15

I am working on this notebook In this section, I am supposed to return a function that feeds model.train()

CSV_COLUMNS = ['fare_amount', 'pickuplon','pickuplat','dropofflon','dropofflat','passengers', 'key']
DEFAULTS = [[0.0], [-74.0], [40.0], [-74.0], [40.7], [1.0], ['nokey']]

# TODO: Create an appropriate input function read_dataset
def read_dataset(filename, mode):
    #TODO Add CSV decoder function and dataset creation and methods
    return dataset

def get_train_input_fn():
  return read_dataset('./taxi-train.csv', mode = tf.estimator.ModeKeys.TRAIN)

def get_valid_input_fn():
  return read_dataset('./taxi-valid.csv', mode = tf.estimator.ModeKeys.EVAL)

I think it should be like this:

def read_dataset(filename, mode, batch_size = 512):
  def fn():
    def decode_csv(value_column):
      columns = tf.decode_csv(value_column, record_defaults = DEFAULTS)
      features = dict(zip(CSV_COLUMNS, columns))
      label = features.pop(LABEL_COLUMN)
      return features, label

    # Create list of file names that match "glob" pattern (i.e. data_file_*.csv)
    filenames_dataset = tf.data.Dataset.list_files(filename)
    # Read lines from text files
    textlines_dataset = filenames_dataset.flat_map(tf.data.TextLineDataset)
    # Parse text lines as comma-separated values (CSV)
    dataset = textlines_dataset.map(decode_csv)

    if mode == tf.estimator.ModeKeys.TRAIN:
        num_epochs = None # indefinitely
        dataset = dataset.shuffle(buffer_size = 10 * batch_size)
    else:
        num_epochs = 1 # end-of-input after this

    dataset = dataset.repeat(num_epochs).batch(batch_size)

    return dataset
  return fn

That is actually reflective of code in the video recap that accompanies this notebook, and very similar to my own attempts before I saw that recap. It is also similar to the next notebook, but that code also unfortunately fails.

With the above code, I am getting this error:

UnimplementedError: Cast string to float is not supported
     [[node linear/head/ToFloat (defined at /usr/local/lib/python3.5/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]

So, I'm not sure how to transform the data to match the datatype.. I cannot cast the data in decode_csv like:

      features = {CSV_COLUMNS[i]: float(cols[i]) for i in range(1, len(CSV_COLUMNS) - 1)}

because the error is happening the line before that is called.

Investigating the data I note:

import csv

with open('./taxi-train.csv') as f: 
    reader = csv.reader(f)
    print(next(reader))

['12.0', '-73.987625', '40.750617', '-73.971163', '40.78518', '1', '0']

that looks like the raw data might actually be a string .. am I correct? How can I solve this?

edit : I have located the csv file, it is not raw string data. Why is the tensorflow import bringing it in as text??

ANSWER

Answered 2020-Jan-04 at 20:00

The training-data-analyst repository you mentioned, also has the solutions to all the notebooks.

From analysing the provided solution it looks like the def fn() part is reduntant. the read_dataset function should simply return a tf.Data.dataset:

def read_dataset(filename, mode, batch_size = 512):
  def decode_csv(row):
    columns = tf.decode_csv(row, record_defaults = DEFAULTS)
    features = dict(zip(CSV_COLUMNS, columns))
    features.pop('key') # discard, not a real feature
    label = features.pop('fare_amount') # remove label from features and store
    return features, label

  # Create list of file names that match "glob" pattern (i.e. data_file_*.csv)
  filenames_dataset = tf.data.Dataset.list_files(filename, shuffle=False)
  # Read lines from text files
  textlines_dataset = filenames_dataset.flat_map(tf.data.TextLineDataset)
  # Parse text lines as comma-separated values (CSV)
  dataset = textlines_dataset.map(decode_csv)

  # Note:
  # use tf.data.Dataset.flat_map to apply one to many transformations (here: filename -> text lines)
  # use tf.data.Dataset.map      to apply one to one  transformations (here: text line -> feature list)

  if mode == tf.estimator.ModeKeys.TRAIN:
      num_epochs = None # loop indefinitely
      dataset = dataset.shuffle(buffer_size = 10 * batch_size, seed=2)
  else:
      num_epochs = 1 # end-of-input after this

  dataset = dataset.repeat(num_epochs).batch(batch_size)

  return dataset

The solutions are located i the same directory as labs. So for example the solution for
training-data-analyst/courses/machine_learning/deepdive/03_tensorflow/labs/c_dataset.ipynb
is located at
training-data-analyst/courses/machine_learning/deepdive/03_tensorflow/c_dataset.ipynb

Source https://stackoverflow.com/questions/59592234

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install training-data-analyst

You can download it from GitHub.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: