zeppelin | Webbased notebook that enables datadriven

by apache Java Version: Current License: Apache-2.0

Download this library from

kandi X-RAY | zeppelin Summary

zeppelin is a Java library typically used in Big Data, Jupyter, Spark applications. zeppelin has build file available, it has a Permissive License and it has medium support. However zeppelin has 711 bugs and it has 28 vulnerabilities. You can install using 'npm i zeppelin-vis' or download it from GitHub, npm.

Zeppelin, a web-based notebook that enables interactive data analytics. You can make beautiful data-driven, interactive and collaborative documents with SQL, Scala and more. To know more about Zeppelin, visit our web site https://zeppelin.apache.org.

Support

Quality

Security

License

Reuse

kandi-support Support

zeppelin has a medium active ecosystem.
It has 5639 star(s) with 2619 fork(s). There are 330 watchers for this library.
It had no major release in the last 12 months.
zeppelin has no issues reported. There are 102 open pull requests and 0 closed requests.
It has a neutral sentiment in the developer community.
The latest version of zeppelin is current.

zeppelin Support

Best in #Java

Average in #Java

zeppelin Support

Best in #Java

Average in #Java

quality kandi Quality

zeppelin has 711 bugs (28 blocker, 3 critical, 320 major, 360 minor) and 5169 code smells.

zeppelin Quality

Best in #Java

Average in #Java

zeppelin Quality

Best in #Java

Average in #Java

security Security

zeppelin has 6 vulnerability issues reported (1 critical, 3 high, 2 medium, 0 low).
zeppelin code analysis shows 22 unresolved vulnerabilities (21 blocker, 0 critical, 1 major, 0 minor).
There are 90 security hotspots that need review.

zeppelin Security

Best in #Java

Average in #Java

zeppelin Security

Best in #Java

Average in #Java

license License

zeppelin is licensed under the Apache-2.0 License. This license is Permissive.
Permissive licenses have the least restrictions, and you can use them in most projects.

zeppelin License

Best in #Java

Average in #Java

zeppelin License

Best in #Java

Average in #Java

build Reuse

zeppelin releases are not available. You will need to build from source code and install.
Deployable package is available in npm.
Build file is available. You can build the component from source.
Installation instructions are available. Examples and code snippets are not available.
zeppelin saves you 203711 person hours of effort in developing the same functionality from scratch.
It has 203831 lines of code, 14402 functions and 1765 files.
It has medium code complexity. Code complexity directly impacts maintainability of the code.

zeppelin Reuse

Best in #Java

Average in #Java

zeppelin Reuse

Best in #Java

Average in #Java

Top functions reviewed by kandi - BETA

kandi has reviewed zeppelin and discovered the below as its top functions. This is intended to give you an instant insight into zeppelin implemented functionality, and help decide if they suit your requirements.

Convert a query to a query
Convert the specified Properties to Jinja parameters .
Performs kerberos authentication .
stream execute request
Splits the given string using the given escape sequence .
Sets up the local resources to launch the VM .
Runs the interpreter .
Copy run files to container .
Creates a programmatic config loader
Splits the given text .

zeppelin Key Features

Web based notebook style editor.

Built-in Apache Spark support

zeppelin Examples and Code Snippets

See all related Code Snippets

Cannot find conda info. Please verify your conda installation on EMR

Copy

Download

wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.9.2-Linux-x86_64.sh  -O /home/hadoop/miniconda.sh \
    && /bin/bash ~/miniconda.sh -b -p $HOME/conda

echo -e '\n export PATH=$HOME/conda/bin:$PATH' >> $HOME/.bashrc && source $HOME/.bashrc


conda config --set always_yes yes --set changeps1 no
conda config -f --add channels conda-forge


conda create -n zoo python=3.7 # "zoo" is conda environment name
conda init bash
source activate zoo
conda install python 3.7.0 -c conda-forge orca 
sudo /home/hadoop/conda/envs/zoo/bin/python3.7 -m pip install virtualenv

“spark.pyspark.python": "/home/hadoop/conda/envs/zoo/bin/python3",
"spark.pyspark.virtualenv.enabled": "true",
"spark.pyspark.virtualenv.type":"native",
"spark.pyspark.virtualenv.bin.path":"/home/hadoop/conda/envs/zoo/bin/,
"zeppelin.pyspark.python" : "/home/hadoop/conda/bin/python",
"zeppelin.python": "/home/hadoop/conda/bin/python"

-----------------------

wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.9.2-Linux-x86_64.sh  -O /home/hadoop/miniconda.sh \
    && /bin/bash ~/miniconda.sh -b -p $HOME/conda

echo -e '\n export PATH=$HOME/conda/bin:$PATH' >> $HOME/.bashrc && source $HOME/.bashrc


conda config --set always_yes yes --set changeps1 no
conda config -f --add channels conda-forge


conda create -n zoo python=3.7 # "zoo" is conda environment name
conda init bash
source activate zoo
conda install python 3.7.0 -c conda-forge orca 
sudo /home/hadoop/conda/envs/zoo/bin/python3.7 -m pip install virtualenv

“spark.pyspark.python": "/home/hadoop/conda/envs/zoo/bin/python3",
"spark.pyspark.virtualenv.enabled": "true",
"spark.pyspark.virtualenv.type":"native",
"spark.pyspark.virtualenv.bin.path":"/home/hadoop/conda/envs/zoo/bin/,
"zeppelin.pyspark.python" : "/home/hadoop/conda/bin/python",
"zeppelin.python": "/home/hadoop/conda/bin/python"

Does Zeppelin 0.10.0 try to run interpreter in k8s cluster?

Copy

Download

gofabric8 start

Printing Unique Values in a Column as a Percentage of Total Rows

Copy

Download

import org.apache.spark.sql.functions.{col, countDistinct, count}

import spark.implicits._

// define a dataframe for example
val data = Seq(("1", "1"), ("1", "2"), ("1", "3"), ("1", "4")).toDF("col_a", "col_b")

data.select(data.columns.map(c => (lit(100) * countDistinct(col(c)) / count(col(c))).alias(c)): _*).show()

// output:
+-----+-----+
|col_a|col_b|
+-----+-----+
| 25.0|100.0|
+-----+-----+

User "system:serviceaccount:default:flink" cannot list resource "nodes" in API group "" at the cluster scope

Copy

Download

...
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
...

CSS Keyframe Animations

Copy

Download

#SLIDE_BG {
  width: 100%;
  height: 100vh;
  background-position: center center;
  background-size: cover;
  background-repeat: no-repeat;
  backface-visibility: hidden;
  animation: slideBg 8s linear infinite 0s;
  animation-timing-function: ease-in-out;
  background-image: url('https://dummyimage.com/800x300');         
}

@keyframes slideBg {
  0% {
    background-image: url('https://upload.wikimedia.org/wikipedia/commons/2/29/Dscn7471_sunset-sundog_crop_800x300.jpg');
  }
  25% {
    background-image: url('https://www.thegrandsiba.com/wp-content/uploads/2017/06/MG_28423-800x300.jpg');
  }
  50% {
    background-image: url('https://dummyimage.com/800x300');
  }
  75% {
    background-image: url('https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/5e2a19d4-f261-4548-b4a7-1e2f5ad139af/d99najf-e9049704-bfc4-4836-952d-9315403bd60f.gif');
  }
  100% {
    background-image: url('https://www.businessclass.co.uk/wp-content/uploads/sites/11/2016/04/Toronto-800x300-800x300.jpg');
  }
}

-----------------------

#SLIDE_BG {
  width: 100%;
  height: 100vh;
  background-position: center center;
  background-size: cover;
  background-repeat: no-repeat;
  backface-visibility: hidden;
  animation: slideBg 8s linear infinite 0s;
  animation-timing-function: ease-in-out;
  background-image: url('https://jooinn.com/images/dramatic-landscape-7.jpg');
}

@keyframes slideBg {
  0% {
    background-image: url('https://jooinn.com/images/dramatic-landscape-7.jpg');
  }
  25% {
    background-image: url('http://www.thewowstyle.com/wp-content/uploads/2015/01/nature-image.jpg');
  }
  50% {
    background-image: url('https://images.designtrends.com/wp-content/uploads/2016/01/04085621/A-Cold-Sunset-Background.jpg');
  }
  75% {
    background-image: url('https://jooinn.com/images/hdr-landscape-1.jpg');
  }
  100% {
    background-image: url('https://www.shutterstock.com/blog/wp-content/uploads/sites/5/2016/03/fall-trees-road-1.jpg');
  }
}

<div id="SLIDE_BG"></div>

-----------------------

#SLIDE_BG {
  width: 100%;
  height: 100vh;
  background-position: center center;
  background-size: cover;
  background-repeat: no-repeat;
  backface-visibility: hidden;
  animation: slideBg 8s linear infinite 0s;
  animation-timing-function: ease-in-out;
  background-image: url('https://jooinn.com/images/dramatic-landscape-7.jpg');
}

@keyframes slideBg {
  0% {
    background-image: url('https://jooinn.com/images/dramatic-landscape-7.jpg');
  }
  25% {
    background-image: url('http://www.thewowstyle.com/wp-content/uploads/2015/01/nature-image.jpg');
  }
  50% {
    background-image: url('https://images.designtrends.com/wp-content/uploads/2016/01/04085621/A-Cold-Sunset-Background.jpg');
  }
  75% {
    background-image: url('https://jooinn.com/images/hdr-landscape-1.jpg');
  }
  100% {
    background-image: url('https://www.shutterstock.com/blog/wp-content/uploads/sites/5/2016/03/fall-trees-road-1.jpg');
  }
}

<div id="SLIDE_BG"></div>

totalsupply() is not a function openzeppelin contracts

Copy

Download

contract Color is ERC721, IERC721Enumerable { // We must extends IERC721Enumerable 
    string[] public colors;
    mapping(string => bool) _colorExists;

    constructor() ERC721("Color", "COLOR") {}

    function mint(string memory _color) public {
        colors.push(_color);
        uint256 _id = colors.length - 1;

        // _mint(msg.sender,_id);
        _colorExists[_color] = true;
    }
    

    // And must override below three functions

    function tokenOfOwnerByIndex(address owner, uint256 index) public view override returns (uint256) {
        // You need update this logic.
        // ...
        return 3;
    }

    function totalSupply() external  view override returns (uint256) {
      // You need update this logic.
      // ...
      return 1;
    }

    function tokenByIndex(uint256 index) external view  override returns (uint256) {
      // You need update this logic.
      // ...
      return 5;
    }
}

-----------------------

pragma solidity ^0.8.0; // Note that this is using a newer version than in 

import "@openzeppelin/contracts/token/ERC721/ERC721.sol";
import "@openzeppelin/contracts/token/ERC721/extensions/ERC721Enumerable.sol";


contract Color is ERC721, ERC721Enumerable {
    string[] public colors;
    mapping(string => bool) _colorExists;

    constructor() ERC721("Color", "COLOR") public {
    }

    function _beforeTokenTransfer(address from, address to, uint256 tokenId)
    internal
    override(ERC721, ERC721Enumerable)
    {
        super._beforeTokenTransfer(from, to, tokenId);
    }

    function supportsInterface(bytes4 interfaceId)
    public
    view
    override(ERC721, ERC721Enumerable)
    returns (bool)
    {
        return super.supportsInterface(interfaceId);
    }


    function mint(string memory _color) public {
        colors.push(_color);
        uint _id = colors.length - 1;
        _mint(msg.sender, _id);
        _colorExists[_color] = true;
    }
}

How can find the find occurence in a window/group and fetch all unbounded rows prior to that?

Copy

Download

SELECT
    accountId,
    customerId,
    sessionDate,
    didPurchase
FROM (
    SELECT 
        *,
        MIN(CASE
            WHEN didPurchase THEN sessionDate
        END) OVER (
            PARTITION BY accountId, customerId
        ) as prior_oldest_purchase
     FROM
        myvisits
) v
WHERE
     v.sessionDate <= v.prior_oldest_purchase
ORDER BY
    accountId DESC, customerId,sessionDate DESC

import org.apache.spark.sql.functions._
import org.apache.spark.sql.expressions.Window

val customerAccountWindow = Window.partitionBy("accountId","customerId")

val outputDf = df.withColumn(
                    "prior_oldest_purchase",
                    min(
                        when(col("didPurchase"),col("sessionDate"))
                    ).over(customerAccountWindow)
               )
               .where(col("sessionDate") <= col("prior_oldest_purchase"))
               .select(
                   col("accountId"),
                   col("customerId"),
                   col("sessionDate"),
                   col("didPurchase")
               )
               .orderBy(
                   col("accountId").desc(),
                   col("customerId"),
                   col("sessionDate").desc()
               )


from pyspark.sql import functions as F
from pyspark.sql import Window

customerAccountWindow = Window.partitionBy("accountId","customerId")

outputDf = (
    df.withColumn(
        "prior_oldest_purchase",
        F.min(
            F.when(F.col("didPurchase"),F.col("sessionDate"))
        ).over(customerAccountWindow)
    )
    .where(F.col("sessionDate") <= F.col("prior_oldest_purchase"))
    .select(
        F.col("accountId"),
        F.col("customerId"),
        F.col("sessionDate"),
        F.col("didPurchase")
     )
     .orderBy(
        F.col("accountId").desc(),
        F.col("customerId"),
        F.col("sessionDate").desc()
      )
)

-----------------------

SELECT
    accountId,
    customerId,
    sessionDate,
    didPurchase
FROM (
    SELECT 
        *,
        MIN(CASE
            WHEN didPurchase THEN sessionDate
        END) OVER (
            PARTITION BY accountId, customerId
        ) as prior_oldest_purchase
     FROM
        myvisits
) v
WHERE
     v.sessionDate <= v.prior_oldest_purchase
ORDER BY
    accountId DESC, customerId,sessionDate DESC

import org.apache.spark.sql.functions._
import org.apache.spark.sql.expressions.Window

val customerAccountWindow = Window.partitionBy("accountId","customerId")

val outputDf = df.withColumn(
                    "prior_oldest_purchase",
                    min(
                        when(col("didPurchase"),col("sessionDate"))
                    ).over(customerAccountWindow)
               )
               .where(col("sessionDate") <= col("prior_oldest_purchase"))
               .select(
                   col("accountId"),
                   col("customerId"),
                   col("sessionDate"),
                   col("didPurchase")
               )
               .orderBy(
                   col("accountId").desc(),
                   col("customerId"),
                   col("sessionDate").desc()
               )


from pyspark.sql import functions as F
from pyspark.sql import Window

customerAccountWindow = Window.partitionBy("accountId","customerId")

outputDf = (
    df.withColumn(
        "prior_oldest_purchase",
        F.min(
            F.when(F.col("didPurchase"),F.col("sessionDate"))
        ).over(customerAccountWindow)
    )
    .where(F.col("sessionDate") <= F.col("prior_oldest_purchase"))
    .select(
        F.col("accountId"),
        F.col("customerId"),
        F.col("sessionDate"),
        F.col("didPurchase")
     )
     .orderBy(
        F.col("accountId").desc(),
        F.col("customerId"),
        F.col("sessionDate").desc()
      )
)

-----------------------

SELECT
    accountId,
    customerId,
    sessionDate,
    didPurchase
FROM (
    SELECT 
        *,
        MIN(CASE
            WHEN didPurchase THEN sessionDate
        END) OVER (
            PARTITION BY accountId, customerId
        ) as prior_oldest_purchase
     FROM
        myvisits
) v
WHERE
     v.sessionDate <= v.prior_oldest_purchase
ORDER BY
    accountId DESC, customerId,sessionDate DESC

import org.apache.spark.sql.functions._
import org.apache.spark.sql.expressions.Window

val customerAccountWindow = Window.partitionBy("accountId","customerId")

val outputDf = df.withColumn(
                    "prior_oldest_purchase",
                    min(
                        when(col("didPurchase"),col("sessionDate"))
                    ).over(customerAccountWindow)
               )
               .where(col("sessionDate") <= col("prior_oldest_purchase"))
               .select(
                   col("accountId"),
                   col("customerId"),
                   col("sessionDate"),
                   col("didPurchase")
               )
               .orderBy(
                   col("accountId").desc(),
                   col("customerId"),
                   col("sessionDate").desc()
               )


from pyspark.sql import functions as F
from pyspark.sql import Window

customerAccountWindow = Window.partitionBy("accountId","customerId")

outputDf = (
    df.withColumn(
        "prior_oldest_purchase",
        F.min(
            F.when(F.col("didPurchase"),F.col("sessionDate"))
        ).over(customerAccountWindow)
    )
    .where(F.col("sessionDate") <= F.col("prior_oldest_purchase"))
    .select(
        F.col("accountId"),
        F.col("customerId"),
        F.col("sessionDate"),
        F.col("didPurchase")
     )
     .orderBy(
        F.col("accountId").desc(),
        F.col("customerId"),
        F.col("sessionDate").desc()
      )
)

1st Sept coming before 31st Aug in bar chart ordering by date in Zeppelin, how to fix please?

Copy

Download

SELECT FROM_TIMESTAMP(DATE_TRUNC('HOUR', concat(replace(my_timestamp,'"',''), "Z")), 'd MMM HH:mm') AS hours, 
       FROM_TIMESTAMP(DATE_TRUNC('HOUR', concat(replace(my_timestamp,'"',''), "Z")), 'yyyy-MM-dd HH:mm') as dt,
       COUNT(my_number) AS "number per hour"
FROM my_table
WHERE unix_timestamp(my_timestamp) --also it seems Z should be removed, etc 
      > (unix_timestamp(now()) - 86400)
GROUP BY dt, hours
ORDER BY dt
LIMIT 24;

AWS EMR: Zeppelin taking numpy version from python 2.7 instead of higher version

Copy

Download


sudo python3 -m pip install h5py==2.10.0
sudo python3 -m pip install keras==2.3.1
sudo python3 -m pip install keras_applications==1.0.8 --no-deps
sudo python3 -m pip install keras_preprocessing==1.1.0 --no-deps

sudo python3 -m pip install tqdm==4.40.0

sudo python3 -m pip install s3fs 
sudo python3 -m pip install ipaddress==1.0.23
sudo python3 -m pip install netaddr==0.7.19 

sudo python3 -m pip install matplotlib

sudo python3 -m pip install pyarrow==0.12.1
sudo python3 -m pip install boto3
sudo python3 -m pip install torch==1.6.0
sudo python3 -m pip install --upgrade scipy==1.4.1
sudo python3 -m pip install torchvision
sudo python3 -m pip install pydot==1.4.1
sudo python3 -m pip install xlrd
sudo python3 -m pip install xlwt
sudo python3 -m pip install pandas==1.2.0
sudo python3 -m pip install scikit-learn
sudo python3 -m pip install scikit-multilearn 
sudo python3 -m pip install wrapt==1.12.0
sudo python3 -m pip install tensorboard==2.1.0
sudo python3 -m pip install tensorflow==2.1.0
sudo python3 -m pip install tensorflow-estimator==2.1.0

Apache Zeppelin - Jquery datepicker in angular cell doesn't work

Copy

Download

%angular
<link rel="stylesheet" href="//code.jquery.com/ui/1.12.0/themes/base/jquery-ui.css">
<script src="https://code.jquery.com/jquery-1.12.4.js"></script>
<script src="https://code.jquery.com/ui/1.12.0/jquery-ui.js"></script>

<script>
    angular.element( function() {
        angular.element( "#todatepicker" ).datepicker({ dateFormat: 'yy-mm-dd',changeMonth: true,changeYear: true, minDate: new Date(1900, 1, 1), yearRange: '1900:+00' });
        angular.element( "#fromdatepicker" ).datepicker({ dateFormat: 'yy-mm-dd',changeMonth: true,changeYear: true,  minDate: new Date(1900, 1, 1), yearRange: '1900:+00' });
    } );
    
    function changeMaxDate(val){
         angular.element('#fromdatepicker').datepicker('option', 'maxDate', new Date(val));
    }
    
    function changeMinDate(val){
        angular.element('#todatepicker').datepicker('option', 'minDate', new Date(val));
    } 
        
</script>

<form class="form-inline">

    <div style="text-align:center; margin-bottom:20px">
    <button type="submit" class="btn btn-primary"  ng-click="z.runParagraph('20210728-173149_661735685')" > Load data </button>
    </div>

    <div style="text-align:center">

            <label for="fromDateId" >From: </label>
            <input type="text"  id="fromdatepicker" ng-model="fromDate" onChange="changeMinDate(this.value);" autocomplete="off"> </input>
            <label for="toDateId"style="margin-left:5px"> to: </label>
            <input type="text" id="todatepicker" ng-model="toDate" onChange="changeMaxDate(this.value);" autocomplete="off"> </input>

            <label style="margin-left:30px"> City: </label>
            <input type="text" ng-model="city"> </input>

            <label for="genders" style="margin-left:30px">Gender:</label>
            <select name="genders" id="genders" ng-model="gender">
                <option value="both">Both</option>
                <option value="F">Female</option>
                <option value="M">Male</option>
            </select>

    </div>
    <div style="text-align:center; margin-top:20px">
    <button type="submit" class="btn btn-primary" ng-click="z.angularBind('toDate',toDate,'20210727-110725_1586668489');z.angularBind('fromDate',fromDate,'20210727-110725_1586668489');z.angularBind('city',city,'20210727-110725_1586668489');z.angularBind('gender',gender,'20210727-110725_1586668489');z.runParagraph('20210727-110725_1586668489');z.runParagraph('20210727-111144_1584153174')">Search</button>
    </div>
</form>

See all related Code Snippets

Community Discussions

Trending Discussions on zeppelin

Cannot find conda info. Please verify your conda installation on EMR
Dataproc Cluster creation is failing with PIP error "Could not build wheels"
Does Zeppelin 0.10.0 try to run interpreter in k8s cluster?
Apache Zeppelin Error When Importing Pandas
Printing Unique Values in a Column as a Percentage of Total Rows
User "system:serviceaccount:default:flink" cannot list resource "nodes" in API group "" at the cluster scope
CSS Keyframe Animations
totalsupply() is not a function openzeppelin contracts
Spark - Scala: "error: not found: value transform"
How can find the find occurence in a window/group and fetch all unbounded rows prior to that?

Trending Discussions on zeppelin

QUESTION

Cannot find conda info. Please verify your conda installation on EMR

Asked 2022-Feb-05 at 00:17

I am trying to install conda on EMR and below is my bootstrap script, it looks like conda is getting installed but it is not getting added to environment variable. When I manually update the $PATH variable on EMR master node, it can identify conda. I want to use conda on Zeppelin.

I also tried adding condig into configuration like below while launching my EMR instance however I still get the below mentioned error.

    "classification": "spark-env",
    "properties": {
        "conda": "/home/hadoop/conda/bin"
    }

[hadoop@ip-172-30-5-150 ~]$ PATH=/home/hadoop/conda/bin:$PATH
[hadoop@ip-172-30-5-150 ~]$ conda
usage: conda [-h] [-V] command ...

conda is a tool for managing and deploying applications, environments and packages.

#!/usr/bin/env bash


# Install conda
wget https://repo.continuum.io/miniconda/Miniconda3-4.2.12-Linux-x86_64.sh -O /home/hadoop/miniconda.sh \
    && /bin/bash ~/miniconda.sh -b -p $HOME/conda


conda config --set always_yes yes --set changeps1 no
conda install conda=4.2.13
conda config -f --add channels conda-forge
rm ~/miniconda.sh
echo bootstrap_conda.sh completed. PATH now: $PATH
export PYSPARK_PYTHON="/home/hadoop/conda/bin/python3.5"

echo -e '\nexport PATH=$HOME/conda/bin:$PATH' >> $HOME/.bashrc && source $HOME/.bashrc


conda create -n zoo python=3.7 # "zoo" is conda environment name, you can use any name you like.
conda activate zoo
sudo pip3 install tensorflow
sudo pip3 install boto3
sudo pip3 install botocore
sudo pip3 install numpy
sudo pip3 install pandas
sudo pip3 install scipy
sudo pip3 install s3fs
sudo pip3 install matplotlib
sudo pip3 install -U tqdm
sudo pip3 install -U scikit-learn
sudo pip3 install -U scikit-multilearn
sudo pip3 install xlutils
sudo pip3 install natsort
sudo pip3 install pydot
sudo pip3 install python-pydot
sudo pip3 install python-pydot-ng
sudo pip3 install pydotplus
sudo pip3 install h5py
sudo pip3 install graphviz
sudo pip3 install recmetrics
sudo pip3 install openpyxl
sudo pip3 install xlrd
sudo pip3 install xlwt
sudo pip3 install tensorflow.io
sudo pip3 install Cython
sudo pip3 install ray
sudo pip3 install zoo
sudo pip3 install analytics-zoo
sudo pip3 install analytics-zoo[ray]
#sudo /usr/bin/pip-3.6 install -U imbalanced-learn

ANSWER

Answered 2022-Feb-05 at 00:17

I got the conda working by modifying the script as below, emr python versions were colliding with the conda version.:

wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.9.2-Linux-x86_64.sh  -O /home/hadoop/miniconda.sh \
    && /bin/bash ~/miniconda.sh -b -p $HOME/conda

echo -e '\n export PATH=$HOME/conda/bin:$PATH' >> $HOME/.bashrc && source $HOME/.bashrc


conda config --set always_yes yes --set changeps1 no
conda config -f --add channels conda-forge


conda create -n zoo python=3.7 # "zoo" is conda environment name
conda init bash
source activate zoo
conda install python 3.7.0 -c conda-forge orca 
sudo /home/hadoop/conda/envs/zoo/bin/python3.7 -m pip install virtualenv

and setting zeppelin python and pyspark parameters to:

“spark.pyspark.python": "/home/hadoop/conda/envs/zoo/bin/python3",
"spark.pyspark.virtualenv.enabled": "true",
"spark.pyspark.virtualenv.type":"native",
"spark.pyspark.virtualenv.bin.path":"/home/hadoop/conda/envs/zoo/bin/,
"zeppelin.pyspark.python" : "/home/hadoop/conda/bin/python",
"zeppelin.python": "/home/hadoop/conda/bin/python"

Orca only support TF upto 1.5 hence it was not working as I am using TF2.

Source https://stackoverflow.com/questions/70901724

QUESTION

Dataproc Cluster creation is failing with PIP error "Could not build wheels"

Asked 2022-Jan-24 at 13:04

We use to spin cluster with below configurations. It used to run fine till last week but now failing with error ERROR: Failed cleaning build dir for libcst Failed to build libcst ERROR: Could not build wheels for libcst which use PEP 517 and cannot be installed directly

Building wheels for collected packages: pynacl, libcst
  Building wheel for pynacl (PEP 517): started
  Building wheel for pynacl (PEP 517): still running...
  Building wheel for pynacl (PEP 517): finished with status 'done'
  Created wheel for pynacl: filename=PyNaCl-1.5.0-cp37-cp37m-linux_x86_64.whl size=201317 sha256=4e5897bc415a327f6b389b864940a8c1dde9448017a2ce4991517b30996acb71
  Stored in directory: /root/.cache/pip/wheels/2f/01/7f/11d382bf954a093a55ed9581fd66c3b45b98769f292367b4d3
  Building wheel for libcst (PEP 517): started
  Building wheel for libcst (PEP 517): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /opt/conda/anaconda/bin/python /opt/conda/anaconda/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpon3bonqi
       cwd: /tmp/pip-install-9ozf4fcp/libcst

Cluster configuration command :

gcloud dataproc clusters create cluster-test \
--enable-component-gateway \
--region us-east1 \
--zone us-east1-b \
--master-machine-type n1-highmem-32 \
--master-boot-disk-size 500 \
--num-workers 3 \
--worker-machine-type n1-highmem-16 \
--worker-boot-disk-size 500 \
--optional-components ANACONDA,JUPYTER,ZEPPELIN \
--image-version 1.5.54-ubuntu18 \
--tags <tag-name> \
--bucket '<cloud storage bucket>' \
--initialization-actions 'gs://goog-dataproc-initialization-actions-us-east1/connectors/connectors.sh','gs://goog-dataproc-initialization-actions-us-east1/python/pip-install.sh' \
--metadata='PIP_PACKAGES=wheel datalab xgboost==1.3.3 shap oyaml click apache-airflow apache-airflow-providers-google' \
--initialization-action-timeout 30m \
--metadata gcs-connector-version=2.1.1,bigquery-connector-version=1.1.1,spark-bigquery-connector-version=0.17.2 \
--project <project-name>

Things I tried: a) I tried to install wheel package explicitly as part of pip packages but the issue does not resolve

b) Gcloud Command with upgrade pip script:

gcloud dataproc clusters create cluster-test \
--enable-component-gateway \
--region us-east1 \
--zone us-east1-b \
--master-machine-type n1-highmem-32 \
--master-boot-disk-size 500 \
--num-workers 3 \
--worker-machine-type n1-highmem-16 \
--worker-boot-disk-size 500 \
--optional-components ANACONDA,JUPYTER,ZEPPELIN \
--image-version 1.5.54-ubuntu18 \
--tags <tag-name> \
--bucket '<cloud storage bucket>' \
--initialization-actions 'gs://goog-dataproc-initialization-actions-us-east1/connectors/connectors.sh','gs://<bucket-path>/upgrade-pip.sh','gs://goog-dataproc-initialization-actions-us-east1/python/pip-install.sh' \
--metadata='PIP_PACKAGES=wheel datalab xgboost==1.3.3 shap oyaml click apache-airflow apache-airflow-providers-google' \
--initialization-action-timeout 30m \
--metadata gcs-connector-version=2.1.1,bigquery-connector-version=1.1.1,spark-bigquery-connector-version=0.17.2 \
--project <project-name>

ANSWER

Answered 2022-Jan-19 at 21:50

Seems you need to upgrade pip, see this question.

But there can be multiple pips in a Dataproc cluster, you need to choose the right one.

For init actions, at cluster creation time, /opt/conda/default is a symbolic link to either /opt/conda/miniconda3 or /opt/conda/anaconda, depending on which Conda env you choose, the default is Miniconda3, but in your case it is Anaconda. So you can run either /opt/conda/default/bin/pip install --upgrade pip or /opt/conda/anaconda/bin/pip install --upgrade pip.
For custom images, at image creation time, you want to use the explicit full path, /opt/conda/anaconda/bin/pip install --upgrade pip for Anaconda, or /opt/conda/miniconda3/bin/pip install --upgrade pip for Miniconda3.

So, you can simply use /opt/conda/anaconda/bin/pip install --upgrade pip for both init actions and custom images.

Source https://stackoverflow.com/questions/70743642

	version
Java	1.8.0_242-b08
Spark	2.4.5
Zeppelin	0.10.0

QUESTION

Apache Zeppelin Error When Importing Pandas

Asked 2021-Nov-14 at 20:16

I'm facing a strange error when importing the Pandas library into my Zeppelin notebook. Here is the basic code that I have as part of my cell:

%python

import pandas as pd

df = pd.read_csv (r'target/youtube_videos.csv')
print (df)

I get the following Error:

Fail to execute line 3: import pandas as pd
Traceback (most recent call last):
  File "/tmp/1636039066525-0/zeppelin_python.py", line 153, in <module>
    exec(code, _zcUserQueryNameSpace)
  File "<stdin>", line 3, in <module>
ModuleNotFoundError: No module named 'pandas'

I tried to see what my Python path looks like and here it is:

%sh
python --version
python3-config --configdir

This gives me the following:

Python 3.7.0b3
/usr/lib/python3.8/config-3.8-x86_64-linux-gnu

I'm using Zeppelin 0.10.0.

EDIT:

I tried the following:

joesan@joesan-InfinityBook-S-14-v5:~/Projects/Private/ml-projects/ml-data-preparation-sandbox$ zstart
Please specify HADOOP_CONF_DIR if USE_HADOOP is true
Zeppelin start                                             [  OK  ]
joesan@joesan-InfinityBook-S-14-v5:~/Projects/Private/ml-projects/ml-data-preparation-sandbox$ python
Python 3.7.0b3 (default, Mar 30 2018, 04:35:22) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'
>>>

Pandas seems to be already installed:

joesan@joesan-InfinityBook-S-14-v5:~/Projects/Private/ml-projects/ml-data-preparation-sandbox$ pip3 install pandas
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (1.3.4)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/lib/python3/dist-packages (from pandas) (2.7.3)
Requirement already satisfied: numpy>=1.17.3 in /usr/lib/python3/dist-packages (from pandas) (1.17.4)
Requirement already satisfied: pytz>=2017.3 in /usr/lib/python3/dist-packages (from pandas) (2019.3)
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
joesan@joesan-InfinityBook-S-14-v5:~/Projects/Private/ml-projects/ml-data-preparation-sandbox$

I have even set the python interpreter in Zeppelin as below:

ANSWER

Answered 2021-Nov-04 at 17:10

Are you sure you even have pandas installed? Unless Zeppelin uses its own Python, that would be the problem. Give pip3 install pandas a shot.

Source https://stackoverflow.com/questions/69842021

QUESTION

User "system:serviceaccount:default:flink" cannot list resource "nodes" in API group "" at the cluster scope

Asked 2021-Oct-13 at 09:37

I am trying to call k8s api in one k8s pod. But hit the following permission issue:

User "system:serviceaccount:default:flink" cannot list resource "nodes" in API group "" at the cluster scope.

In my yaml file, I already have specified the Role & RoleBinding. What do I miss here?

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flink
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: zeppelin-server-role
rules:
- apiGroups: [""]
  resources: ["pods", "services", "configmaps", "deployments", "nodes"]
  verbs: ["create", "get", "update", "patch", "list", "delete", "watch"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["roles", "rolebindings"]
  verbs: ["bind", "create", "get", "update", "patch", "list", "delete", "watch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: zeppelin-server-role-binding
  namespace: default
subjects:
- kind: ServiceAccount
  name: flink
roleRef:
  kind: ClusterRole
  name: zeppelin-server-role
  apiGroup: rbac.authorization.k8s.io

ANSWER

Answered 2021-Oct-12 at 13:12

You are deploying zeppelin-server on Kubernetes, right? Your yaml file with the service account looks good as I suppose, however to be sure that this works, you should follow the next steps:

kubectl get clusterrole

and you should get zeppelin-server-role role.

check if your account 'flink' has a binding to clusterrole "zeppelin-server-role"

kubectl get clusterrole clusterrolebinding

if there is no, you can create it by the following command:

kubectl create clusterrolebinding zeppelin-server-role-binding --clusterrole=zeppelin-server-role --serviceaccount=default:flink

finally, check if you really act as this account:

kubectl get deploy flink-deploy -o yaml

if you can't see the settings "serviceAccount" and "serviceAccountName" from the output something like:

...
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
...

then add this account you want flink to use:

kubectl patch deploy flink-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

Source https://stackoverflow.com/questions/69535118

QUESTION

CSS Keyframe Animations

Asked 2021-Oct-06 at 19:16

I have a really basic bit of HTML & CSS and trying to achieve a keyframes animation that's an automatic slideshow. I copied a tutorial that worked absolutely fine but struggling to configure it myself.

The html & CSS is :

#SLIDE_BG {
  width: 100%;
  height: 100vh;
  background-position: center center;
  background-size: cover;
  background-repeat: no-repeat;
  backface-visibility: hidden;
  animation: slideBg 8s linear infinite 0s;
  animation-timing-function: ease-in-out;
  background-image: url('/assets/images/jack-test/beatles-one.jpg');
}

@keyframes slideBg {
  0% {
    background-image: url('/assets/images/jack-test/beatles-oone.jpeg');
  }
  25% {
    background-image: url('/assets/images/jack-test/beatles-two.jpeg');
  }
  50% {
    background-image: url('/assets/images/jack-test/jimi-hendix.jpeg');
  }
  75% {
    background-image: url('/assets/images/jack-test/led-zeppelin.jpeg');
  }
  100% {
    background-image: url('/assets/images/jack-test/rock-band.jpeg');
  }
}

<div id="SLIDE_BG"></div>

ANSWER

Answered 2021-Oct-04 at 13:36

You need to give image path properly check below

#SLIDE_BG {
  width: 100%;
  height: 100vh;
  background-position: center center;
  background-size: cover;
  background-repeat: no-repeat;
  backface-visibility: hidden;
  animation: slideBg 8s linear infinite 0s;
  animation-timing-function: ease-in-out;
  background-image: url('https://dummyimage.com/800x300');         
}

@keyframes slideBg {
  0% {
    background-image: url('https://upload.wikimedia.org/wikipedia/commons/2/29/Dscn7471_sunset-sundog_crop_800x300.jpg');
  }
  25% {
    background-image: url('https://www.thegrandsiba.com/wp-content/uploads/2017/06/MG_28423-800x300.jpg');
  }
  50% {
    background-image: url('https://dummyimage.com/800x300');
  }
  75% {
    background-image: url('https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/5e2a19d4-f261-4548-b4a7-1e2f5ad139af/d99najf-e9049704-bfc4-4836-952d-9315403bd60f.gif');
  }
  100% {
    background-image: url('https://www.businessclass.co.uk/wp-content/uploads/sites/11/2016/04/Toronto-800x300-800x300.jpg');
  }
}

Source https://stackoverflow.com/questions/69436632

QUESTION

totalsupply() is not a function openzeppelin contracts

Asked 2021-Oct-04 at 08:25

I'm trying to import some contract files from open zeppelin so my solidity smart contracts can inherit their functionality, when trying to write chai tests that run on my smart contracts at compile time I get an error in my chai test.

  3 passing (2s)
  1 failing

  1) Contract: Color
       minting
         creates a new token:
     TypeError: contract.totalSupply is not a function

my contract importing the openzeppelin contracts

pragma solidity 0.8.7;

import "@openzeppelin/contracts/token/ERC721/ERC721.sol"; //import base functionality
import "@openzeppelin/contracts/token/ERC721/extensions/ERC721Enumerable.sol"; //import totalsupply()

contract color is ERC721 {
    string[] public colors;
    mapping(string => bool) _colorExists; //mappings are like json objects where value a is searched and its value is returned
    constructor() ERC721("Color", "COLOR") {
    }

    function mint(string memory _color) public{
      colors.push(_color);
      uint _id = colors.length -1;

      _mint(msg.sender,_id);
      _colorExists[_color] = true;
 
    }
}

and lastly my test file ( I have shortened it to show only the test giving me errors)

const { assert } = require('chai')
const Color = artifacts.require('./Color.sol')

require('chai')
.use(require('chai-as-promised'))
.should()

contract('Color', (accounts) =>{
    let FormControlStatic

    before(async ()=>{
        contract = 
        await Color.deployed()
    })
    
    describe('minting', async ()=>{
        
        it('creates a new token', async ()=>{
            const result = await contract.mint('#EC058E')
            console.log(result)
            const totalSupply = await contract.totalSupply()

            assert.equal(totalSupply,1)
            console.log(result)
        })
    })
})

also if we look at the file containing the function totalSupply() it is publicly scoped so it should be visible outside the function via import

I did some digging and imported the file that the actual function IS in from openzeppelin however it seems that I still get the same error, I tried compiling separately to see if recompiling after changing would resolve but it didn't

not sure if anyone else has gone through this recently or might have a solution also I'm importing the current version here https://www.npmjs.com/package/@openzeppelin/contracts

thanks!

ANSWER

Answered 2021-Aug-18 at 11:32

We must extend IERC721Enumerable contracts, and, implements its virtual functions.

contract Color is ERC721, IERC721Enumerable { // We must extends IERC721Enumerable 
    string[] public colors;
    mapping(string => bool) _colorExists;

    constructor() ERC721("Color", "COLOR") {}

    function mint(string memory _color) public {
        colors.push(_color);
        uint256 _id = colors.length - 1;

        // _mint(msg.sender,_id);
        _colorExists[_color] = true;
    }
    

    // And must override below three functions

    function tokenOfOwnerByIndex(address owner, uint256 index) public view override returns (uint256) {
        // You need update this logic.
        // ...
        return 3;
    }

    function totalSupply() external  view override returns (uint256) {
      // You need update this logic.
      // ...
      return 1;
    }

    function tokenByIndex(uint256 index) external view  override returns (uint256) {
      // You need update this logic.
      // ...
      return 5;
    }
}

Then, we can call totalSupply() method

Source https://stackoverflow.com/questions/68810515

QUESTION

How can find the find occurence in a window/group and fetch all unbounded rows prior to that?

Asked 2021-Sep-29 at 09:38

I have the following Dataframe that contains sessions along with the customer-id and b boolean indicating a purchase.

+---------+----------+-----------+-----------+
|accountId|customerId|sessionDate|didPurchase|
+---------+----------+-----------+-----------+
|walgreens|       001| 2021-09-30|      false|
|walgreens|       001| 2021-07-30|       true|
|walgreens|       001| 2021-05-30|       true|
|walgreens|       001| 2021-04-30|      false|
|walgreens|       002| 2021-08-04|       true|
|walgreens|       002| 2021-07-06|      false|
|walgreens|       002| 2021-07-05|      false|
|walgreens|       002| 2021-07-01|      false|
|    tesco|       001| 2021-09-21|      false|
|    tesco|       001| 2021-09-20|      false|
|    tesco|       001| 2021-01-30|      false|
|    tesco|       001| 2021-01-01|      true |
+---------+----------+-----------+-----------+

For each window of accountId and customerId, I need to get all rows prior to the oldest purchase (i.e. didPurchase should be true) when you group by accountId and customerId and sort by sessionDate descending.

My expected resultset should be so:

+---------+----------+-----------+-----------+
|accountId|customerId|sessionDate|didPurchase|
+---------+----------+-----------+-----------+
|walgreens|       001| 2021-05-30|       true|
|walgreens|       001| 2021-04-30|      false|
|walgreens|       002| 2021-08-04|       true|
|walgreens|       002| 2021-07-06|      false|
|walgreens|       002| 2021-07-05|      false|
|walgreens|       002| 2021-07-01|      false|
|    tesco|       001| 2021-01-01|      true |
+---------+----------+-----------+-----------+

The snippet below is what I've used in Zeppelin to create the df. I'm familiar with SQL window functions but baffled me.

   SELECT accountId
        , customerId
        , sessionDate
        , ROW_NUMBER() OVER (PARTITION BY accountId, customerId ORDER BY sessionDate DESC) AS rn
        , COUNT(*) OVER (PARTITION BY accountId, customerId) AS visits_per_customer
     FROM myvisits

Is this where I need to use UNBOUNDED PRECEDING? is so, a small SQL snippet would help. Thank you.

import java.sql.Date

val df = sc.parallelize(Seq(
  ("walgreens", "001", Date.valueOf("2021-09-30"),  false),
  ("walgreens", "001", Date.valueOf("2021-07-30"),  true), 
  ("walgreens", "001", Date.valueOf("2021-05-30"),  true), 
  ("walgreens", "001", Date.valueOf("2021-04-30"),  false),
  ("walgreens", "002", Date.valueOf("2021-08-04"),  true),
  ("walgreens", "002", Date.valueOf("2021-07-06"),  false), 
  ("walgreens", "002", Date.valueOf("2021-07-05"),  false), 
  ("walgreens", "002", Date.valueOf("2021-07-01"),  false),
  ("tesco",     "001", Date.valueOf("2021-09-21"),  false), 
  ("tesco",     "001", Date.valueOf("2021-09-20"),  false),
  ("tesco",     "001", Date.valueOf("2021-01-30"),  false),
  ("tesco",     "001", Date.valueOf("2021-01-01"),  true)))
  .toDF("accountId", "customerId", "sessionDate", "didPurchase")
  
df.registerTempTable("myvisits")
df.show()

ANSWER

Answered 2021-Sep-29 at 09:38

You may use the MIN window function with a CASE expression to identify the oldest purchase sessionDate and filter records that have occurred on or before that to achieve your desired results.

The sql to achieve this could look like

SELECT
    accountId,
    customerId,
    sessionDate,
    didPurchase
FROM (
    SELECT 
        *,
        MIN(CASE
            WHEN didPurchase THEN sessionDate
        END) OVER (
            PARTITION BY accountId, customerId
        ) as prior_oldest_purchase
     FROM
        myvisits
) v
WHERE
     v.sessionDate <= v.prior_oldest_purchase
ORDER BY
    accountId DESC, customerId,sessionDate DESC

View live demo working db fiddle

Using the scala api, this could look like

import org.apache.spark.sql.functions._
import org.apache.spark.sql.expressions.Window

val customerAccountWindow = Window.partitionBy("accountId","customerId")

val outputDf = df.withColumn(
                    "prior_oldest_purchase",
                    min(
                        when(col("didPurchase"),col("sessionDate"))
                    ).over(customerAccountWindow)
               )
               .where(col("sessionDate") <= col("prior_oldest_purchase"))
               .select(
                   col("accountId"),
                   col("customerId"),
                   col("sessionDate"),
                   col("didPurchase")
               )
               .orderBy(
                   col("accountId").desc(),
                   col("customerId"),
                   col("sessionDate").desc()
               )

or using the pyspark api


from pyspark.sql import functions as F
from pyspark.sql import Window

customerAccountWindow = Window.partitionBy("accountId","customerId")

outputDf = (
    df.withColumn(
        "prior_oldest_purchase",
        F.min(
            F.when(F.col("didPurchase"),F.col("sessionDate"))
        ).over(customerAccountWindow)
    )
    .where(F.col("sessionDate") <= F.col("prior_oldest_purchase"))
    .select(
        F.col("accountId"),
        F.col("customerId"),
        F.col("sessionDate"),
        F.col("didPurchase")
     )
     .orderBy(
        F.col("accountId").desc(),
        F.col("customerId"),
        F.col("sessionDate").desc()
      )
)

Let me know if this works for you.

Source https://stackoverflow.com/questions/69368505

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

CVE-2017-12619 HIGH

Apache Zeppelin prior to 0.7.3 was vulnerable to session fixation which allowed an attacker to hijack a valid user session. Issue was reported by "stone lone".

CVE-2018-1317 HIGH

In Apache Zeppelin prior to 0.8.0 the cron scheduler was enabled by default and could allow users to run paragraphs as other users without authentication.

CVE-2018-1328 MEDIUM

Apache Zeppelin prior to 0.8.0 had a stored XSS issue via Note permissions. Issue reported by "Josna Joseph".

Install zeppelin

Please go to install to install Apache Zeppelin from binary package.
Please check Build from source to build Zeppelin from source.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .