Flutter - How to refresh a Widget on button Click?
Find element by Xpath. How to split the element I don't want inside the Xpath
Can't use Selenium to find elements. Try to avoid deprecated commands warning, got result only one element. (on Colab)
How to programmatically validate Tether (TRC20) Wallet Addresses in PHP?
Custom Loss Function - Keras
Running python file on windows CMD vs VScode
How to turn HTML widget code into NEXTjs code to use in an app (CoinMarketCap Price Marquee Ticker)
How can I properly graph these two datasets using Pandas in Matplotlib?
For each item update database
Why are certain time values not plotting on this graph?
QUESTION
Flutter - How to refresh a Widget on button Click?
Asked 2022-Mar-01 at 10:15
I'm making an app where I plot charts of cryptocurrency. I have a file like this, that renders a chart of btc/24h in it. I'm using chart_sparkline package to plot the chart.
I have the following code, and it is not working on a button click, how do I fix this?
Basically, the elevatedButton should update the chart, but it is not working in my case, how to fix this?
ANSWER
Answered 2022-Mar-01 at 10:15
I fixed this, I was passing constructor parameters of Portfolio to _PortfolioState, this won't update the values the next time. Instead of passing these values to _PortfolioState, I used widget.coin and widget.days to extract the data from the Portfolio class.
Find element by Xpath. How to split the element I don't want inside the Xpath
Asked 2022-Feb-15 at 09:12
I try to scrape a website using Selenium.
I got a problem when I try to get the coins name. because there're 2 elements inside 'td'
How can I get rid of another element I don't want. or keep track to only its first element.
(I found this post but I'm not sure if it answer my issue or not)
This is my whole code
1#driver chrome def
2website = 'https://www.bitkub.com/fee/cryptocurrency'
3path = r"C:\\Users\\USER\\Downloads\\chromedriver.exe"
4options = Options()
5options.add_argument("start-maximized")
6driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
7driver.get(website)
89#giving variable
10coin_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[2]//span")))]
11chain_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[3]//div")))]
12withdrawal_fees = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[4]//div")))]
13#print(coin_name)
14#print(chain_name)
15#print(withdrawal_fees)
161718#for loop make list
1920for coin, chains, wdf in zip(coin_name, chain_name, withdrawal_fees):
21 print("Coin name: {} Chain: {} Fee: {}".format(coin, chains, wdf))
22
The input of coin_name (which I mentioned that it got 2 elements)
1#driver chrome def
2website = 'https://www.bitkub.com/fee/cryptocurrency'
3path = r"C:\\Users\\USER\\Downloads\\chromedriver.exe"
4options = Options()
5options.add_argument("start-maximized")
6driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
7driver.get(website)
89#giving variable
10coin_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[2]//span")))]
11chain_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[3]//div")))]
12withdrawal_fees = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[4]//div")))]
13#print(coin_name)
14#print(chain_name)
15#print(withdrawal_fees)
161718#for loop make list
1920for coin, chains, wdf in zip(coin_name, chain_name, withdrawal_fees):
21 print("Coin name: {} Chain: {} Fee: {}".format(coin, chains, wdf))
22['Civic(CVC)', '(CVC)', 'Bitcoin SV(BSV)', '(BSV)', 'Ethereum(ETH)', '(ETH)', 'Bitkub Coin(KUB)', '(KUB)', 'Compound(COMP)', '(COMP)', 'Curve DAO Token(CRV)', '(CRV)', .... ]
23
This is how element on the website look like
I wanted input to look like this so I can make dataframe out of it
1#driver chrome def
2website = 'https://www.bitkub.com/fee/cryptocurrency'
3path = r"C:\\Users\\USER\\Downloads\\chromedriver.exe"
4options = Options()
5options.add_argument("start-maximized")
6driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
7driver.get(website)
89#giving variable
10coin_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[2]//span")))]
11chain_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[3]//div")))]
12withdrawal_fees = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[4]//div")))]
13#print(coin_name)
14#print(chain_name)
15#print(withdrawal_fees)
161718#for loop make list
1920for coin, chains, wdf in zip(coin_name, chain_name, withdrawal_fees):
21 print("Coin name: {} Chain: {} Fee: {}".format(coin, chains, wdf))
22['Civic(CVC)', '(CVC)', 'Bitcoin SV(BSV)', '(BSV)', 'Ethereum(ETH)', '(ETH)', 'Bitkub Coin(KUB)', '(KUB)', 'Compound(COMP)', '(COMP)', 'Curve DAO Token(CRV)', '(CRV)', .... ]
23['Civic(CVC)', 'Bitcoin SV(BSV)', 'Ethereum(ETH)', 'Bitkub Coin(KUB)', 'Compound(COMP)', 'Curve DAO Token(CRV)', .... ]
24
ANSWER
Answered 2022-Feb-15 at 09:12
Hope this helps:
1#driver chrome def
2website = 'https://www.bitkub.com/fee/cryptocurrency'
3path = r"C:\\Users\\USER\\Downloads\\chromedriver.exe"
4options = Options()
5options.add_argument("start-maximized")
6driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
7driver.get(website)
89#giving variable
10coin_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[2]//span")))]
11chain_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[3]//div")))]
12withdrawal_fees = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[4]//div")))]
13#print(coin_name)
14#print(chain_name)
15#print(withdrawal_fees)
161718#for loop make list
1920for coin, chains, wdf in zip(coin_name, chain_name, withdrawal_fees):
21 print("Coin name: {} Chain: {} Fee: {}".format(coin, chains, wdf))
22['Civic(CVC)', '(CVC)', 'Bitcoin SV(BSV)', '(BSV)', 'Ethereum(ETH)', '(ETH)', 'Bitkub Coin(KUB)', '(KUB)', 'Compound(COMP)', '(COMP)', 'Curve DAO Token(CRV)', '(CRV)', .... ]
23['Civic(CVC)', 'Bitcoin SV(BSV)', 'Ethereum(ETH)', 'Bitkub Coin(KUB)', 'Compound(COMP)', 'Curve DAO Token(CRV)', .... ]
24for name in coin_name:
25 if name[0] == "(":
26 coin_name.remove(name)
27
Can't use Selenium to find elements. Try to avoid deprecated commands warning, got result only one element. (on Colab)
Asked 2022-Feb-09 at 22:06
As my title said. To avoid deprecated warning of Selenium.I try to use the new format as I found.
and somehow result return only 1 element. How can I fix it to return elements?
FYI I scraping website. I'm working on Colab so I can share with my workmates easier.
I am developing a regression model to predict cryptocurrency prices, and I have created a simple loss function. The idea is simple the Y target is the price change from a certain lookup window, so either the values are positive or negative. The idea is to first apply an mae loss function and then penalize where y_pred is positive and y_true is negative and vice versa. And reduce the loss where y_pred is positive and y_true is also positive and vice versa. Yet when I train with my loss function the precision does not get higher then 0.50 where it gets to around 0.535 with the regular mae loss function. Any idea what could cause this?
The loss function:
1def loss_fn(
2 # the loss mode [mae, rmse, mape, huber].
3 mode="mae",
4 # the threshold.
5 threshold=0.0,
6 # penalize incorrect predictions (when predicted positive and is negative and reversed) (should be >= 1).
7 penalizer=1.0,
8 # reduce correct predictions (when predicted positive and is positive and reversed) (should be <= 1).
9 reducer=1.0,
10):
11 def loss_function(y_true, y_pred):
12 if mode == "mae":
13 loss = keras.losses.MAE(y_true, y_pred)
14 elif mode == "rmse":
15 loss = K.sqrt(K.mean(K.square(y_pred - y_true)))
16 elif mode == "mape":
17 loss = keras.losses.mean_absolute_percentage_error(y_true, y_pred)
18 elif mode == "huber":
19 loss = keras.losses.Huber()(y_true, y_pred)
20 if penalizer != 1.0 or reducer != 1.0:
21 # apply penalizer.
22 mask = tf.where(
23 tf.logical_or(
24 tf.logical_and(K.less_equal(y_pred, -1 * threshold), K.greater(y_true, 0.0)),
25 tf.logical_and(K.greater_equal(y_pred, threshold), K.less(y_true, 0.0)),
26 ),
27 penalizer,
28 1.0,
29 )[:, 0]
30 loss = tf.multiply(loss, mask)
31 # apply reducer.
32 mask = tf.where(
33 tf.logical_or(
34 tf.logical_and(K.less_equal(y_pred, -1 * threshold), K.less(y_true, 0.0)),
35 tf.logical_and(K.greater_equal(y_pred, threshold), K.greater(y_true, 0.0)),
36 ),
37 reducer,
38 1.0,
39 )[:, 0]
40 loss = tf.multiply(loss, mask)
41 loss = tf.math.reduce_mean(loss)
42 return loss
43 return loss_function
44loss = loss_fn(mode="mae", threshold=0.0, penalizer=3.0, reducer=1.0/3.0)
45
Does anyone see any errors or mistakes that could cause this?
Assuming you set the seed for reproducibility, otherwise it could simply be initialization, when you change the loss function you change the domain over which the gradient is going to iterate to optimize your network.
And since you don't have any guarantee that your model is going to reach the global minima but, most likely, will stop at a local minima it coud just mean that, given the same initialization (set seed) the optimization process stopped at a different local minima.
I am running a Cryptocurrency-trading python program on my laptop
It monitors the market every second and when the condition is satisfied, it tries transactions.
When I run it on windows CMD, it causes many problems below.
1. Sometimes, it halts till I click the cmd window and press 'enter' key
2. Sometimes, it causes many unknown errors.
However, when I run that on VScode, it does not cause any problem.
I wonder what makes the difference between those two environments.
ANSWER
Answered 2022-Feb-08 at 12:49
I have had some issues with VSCode not finding libraries and similar, and the reason for that is that VSCode runs its own python. In IDLE (and CMD) you run the raw python in AppData, but VSCode runs an executable they have downloaded via extensions through the program files folder.
My hypothesis is that Python as in the AppData folder is corrupted or wrong in some way, but the Program Files folder is correct. It may even be on a different python version.
Try reinstalling python from the official python website, and run it again. Also double-check that the VSCode Python extension version is the same as the one saved in "C:\Users\YOURUSERNAME\AppData\Local\Programs\Python\Python39".
How to turn HTML widget code into NEXTjs code to use in an app (CoinMarketCap Price Marquee Ticker)
Asked 2022-Feb-08 at 01:26
I'm trying to embed this CoinMarketCap Price Marquee Ticker Widget into my NEXTjs app and am having issues with this. I'll go through what I'm trying to do and walk through my process. Hopefully someone has maybe tried to do this and might have some suggestions.
I tried to create it as a separate div and import the script from off of next/script. However, I don't see anything in the window on my app when I save and load the app. I even added some styling, but I'm not seeing that the script is executing.
This is what I thought would work, but I get this error on "coins" with NEXTjs, where it says the below and same as above, the script doesn't load into the widget div space:
(JSX attribute) coins: string Type '{ id: string; coins: string; currency: string; theme: string; transparent: string; "show-symbol-logo": string; }' is not assignable to type 'DetailedHTMLProps<HTMLAttributes<HTMLDivElement>, HTMLDivElement>'. Property 'coins' does not exist on type 'DetailedHTMLProps<HTMLAttributes<HTMLDivElement>, HTMLDivElement>'.
As an alternative, if you are only using Javascript and React you can create the widget in the index.html and append it to a div when your React component mounts.
How can I properly graph these two datasets using Pandas in Matplotlib?
Asked 2022-Jan-26 at 05:42
I have two datasets that I am trying to plot over each other.
The first dataset is the daily price of Dogecoin. I am using yfinance and mplfinance to chart this.
The second dataset is a CSV file of Dogecoin wallet transactions, which has a column named "Balance", that shows the Balance of the Dogecoin Wallet at the time of the transaction. The balance fluctuates as cryptocurrency comes in/out. Below is the CSV for reference.
I am trying have the Balance as a line chart, to show the fluctuations in balance.
Below is my code. What I am trying to accomplish with this code is to chart the Dogecoin Price, then chart the Balance from the CSV as a line chart, and have the charts overlayed with each other. When displayed on the chart, I am trying to have the dates from both datasets be the same, so the data is properly displayed.
The first problem is I have been unable to figure out how to plot these two charts over each other. The first chart comes from mplfinance and the second chart comes from matplotlib. If these two modules cannot plot over each other, then I can use a csv of the Daily dogecoin price instead of mplfinance and yfinance.
The second problem I have ran into is my Balance plot does not fluctuate when the balance decreases, it only increases.
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25
ANSWER
Answered 2022-Jan-26 at 05:42
Before you can line up the timestamps from the two data sets, there are a number of issues with the csv file that have to be cleaned up first.
This is what the csv file looks like as you are reading it:
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26
A few things to notice about this file:
The time stamp exists in both the Time column, and in the Block column (which you have set as the index), but the block column also contains the block number next to its timestamp.
The balance column contains the word "DOGE" and is therefore obviously a string (not a float).
In fact, all the columns read from the csv file in this way, are strings (except for the Time column due to parse_dates).
I suggest, to begin, only read the Time and Balance colums, and set the time column as the index. At the same time you can reverse the data so that it is in time order from earliest to latest:
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29
Now you can clean up the Balance column by splitting the column string into the actual balance and the word "DOGE", and converting the actual balance to a float:
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29dfb["Balance"] = dfb["Balance"].str.split(expand=True).iloc[:,0] # [:,0] to take only balance and throw away "DOGE"
30dfb["Balance"] = dfb["Balance"].str.replace(',','').astype(float) # remove commas from balance and convert to float.
31print(dfb.head(16))
32print(dfb.tail())
33
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29dfb["Balance"] = dfb["Balance"].str.split(expand=True).iloc[:,0] # [:,0] to take only balance and throw away "DOGE"
30dfb["Balance"] = dfb["Balance"].str.replace(',','').astype(float) # remove commas from balance and convert to float.
31print(dfb.head(16))
32print(dfb.tail())
33
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29dfb["Balance"] = dfb["Balance"].str.split(expand=True).iloc[:,0] # [:,0] to take only balance and throw away "DOGE"
30dfb["Balance"] = dfb["Balance"].str.replace(',','').astype(float) # remove commas from balance and convert to float.
31print(dfb.head(16))
32print(dfb.tail())
33
Now take a look at the data from yfinance, and compare it to the csv file data:
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29dfb["Balance"] = dfb["Balance"].str.split(expand=True).iloc[:,0] # [:,0] to take only balance and throw away "DOGE"
30dfb["Balance"] = dfb["Balance"].str.replace(',','').astype(float) # remove commas from balance and convert to float.
31print(dfb.head(16))
32print(dfb.tail())
33df = yf.Ticker("DOGE-USD").history(period='max')
34df = df.loc["2021-01-01":]
35print(df.head(8))
36print(df.tail())
37
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29dfb["Balance"] = dfb["Balance"].str.split(expand=True).iloc[:,0] # [:,0] to take only balance and throw away "DOGE"
30dfb["Balance"] = dfb["Balance"].str.replace(',','').astype(float) # remove commas from balance and convert to float.
31print(dfb.head(16))
32print(dfb.tail())
33df = yf.Ticker("DOGE-USD").history(period='max')
34df = df.loc["2021-01-01":]
35print(df.head(8))
36print(df.tail())
37
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29dfb["Balance"] = dfb["Balance"].str.split(expand=True).iloc[:,0] # [:,0] to take only balance and throw away "DOGE"
30dfb["Balance"] = dfb["Balance"].str.replace(',','').astype(float) # remove commas from balance and convert to float.
31print(dfb.head(16))
32print(dfb.tail())
33df = yf.Ticker("DOGE-USD").history(period='max')
34df = df.loc["2021-01-01":]
35print(df.head(8))
36print(df.tail())
37
A couple of things to notice:
There is only one row per day in the data from yfinance
The csv data has
multiple rows per day
some days have more rows that others
some days are missing altogether
To be able to join the two data sets, I would suggest first resampling the balance data so that you have only one row per day. I would also suggest doing this in such a way that the balance for each day is the final balance on that date. This can be done using pandas's ohlc() (open,high,low,close) aggregator and then just taking the "close" for each day as the final balance for that date:
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29dfb["Balance"] = dfb["Balance"].str.split(expand=True).iloc[:,0] # [:,0] to take only balance and throw away "DOGE"
30dfb["Balance"] = dfb["Balance"].str.replace(',','').astype(float) # remove commas from balance and convert to float.
31print(dfb.head(16))
32print(dfb.tail())
33df = yf.Ticker("DOGE-USD").history(period='max')
34df = df.loc["2021-01-01":]
35print(df.head(8))
36print(df.tail())
37newdfb = dfb['Balance'].resample('D').ohlc().dropna() # dropna gets rid of rows that have no data
38newdfb.drop(['open','high','low'],axis=1,inplace=True) # keep only "close"
39newdfb.columns = ['Balance'] # rename "close" to "Balance"
40print(newdfb.head())
41
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29dfb["Balance"] = dfb["Balance"].str.split(expand=True).iloc[:,0] # [:,0] to take only balance and throw away "DOGE"
30dfb["Balance"] = dfb["Balance"].str.replace(',','').astype(float) # remove commas from balance and convert to float.
31print(dfb.head(16))
32print(dfb.tail())
33df = yf.Ticker("DOGE-USD").history(period='max')
34df = df.loc["2021-01-01":]
35print(df.head(8))
36print(df.tail())
37newdfb = dfb['Balance'].resample('D').ohlc().dropna() # dropna gets rid of rows that have no data
38newdfb.drop(['open','high','low'],axis=1,inplace=True) # keep only "close"
39newdfb.columns = ['Balance'] # rename "close" to "Balance"
40print(newdfb.head())
41
Now, before we can join the two dataframes, notice that the yfinance dataframe has only dates in the index, whereas the balance data has entire timestamps. We can convert the balance data to have only dates in the index as follows:
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29dfb["Balance"] = dfb["Balance"].str.split(expand=True).iloc[:,0] # [:,0] to take only balance and throw away "DOGE"
30dfb["Balance"] = dfb["Balance"].str.replace(',','').astype(float) # remove commas from balance and convert to float.
31print(dfb.head(16))
32print(dfb.tail())
33df = yf.Ticker("DOGE-USD").history(period='max')
34df = df.loc["2021-01-01":]
35print(df.head(8))
36print(df.tail())
37newdfb = dfb['Balance'].resample('D').ohlc().dropna() # dropna gets rid of rows that have no data
38newdfb.drop(['open','high','low'],axis=1,inplace=True) # keep only "close"
39newdfb.columns = ['Balance'] # rename "close" to "Balance"
40print(newdfb.head())
41dates = [d.date() for d in newdfb.index]
42newdfb.index = pd.DatetimeIndex(dates)
43newdfb.index.name = 'Time'
44print(newdfb.head())
45
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29dfb["Balance"] = dfb["Balance"].str.split(expand=True).iloc[:,0] # [:,0] to take only balance and throw away "DOGE"
30dfb["Balance"] = dfb["Balance"].str.replace(',','').astype(float) # remove commas from balance and convert to float.
31print(dfb.head(16))
32print(dfb.tail())
33df = yf.Ticker("DOGE-USD").history(period='max')
34df = df.loc["2021-01-01":]
35print(df.head(8))
36print(df.tail())
37newdfb = dfb['Balance'].resample('D').ohlc().dropna() # dropna gets rid of rows that have no data
38newdfb.drop(['open','high','low'],axis=1,inplace=True) # keep only "close"
39newdfb.columns = ['Balance'] # rename "close" to "Balance"
40print(newdfb.head())
41dates = [d.date() for d in newdfb.index]
42newdfb.index = pd.DatetimeIndex(dates)
43newdfb.index.name = 'Time'
44print(newdfb.head())
45
Now we can join the two dataframes. DataFrame.join() will join the dataframes based on the index, which in our case are dates, therefore the data will become aligned by date. Also, we will do an outer join, and .dropna() so that only dates that exist in both dataframes will be included in the final dataframe. This the cleanest way to be able to plot the data together on the same plot:
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29dfb["Balance"] = dfb["Balance"].str.split(expand=True).iloc[:,0] # [:,0] to take only balance and throw away "DOGE"
30dfb["Balance"] = dfb["Balance"].str.replace(',','').astype(float) # remove commas from balance and convert to float.
31print(dfb.head(16))
32print(dfb.tail())
33df = yf.Ticker("DOGE-USD").history(period='max')
34df = df.loc["2021-01-01":]
35print(df.head(8))
36print(df.tail())
37newdfb = dfb['Balance'].resample('D').ohlc().dropna() # dropna gets rid of rows that have no data
38newdfb.drop(['open','high','low'],axis=1,inplace=True) # keep only "close"
39newdfb.columns = ['Balance'] # rename "close" to "Balance"
40print(newdfb.head())
41dates = [d.date() for d in newdfb.index]
42newdfb.index = pd.DatetimeIndex(dates)
43newdfb.index.name = 'Time'
44print(newdfb.head())
45dfc = df.join(newdfb, how='outer').dropna()
46dfc.index.name = 'Date'
47print(dfc.head())
48print(dfc.tail())
49
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29dfb["Balance"] = dfb["Balance"].str.split(expand=True).iloc[:,0] # [:,0] to take only balance and throw away "DOGE"
30dfb["Balance"] = dfb["Balance"].str.replace(',','').astype(float) # remove commas from balance and convert to float.
31print(dfb.head(16))
32print(dfb.tail())
33df = yf.Ticker("DOGE-USD").history(period='max')
34df = df.loc["2021-01-01":]
35print(df.head(8))
36print(df.tail())
37newdfb = dfb['Balance'].resample('D').ohlc().dropna() # dropna gets rid of rows that have no data
38newdfb.drop(['open','high','low'],axis=1,inplace=True) # keep only "close"
39newdfb.columns = ['Balance'] # rename "close" to "Balance"
40print(newdfb.head())
41dates = [d.date() for d in newdfb.index]
42newdfb.index = pd.DatetimeIndex(dates)
43newdfb.index.name = 'Time'
44print(newdfb.head())
45dfc = df.join(newdfb, how='outer').dropna()
46dfc.index.name = 'Date'
47print(dfc.head())
48print(dfc.tail())
49
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29dfb["Balance"] = dfb["Balance"].str.split(expand=True).iloc[:,0] # [:,0] to take only balance and throw away "DOGE"
30dfb["Balance"] = dfb["Balance"].str.replace(',','').astype(float) # remove commas from balance and convert to float.
31print(dfb.head(16))
32print(dfb.tail())
33df = yf.Ticker("DOGE-USD").history(period='max')
34df = df.loc["2021-01-01":]
35print(df.head(8))
36print(df.tail())
37newdfb = dfb['Balance'].resample('D').ohlc().dropna() # dropna gets rid of rows that have no data
38newdfb.drop(['open','high','low'],axis=1,inplace=True) # keep only "close"
39newdfb.columns = ['Balance'] # rename "close" to "Balance"
40print(newdfb.head())
41dates = [d.date() for d in newdfb.index]
42newdfb.index = pd.DatetimeIndex(dates)
43newdfb.index.name = 'Time'
44print(newdfb.head())
45dfc = df.join(newdfb, how='outer').dropna()
46dfc.index.name = 'Date'
47print(dfc.head())
48print(dfc.tail())
49
Now finally, we can plot the 'Balance' together with the ohlc candlesticks:
1import yfinance as yf
2import matplotlib
3from matplotlib import pyplot as plt
4import numpy as np
5import pandas as pd
6import mplfinance as mpf
78#This charts the Dogecoin Price
910df = yf.Ticker("DOGE-USD").history(period='max')
1112df = df.loc["2021-01-01":]
1314mpf.plot(df, type="candle")
151617#This charts the balance from CSV
1819parse_dates = ['Time']
20df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
2122plt.plot(df["Time"], df["Balance"])
23plt.gca().invert_yaxis()
24plt.show()
25df = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv', index_col=0, parse_dates=parse_dates)
26dfb = pd.read_csv('DSb5CvAXhXnzFoxmiMaWpgxjDF6CfMK7h2.csv',usecols=['Time','Balance'],index_col=0, parse_dates=True)
27dfb = dfb.iloc[::-1] # reverse the data
28print(dfb.head(8))
29dfb["Balance"] = dfb["Balance"].str.split(expand=True).iloc[:,0] # [:,0] to take only balance and throw away "DOGE"
30dfb["Balance"] = dfb["Balance"].str.replace(',','').astype(float) # remove commas from balance and convert to float.
31print(dfb.head(16))
32print(dfb.tail())
33df = yf.Ticker("DOGE-USD").history(period='max')
34df = df.loc["2021-01-01":]
35print(df.head(8))
36print(df.tail())
37newdfb = dfb['Balance'].resample('D').ohlc().dropna() # dropna gets rid of rows that have no data
38newdfb.drop(['open','high','low'],axis=1,inplace=True) # keep only "close"
39newdfb.columns = ['Balance'] # rename "close" to "Balance"
40print(newdfb.head())
41dates = [d.date() for d in newdfb.index]
42newdfb.index = pd.DatetimeIndex(dates)
43newdfb.index.name = 'Time'
44print(newdfb.head())
45dfc = df.join(newdfb, how='outer').dropna()
46dfc.index.name = 'Date'
47print(dfc.head())
48print(dfc.tail())
49ap = mpf.make_addplot(dfc['Balance'])
50mpf.plot(dfc,type='candle',addplot=ap)
51
I'm a total beginner with Python/Django and trying to understand why this isn't working. I have a function that contains a for loop, doing some logic and then updating a model. but when I have more than 1 item in the loop I get a UNIQUE constraint failed: app_token.token_name error.
So I think I'm misunderstanding how the loop is working?
I'm expecting the this Token.objects.update(token_capture_date = formatedDate, token_price = price, token_name=item.token_name ) to update the model based on the item loop?
I'm using the update on the objects and have tried removing the token_name, and tried using token.token_name
If I remove token_name= it updates both items in the database with the same values? which makes me think its this line r = Token.objects.update(token_capture_date = formatedDate, token_price = price, token_name=item.token_name ) do i need to apply some kinda of filter?
Thanks
ANSWER
Answered 2022-Jan-22 at 10:31
I believe that by calling Token.objects.update() you actually end up trying to update all Token objects. Since token_name has to be unique, and you are giving it the same name as another Token object it throws that error.
Since you are already in a for loop, you can simply update the token that is currently being processed.
This will make it so that the current token object which is being processed in the for loop has its respective field values updated and saved in the database.
Also, this line is unnecessary: token = Token.objects.get(pk=item.id) as we already have access to the token through the looping variable item.
Why are certain time values not plotting on this graph?
Asked 2022-Jan-10 at 20:11
Creating a scatter plot with vega points are only plotted up to the 11/12th of every month? I'm confused as I feel like everything should be working fine, however this clearly doesnt seem to be the case. Is this due to the data? or the choice of graph maybe? Feel as though there should be a solution but going through the documentation i can't find one
Your source data appears to store dates using DD/MM/YYYY format, which Javascript date parsing assumes is MM/DD/YYYY. Because of this, when the day is larger than 12, the result is an invalid date.
You can fix this by declaring the correct date format in your specification:
Flutter - How to refresh a Widget on button Click?
Find element by Xpath. How to split the element I don't want inside the Xpath
Can't use Selenium to find elements. Try to avoid deprecated commands warning, got result only one element. (on Colab)
How to programmatically validate Tether (TRC20) Wallet Addresses in PHP?
Custom Loss Function - Keras
Running python file on windows CMD vs VScode
How to turn HTML widget code into NEXTjs code to use in an app (CoinMarketCap Price Marquee Ticker)
How can I properly graph these two datasets using Pandas in Matplotlib?
For each item update database
Why are certain time values not plotting on this graph?
QUESTION
Flutter - How to refresh a Widget on button Click?
Asked 2022-Mar-01 at 10:15
I'm making an app where I plot charts of cryptocurrency. I have a file like this, that renders a chart of btc/24h in it. I'm using chart_sparkline package to plot the chart.
I have the following code, and it is not working on a button click, how do I fix this?
Basically, the elevatedButton should update the chart, but it is not working in my case, how to fix this?
ANSWER
Answered 2022-Mar-01 at 10:15
I fixed this, I was passing constructor parameters of Portfolio to _PortfolioState, this won't update the values the next time. Instead of passing these values to _PortfolioState, I used widget.coin and widget.days to extract the data from the Portfolio class.
Find element by Xpath. How to split the element I don't want inside the Xpath
Asked 2022-Feb-15 at 09:12
I try to scrape a website using Selenium.
I got a problem when I try to get the coins name. because there're 2 elements inside 'td'
How can I get rid of another element I don't want. or keep track to only its first element.
(I found this post but I'm not sure if it answer my issue or not)
This is my whole code
1#driver chrome def
2website = 'https://www.bitkub.com/fee/cryptocurrency'
3path = r"C:\\Users\\USER\\Downloads\\chromedriver.exe"
4options = Options()
5options.add_argument("start-maximized")
6driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
7driver.get(website)
89#giving variable
10coin_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[2]//span")))]
11chain_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[3]//div")))]
12withdrawal_fees = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[4]//div")))]
13#print(coin_name)
14#print(chain_name)
15#print(withdrawal_fees)
161718#for loop make list
1920for coin, chains, wdf in zip(coin_name, chain_name, withdrawal_fees):
21 print("Coin name: {} Chain: {} Fee: {}".format(coin, chains, wdf))
22
The input of coin_name (which I mentioned that it got 2 elements)
1#driver chrome def
2website = 'https://www.bitkub.com/fee/cryptocurrency'
3path = r"C:\\Users\\USER\\Downloads\\chromedriver.exe"
4options = Options()
5options.add_argument("start-maximized")
6driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
7driver.get(website)
89#giving variable
10coin_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[2]//span")))]
11chain_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[3]//div")))]
12withdrawal_fees = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[4]//div")))]
13#print(coin_name)
14#print(chain_name)
15#print(withdrawal_fees)
161718#for loop make list
1920for coin, chains, wdf in zip(coin_name, chain_name, withdrawal_fees):
21 print("Coin name: {} Chain: {} Fee: {}".format(coin, chains, wdf))
22['Civic(CVC)', '(CVC)', 'Bitcoin SV(BSV)', '(BSV)', 'Ethereum(ETH)', '(ETH)', 'Bitkub Coin(KUB)', '(KUB)', 'Compound(COMP)', '(COMP)', 'Curve DAO Token(CRV)', '(CRV)', .... ]
23
This is how element on the website look like
I wanted input to look like this so I can make dataframe out of it
1#driver chrome def
2website = 'https://www.bitkub.com/fee/cryptocurrency'
3path = r"C:\\Users\\USER\\Downloads\\chromedriver.exe"
4options = Options()
5options.add_argument("start-maximized")
6driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
7driver.get(website)
89#giving variable
10coin_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[2]//span")))]
11chain_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[3]//div")))]
12withdrawal_fees = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[4]//div")))]
13#print(coin_name)
14#print(chain_name)
15#print(withdrawal_fees)
161718#for loop make list
1920for coin, chains, wdf in zip(coin_name, chain_name, withdrawal_fees):
21 print("Coin name: {} Chain: {} Fee: {}".format(coin, chains, wdf))
22['Civic(CVC)', '(CVC)', 'Bitcoin SV(BSV)', '(BSV)', 'Ethereum(ETH)', '(ETH)', 'Bitkub Coin(KUB)', '(KUB)', 'Compound(COMP)', '(COMP)', 'Curve DAO Token(CRV)', '(CRV)', .... ]
23['Civic(CVC)', 'Bitcoin SV(BSV)', 'Ethereum(ETH)', 'Bitkub Coin(KUB)', 'Compound(COMP)', 'Curve DAO Token(CRV)', .... ]
24
ANSWER
Answered 2022-Feb-15 at 09:12
Hope this helps:
1#driver chrome def
2website = 'https://www.bitkub.com/fee/cryptocurrency'
3path = r"C:\\Users\\USER\\Downloads\\chromedriver.exe"
4options = Options()
5options.add_argument("start-maximized")
6driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
7driver.get(website)
89#giving variable
10coin_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[2]//span")))]
11chain_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[3]//div")))]
12withdrawal_fees = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//tbody//tr//td[4]//div")))]
13#print(coin_name)
14#print(chain_name)
15#print(withdrawal_fees)
161718#for loop make list
1920for coin, chains, wdf in zip(coin_name, chain_name, withdrawal_fees):
21 print("Coin name: {} Chain: {} Fee: {}".format(coin, chains, wdf))
22['Civic(CVC)', '(CVC)', 'Bitcoin SV(BSV)', '(BSV)', 'Ethereum(ETH)', '(ETH)', 'Bitkub Coin(KUB)', '(KUB)', 'Compound(COMP)', '(COMP)', 'Curve DAO Token(CRV)', '(CRV)', .... ]
23['Civic(CVC)', 'Bitcoin SV(BSV)', 'Ethereum(ETH)', 'Bitkub Coin(KUB)', 'Compound(COMP)', 'Curve DAO Token(CRV)', .... ]
24for name in coin_name:
25 if name[0] == "(":
26 coin_name.remove(name)
27
Can't use Selenium to find elements. Try to avoid deprecated commands warning, got result only one element. (on Colab)
Asked 2022-Feb-09 at 22:06
As my title said. To avoid deprecated warning of Selenium.I try to use the new format as I found.
and somehow result return only 1 element. How can I fix it to return elements?
FYI I scraping website. I'm working on Colab so I can share with my workmates easier.
