surging | service engine that provides a lightweight
kandi X-RAY | surging Summary
Support
Quality
Security
License
Reuse
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample Here
surging Key Features
surging Examples and Code Snippets
Trending Discussions on surging
Trending Discussions on surging
QUESTION
I have encountered some issue when trying to figure out and calculate the firestore read count. The firestore read count is always surging up at a very high rate (100 counts increment every time I reload the page) even though there are only around 15 users documents. Even when I am not reloading the page, the firestore read count would go up itself, is this due to the subscribe behaviour that cause the read data action refreshing from time to time? (I have read some articles recommend to use "once" if user want to extract data just once).
Bellow is the code snippet (ts):
// All buddy users from Firebase
private usersCollection: AngularFirestoreCollection;
users: Observable;
usersFirebase: Profile[] = [];
getUserDataFromFirebase() {
this.isImageLoading = false;
this.users.subscribe(async results => {
var ref;
for(let result of results) {
if(result.imageName) {
ref = this.store.ref('images/' + result.userId + '/profiles/' + result.imageName);
} else {
// Get default image is image not existing
ref = this.store.ref('images/ironman.jpg');
}
await ref.getDownloadURL().toPromise().then(urlString => {
result.profileURL = urlString;
// Change availibility date from timestamp to date format
try {
result.availability = this.datePipe.transform(result.availability.toDate(), 'yyyy-MM-dd');
} catch (error) {}
result.flip = 'inactive';
if(result.identity == 'Tenant')
{
this.usersFirebase.push(result);
}
return new Promise((resolve, reject) => {
resolve();
})
});
}
console.log(this.usersFirebase);
});
}
How does the firestore read count works, is the count increment based on document queries, will it continue to query itself after a certain amount of time? Firestore read count increases more than users documents
ANSWER
Answered 2020-Oct-14 at 13:05The reads counts are focused on the number of documents retrieved.
Lets set these 4 scenarios:
- A collection of 10 users, you run a
collection("users").get()
call: you will get back 10 employee documents, and be charged for 10 reads. - A collection of 10,000 users, you run a
collection("users").get()
call: you will get back 10,000 users, and be charged for 10,000 reads. - A collection of 10,000 employees, you run a
collection("users").get().limit(10)
call: you will get back 10 users, and be charged for 10 reads. - A collection of 10,000 users, 15 of which are named "Carl" and you run a
collection("users").where("first_name", "==", "Carl")
call, you will get back 4 users and be charged for 15 reads.
On the other hand, if you are listening to the whole collection users
(no where() or .orderBy() clauses) and you have an active onSnapshot() listener then you will be charged for a document read operation each time a new document is added, changed, or deleted in the collection users
.
You might want to take a look into your workflow to check whether other process are making changes on your collection when checking the read operations.
Finally, something to keep in mind is that the read ops in your report might not match with the billing and quota usage. There is a feature requests in the Public Issue Tracker realted to this inquiry - reads on Firestore: here. You can "star" it and set the notifications to get updates on them. Also, you can create a new one if necessary.
QUESTION
I am interested in how the COVID pandemic is affecting meat processing plants across the country. I retrieved NYT COVID data by county level and statistical data from the food agency. Here I am exploring how COVID cases are surging in counties where major food processing plants are located because more sick employees in plants might bring negative impacts to the business. In my first attempt, I figured out moving average time series plots where COVID new cases vs 7 days rolling mean along the date.
But, I think it would be more efficient I could replace the graph which represents num-emp
and new-cases
by counties in the for loop. To achieve this, I think it would be better to cluster them by company level and expand them into multiple graphs to prevent the lines from overlapping and becoming to difficult to see. I am not sure how to achieve this from my current attempt. Can anyone suggest a possible ways of doing this in matplotlib? Any idea?
my current attempt:
Here is the reproducible data in this gist that I used in my experiment:
import pandas as pd import matplotlib.pyplot as plt import matplotlib.dates as mdates import seaborn as sns from datetime import timedelta, datetime
df = pd.read_csv("https://gist.githubusercontent.com/jerry-shad/7eb2dd4ac75034fcb50ff5549f2e5e21/raw/477c07446a8715f043c9b1ba703a03b2f913bdbf/covid_tsdf.csv")
df.drop(['Unnamed: 0', 'fips', 'non-fed-slaughter', 'fed-slaughter', 'total-slaughter', 'mcd-asl'], axis=1, inplace=True)
for ct in df['county_state'].unique():
dd = df[df['county_state'] == ct].groupby(['county_state', 'date', 'est'])[['cases','new_cases']].sum().unstack().reset_index()
dd.columns= ['county_state','date', 'cases', 'new_cases']
dd['date'] = pd.to_datetime(dd['date'])
dd['rol7'] = dd[['date','new_cases']].rolling(7).mean()
fig = plt.figure(figsize=(8,6),dpi=144)
ax = fig.add_subplot(111)
colors = sns.color_palette()
ax2 = ax.twinx()
ax = sns.lineplot('date', 'rol7', data=dd, color=colors[1], ax=ax)
ax2 = sns.lineplot('date', 'cases', data=dd, color=colors[0], ax=ax2)
ax.set_xlim(dd.date.min(), dd.date.max())
fig.legend(['rolling7','cases'],loc="upper left", bbox_to_anchor=(0.01, 0.95), bbox_transform=ax.transAxes)
ax.grid(axis='both', lw=0.5)
locator = mdates.AutoDateLocator()
ax.xaxis.set_major_locator(locator)
fig.autofmt_xdate(rotation=45)
ax.set(title=f'US covid tracking in meat processing plants by county - Linear scale')
plt.show()
here is my current output:
but this output is not quite significant to understand how food processing company' is affected by COVID because of infected employees. To make this more visually accessible to understand, I think we can replace the two graphs with num-emp
and newly infected case new_cases
and draw the counties we need in the loop process. At that point, it would be better to cluster them by company characteristics, etc. and expand them into multiple graphs to prevent the lines from overlapping and becoming difficult to see. I want to make EDA that provides this sort of information visually. Can anyone suggest possible ways of doing this with matplotlib
? Any thoughts? Thanks!
ANSWER
Answered 2020-Sep-14 at 23:32- There were a couple of issues, I've made inline notations
- The main issue was in the
.groupby
- The data is already selected by
'country_state'
so there's no need togroupby
it - Only
reset_index(level=1)
, keep date in the index for rolling .unstack()
was creating multi-level column names.
- The data is already selected by
- Set
ci=None
for plotting. - It doesn't make sense to use
'num-emp'
as a metrics. It's constant across time.- If you want to see the plot, swap
'cases'
in the loop, for'num-emp'
.
- If you want to see the plot, swap
- I think the best way to see the impact of COVID on a given company, is to find a dataset with revenue.
- Because food processing plants are considered critical infrastructure, there probably won't be much change in their head count, and anyone who is sick, is probably on sick leave vs. termination.
import pandas as pd
import matplotlib.pyplot as plt
url = 'https://gist.githubusercontent.com/jerry-shad/7eb2dd4ac75034fcb50ff5549f2e5e21/raw/477c07446a8715f043c9b1ba703a03b2f913bdbf/covid_tsdf.csv'
# load the data and parse the dates
df = pd.read_csv(url, parse_dates=['date'])
# drop unneeded columns
df.drop(['Unnamed: 0', 'fips', 'non-fed-slaughter', 'fed-slaughter', 'total-slaughter', 'mcd-asl'], axis=1, inplace=True)
for ct in df['county_state'].unique():
# groupby has been updated: no need for county becasue they're all the same, given the loop; keep date in the index for rolling
dd = df[df['county_state'] == ct].groupby(['date', 'est', 'packer'])[['cases','new_cases']].sum().reset_index(level=[1, 2])
dd['rol7'] = dd[['new_cases']].rolling(7).mean()
colors = sns.color_palette()
fig, ax = plt.subplots(figsize=(8, 6), dpi=144)
ax2 = ax.twinx()
sns.lineplot(dd.index, 'rol7', ci=None, data=dd, color=colors[1], ax=ax) # date is in the index
sns.lineplot(dd.index, 'cases', ci=None, data=dd, color=colors[0], ax=ax2) # date is in the index
ax.set_xlim(dd.index.min(), dd.index.max()) # date is in the index
fig.legend(['rolling7','cases'], loc="upper left", bbox_to_anchor=(0.01, 0.95), bbox_transform=ax.transAxes)
# set y labels
ax.set_ylabel('7-day Rolling Mean')
ax2.set_ylabel('Current Number of Cases')
ax.grid(axis='both', lw=0.5)
locator = mdates.AutoDateLocator()
ax.xaxis.set_major_locator(locator)
fig.autofmt_xdate(rotation=45)
# create a dict for packer and est
vals = dict(dd[['packer', 'est']].reset_index(drop=True).drop_duplicates().values.tolist())
# create a custom string from vals, for the title
insert = ', '.join([f'{k}: {v}' for k, v in vals.items()])
# ax.set(title=f'US covid tracking in meat processing plants for {ct} \nPacker: {", ".join(dd.packer.unique())}\nEstablishments: {", ".join(dd.est.unique())}')
# alternate title based on comment request
ax.set(title=f'US covid tracking in meat processing plants for {ct} \n{insert}')
plt.savefig(f'images/{ct}.png') # save files by ct name to images directory
plt.show()
plt.close()
QUESTION
I want to understand how covid pandemic is affecting the supply chain industry such as meat processing plants. I retrieved NYT covid data by county level and statistical data from food agency, so I want to understand how covid cases are surging in counties where major food processing plants are located. To do so, I figured out the right data and able to make it ready for rendering a nice time series chart. However, I found issues of getting the right plotting data for that because the resulted plot is not getting the expected output. Here is what I tried so far:
my attempt:
Here is the final aggregated covid time series data that I am interested in this gist. Here is my current attempt:
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import seaborn as sns
from datetime import timedelta, datetime
df = pd.read_csv("https://gist.githubusercontent.com/jerry-shad/7eb2dd4ac75034fcb50ff5549f2e5e21/raw/477c07446a8715f043c9b1ba703a03b2f913bdbf/covid_tsdf.csv")
df.drop(['Unnamed: 0', 'fips', 'non-fed-slaughter', 'fed-slaughter', 'total-slaughter', 'mcd-asl'], axis=1, inplace=True)
for ct in df['county_state'].unique():
dd = df.groupby([ct, 'date', 'est'])['num-emp'].sum().unstack().reset_index()
p = sns.lineplot('date', 'values', data=dd, hue='packer', markers=markers, style='cats', ax=axes[j, 0])
p.set_xlim(data.date.min() - timedelta(days=60), data.date.max() + timedelta(days=60))
plt.legend(bbox_to_anchor=(1.04, 0.5), loc="center left", borderaxespad=0)
but looks I made the wrong aggregation above, this attempt is not working. My intention is basically if a company has multiple establishments (a.k.a est
), then I need to take sum of its num-emp
: # of employees, then get the ratio of # of new_deaths / num-emp
along the time. Basically I want to track whether company's staff are affected by covid or not with some approximate sense. I am not quite sure what would be the correct way of doing this with matplotlib
in python. Can anyone suggest possible of correction to make this right? Any idea?
second attempt
I got some inspiration from recent covid19 related post, so this is another way of trying to do what I want to make in matplotlib
. I aggregated data in this way with custom plotting helper function also:
df = pd.read_csv("https://gist.githubusercontent.com/jerry-shad/7eb2dd4ac75034fcb50ff5549f2e5e21/raw/477c07446a8715f043c9b1ba703a03b2f913bdbf/covid_tsdf.csv")
ds_states = df.groupby('county_state').sum().rename({'county_state': 'location'})
ds_states['mortality'] = ds_states['deaths'] / ds_states['popestimate2019'] * 1_000_000
ds_states['daily_mortality'] = ds_states['new_deaths'] / ds_states['popestimate2019'] * 1_000_000
ds_states['daily_mortality7'] = ds_states['daily_mortality'].rolling({'time': 7}).mean()
then this is plotting helper function that I came up:
def subplots(*args, tick_right=True, **kwargs):
f, ax = plt.subplots(*args, **kwargs)
if tick_right:
ax.yaxis.tick_right()
ax.yaxis.set_label_position("right")
ax.yaxis.grid(color="lightgrey", linewidth=0.5)
ax.xaxis.grid(color="lightgrey", linewidth=0.5)
ax.xaxis.set_tick_params(labelsize=14)
return f, ax
_, ax1 = subplots(subplot_kw={'xlim': XLIM})
ax1.set(title=f'US covid tracking in meat processing plants by county - Linear scale')
ax2 = ax1.twinx()
but I trapped again here how to make this right. My essential goal is basically whether how much meat processing companies are affected by covid because if its worker got infected by covid, companies' performance will be dropped. I want to make eda that provides this sort of information visually. Can anyone suggest possible ways of doing this with matplotlib
? I am open to any feasible eda attempt that makes this question more realistic or meaningful.
desired output
I am thinking about to make eda output something like below:
what I want to see, by county level, how every company's performance is varied because of covid. Can anyone point me out anyway to achieve possible eda output? Thanks
update
since what kind od eda that I want to make is not quite solid in my mind, so I am open to hearing any possible eda that fit the context of the problem that I raised above. Thanks in advance!
ANSWER
Answered 2020-Sep-10 at 09:25We have graphed the moving average of the number of outbreaks and new outbreaks for one state only. The process involved adding the moving average columns to the data frame extracted for a particular state and drawing a two-axis graph.
ct = 'Maricopa_Arizona'
dd = df[df['county_state'] == ct].groupby(['county_state', 'date', 'est'])[['cases','new_cases']].sum().unstack().reset_index()
dd.columns= ['county_state','date', 'cases', 'new_cases']
dd['date'] = pd.to_datetime(dd['date'])
dd['rol7'] = dd[['date','new_cases']].rolling(7).mean()
dd.tail()
county_state date cases new_cases exp7 rol7
216 Maricopa_Arizona 2020-08-29 133389.0 403.0 306.746942 243.428571
217 Maricopa_Arizona 2020-08-30 133641.0 252.0 293.060207 264.857143
218 Maricopa_Arizona 2020-08-31 133728.0 87.0 241.545155 252.285714
219 Maricopa_Arizona 2020-09-01 134004.0 276.0 250.158866 244.857143
220 Maricopa_Arizona 2020-09-02 134346.0 342.0 273.119150 273.142857
fig = plt.figure(figsize=(8,6),dpi=144)
ax = fig.add_subplot(111)
colors = sns.color_palette()
ax2 = ax.twinx()
ax = sns.lineplot('date', 'rol7', data=dd, color=colors[1], ax=ax)
ax2 = sns.lineplot('date', 'cases', data=dd, color=colors[0], ax=ax2)
ax.set_xlim(dd.date.min(), dd.date.max())
fig.legend(['rolling7','cases'],loc="upper left", bbox_to_anchor=(0.01, 0.95), bbox_transform=ax.transAxes)
ax.grid(axis='both', lw=0.5)
locator = mdates.AutoDateLocator()
ax.xaxis.set_major_locator(locator)
fig.autofmt_xdate(rotation=45)
ax.set(title=f'US covid tracking in meat processing plants by county - Linear scale')
plt.show()
QUESTION
I retrieved a dataset from a news API in JSON format. I want to extract the news description from the JSON data.
This is my code:-
import requests
import json
url = ('http://newsapi.org/v2/top-headlines?'
'country=us&'
'apiKey=608bf565c67f4d99994c08d74db82f54')
response = requests.get(url)
di=response.json()
di = json.dumps(di)
for di['articles'] in di:
print(article['title'])
The dataset looks like this:-
{'status': 'ok',
'totalResults': 38,
'articles': [
{'source':
{'id': 'the-washington-post',
'name': 'The Washington Post'},
'author': 'Derek Hawkins, Marisa Iati',
'title': 'Coronavirus updates: Texas, Florida and Arizona officials say early reopenings fueled an explosion of cases - The Washington Post',
'description': 'Local officials in states with surging coronavirus cases issued dire warnings Sunday about the spread of infections, saying the virus was rapidly outpacing containment efforts.',
'url': 'https://www.washingtonpost.com/nation/2020/07/05/coronavirus-update-us/',
'urlToImage': 'https://www.washingtonpost.com/wp-apps/imrs.php?src=https://arc-anglerfish-washpost-prod-washpost.s3.amazonaws.com/public/K3UMAKF6OMI6VF6BNTYRN77CNQ.jpg&w=1440',
'publishedAt': '2020-07-05T18:32:44Z',
'content': 'Here are some significant developments:\r\n
- The rolling seven-day average for daily new cases in the United States reached a record high for the 27th day in a row, climbing to 48,606 on Sunday, … [+5333 chars]'}])
Please guide me with this!
ANSWER
Answered 2020-Jul-05 at 20:08There are few corrections needed in your code.. below code should work and i have removed API KEY in answer make sure that you add one before testing
import requests
import json
url = ('http://newsapi.org/v2/top-headlines?'
'country=us&'
'apiKey=')
di=response.json()
#You don't need to dump json that is already in json format
#di = json.dumps(di)
#your loop is not correctly defined, below is correct way to do it
for article in di['articles']:
print(article['title'])
QUESTION
Currently, I am writing some node.js code to remove all HTML in a string. This one of the string that I need to process:
"Iconic powerful bass resonance of Bluedio: 57mm ultra-large dynamic drivers, turbine style housing, with the iconic Bluedio surging low-frequency shock, let you feel the bass resonate deep in the chest, enjoying the best sound quality. Clear and transparent bass, mids and treble, fully exposed to all the details of song, you can hear what the artists really want you to hear, Coldplay or Linkin Park concert played in your ear
"
This is the code I am using for removing HTML:
html=html.replace(/<\w+>/,'').replace(/<\/\w+>/,'').trim();
This is the output:
"Iconic powerful bass resonance of Bluedio: 57mm ultra-large dynamic drivers, turbine style housing, with the iconic Bluedio surging low-frequency shock, let you feel the bass resonate deep in the chest, enjoying the best sound quality. Clear and transparent bass, mids and treble, fully exposed to all the details of song, you can hear what the artists really want you to hear, Coldplay or Linkin Park concert played in your ear
"
As you see, the HTML are not completely removed from string yet. The
which is at the end of the string still remains.
Why does this happen? How can I fix this problem? I want to learn more about regular expression. Please do not give me a URL of a library. Thanks.
ANSWER
Answered 2018-Aug-21 at 11:58You need to add g
to perform global matching
html ="Iconic powerful bass resonance of Bluedio: 57mm ultra-large dynamic drivers, turbine style housing, with the iconic Bluedio surging low-frequency shock, let you feel the bass resonate deep in the chest, enjoying the best sound quality. Clear and transparent bass, mids and treble, fully exposed to all the details of song, you can hear what the artists really want you to hear, Coldplay or Linkin Park concert played in your ear
"
html=html.replace(/<\w+>/g,'').replace(/<\/\w+>/g,'').trim();
console.log(html)
QUESTION
i can't figured out why this doesn't works:
import nltk
from nltk.corpus import stopwords
import string
with open('moby.txt', 'r') as f:
moby_raw = f.read()
stop = set(stopwords.words('english'))
moby_tokens = nltk.word_tokenize(moby_raw)
text_no_stop_words_punct = [t for t in moby_tokens if t not in stop or t not in string.punctuation]
print(text_no_stop_words_punct)
looking at the output i have this:
[...';', 'surging', 'from', 'side', 'to', 'side', ';', 'spasmodically', 'dilating', 'and', 'contracting',...]
seems that the punctuation is still there. what i'm doing wrong?
ANSWER
Answered 2017-Aug-04 at 22:21In this line change try changing 'or' to 'and' that way your list will return only words that are both not a stop word and are not punctuation.
text_no_stop_words = [t for t in moby_tokens if t not in stop or t not in string.punctuation]
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install surging
Support
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesExplore Kits - Develop, implement, customize Projects, Custom Functions and Applications with kandi kits
Save this library and start creating your kit
Share this Page