scholar | Fetchs google scholar queries and outputs

I am creating part of one tool, which merges two SAP exports into one.

I know how many account numbers might be in the export (not all of them every month) and I have it almost complete except for the find issue, when account number is not in dataset, but is found and data merged are from the last exported account number

...

I am using the following code to scrape some information of different pages of Google Scholar using Selenium and Beautiful Soup.

I can print all the scraped information but I can't save the results into one Dataframe for export.

How do I save the results (Title, Author, Link, Abstract) for each result of the search?

...

How are U? I'm dealing with this problem: I have two columns with equal width, but in the right columns under the name, appear an extra height so the text are not equal aligned between both columns. I left here the link of the page that I'm working and the code is someone can help me...

---LINK: http://c2260485.ferozo.com/the-team/

...

I've come to learn that there are linear time sorting algorithms that don't run by comparisons like radix sort. My hope is to have a sorting algorithm that runs in linear time but can also run in constant time by running n threads for n elements. From the research I've done, this seems possible on a PRAM CRCW machine but I've found conflicting information as to whether the algorithm that runs on a PRAM CRCW machine can be run on a standard consumer computer in the same constant time.

FYI, the algorithm in question is here. This is pretty interesting as well.

Is it possible?

...

I knew there is tons of post regarding this error. I think what I got is pretty strange.

Ok here it is.

models.py

...

I am trying to scrape some data from Google Scholar with scrapy, my code is the following:

...

I'm reviewing and experimenting with outlier flagging strategies, and keep running into references to Sn and Qn from Rousseeuw and Croux in Alternatives to the Median Absolute Deviation.

http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf

They sound quite excellent, and seem to be widely used in academic and applied stats across disciplines. I checked Google Scholar, and that paper has over 2,100 citations.

The appealing feature of this technique is that it isn't heavily impacted by asymmetric distributions. Which is what we've got, most of the time. Sometimes quite extremely.

This is of course available in R, but I'm not a stats person, we don't have server-side access to R (or Python), and would like to do some searches directly in Postgres. I haven't been able to find anything in any SQL idiom, and am hoping that some stats lover out there has some Postgres code up their sleeve.

...

I'm making some wordclouds for a project on kaggle, but this line of code isn't working. I am trying to remove all the apostrophes from a column containing text. In my corups "'s" and "'re" are two fo my most frequent "words". While the data is still in the form of a data frame I have been using this line of code df$col <- gsub("\'","", df$col).

Below is some sample data. In my kaggle project, the text data comes in a column of a dataframe. Am I missing something? I've also tried str_replace_all and sub.

EDIT: dput(head(df))

...

Question: How can I improve either my method ("expand_traits" posted below) or the data structure I am trying to use? I estimate the runtime of my solution to be a few hours, which seems like I went very wrong somewhere (considering it takes ~ 10 minutes to collect all of the data, and possibly a few hours to transform it into something I can analyze).

I have collected some data that is essentially a Pandas DataFrame, where some columns in the table are a list of lists (technically formatted as strings, so when I evaluate them I am using ast.literal_eval(column) - if that's relevant).

To explain the context a bit:

The data contains historical stats from League of Legends TFT game mode. I am aiming to perform some analysis on it in terms of being able to group by each item in the list, and see how they perform on average. I can only really think of doing this in terms of tables - something like df.groupby(by='Trait').mean() to get the average win-rate for each trait, but am open to other ideas.

Here is an example of the dataset:

Rank Summoner Traits Units 1 name1 ['7 Innovator', '1 Transformer', '3 Enchanter', '2 Socialite', '2 Clockwork', '2 Scholar', '2 Scrap'] ['Ezreal', 'Singed', 'Zilean', 'Taric', 'Heimerdinger', 'Janna', 'Orianna', 'Seraphine', 'Jayce'] 2 name2 ['1 Cuddly', '1 Glutton', '5 Mercenary', '4 Bruiser', '6 Chemtech', '2 Scholar', '1 Socialite', '2 Twinshot'] ['Illaoi', 'Gangplank', 'MissFortune', 'Lissandra', 'Zac', 'Urgot', 'DrMundo', 'TahmKench', 'Yuumi', 'Viktor']

The total records in the table is approximately 40,000 (doesn't sound like much) but my original idea was to basically "unpivot" the nested lists into their own record.

My idea looks a little something like:

Summoner Trait Record_ID name1 7 Innovator id_1 name1 1 Transformer id_1 ... ... ... name2 1 Cuddly id_2 name2 1 Glutton id_2

Due to the number of items in each list, this transformation will turn my ~40,000 records into a few hundred thousand.

Another thing to note is that because this transformation would be unique to each column that contains lists, I would need to perform it separately (as far as I know) on each column. Here is the current code I am using to do this to the "Traits" column, which takes my computer around 35 mins to complete (also pretty average PC - nothing crazy but equivalent to intel i5 & 16 gigs of RAM.

...