scraperjs | A complete and versatile web scraper | Reactive Programming library

by ruipgil JavaScript Version: 1.2.0 License: MIT

X-Ray Key Features Code Snippets Community Discussions(3)Vulnerabilities Install Support

kandi X-RAY | scraperjs Summary

scraperjs is a JavaScript library typically used in Programming Style, Reactive Programming, Nodejs applications. scraperjs has no bugs, it has no vulnerabilities, it has a Permissive License and it has medium support. You can install using 'npm i scraperjs' or download it from GitHub, npm.

Scraperjs uses promises whenever possible. So, the scrapers should be used with the ScraperPromise. By creating a scraper. The following promises can be made over it, they all return a scraper promise,. All callback functions receive as their last parameter a utils object, with it the parameters of an url from a router can be accessed. Also the chain can be stopped. The promise chain is fired with the same sequence it was declared, with the exception of the promises get and request that fire the chain when they've received a valid response, and the promises done and catch, which were explained above. You can also waterfall values between promises by returning them (with the exception of the promise timeout, that will always return undefined) and it can be access through utils.lastReturn.

Support

Quality

Security

License

Reuse

Support

scraperjs has a medium active ecosystem.

It has 3675 star(s) with 202 fork(s). There are 94 watchers for this library.

It had no major release in the last 12 months.

There are 24 open issues and 43 have been closed. On average issues are closed in 135 days. There are 3 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of scraperjs is 1.2.0

Quality

scraperjs has 0 bugs and 0 code smells.

Security

scraperjs has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

scraperjs code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

scraperjs is licensed under the MIT License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

scraperjs releases are not available. You will need to build from source code and install.

Deployable package is available in npm.

Installation instructions, examples and code snippets are available.

scraperjs saves you 15 person hours of effort in developing the same functionality from scratch.

It has 43 lines of code, 0 functions and 25 files.

It has low code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of scraperjs

Get all kandi verified functions for this library.

scraperjs Key Features

No Key Features are available at this moment for scraperjs.

scraperjs Examples and Code Snippets

No Code Snippets are available at this moment for scraperjs.

Community Discussions

Trending Discussions on scraperjs

Asynchronous function got jammed when used inside while loop (scraperjs)

Getting 'undefined' when passing an array through Express res.render function

Trying to run a script with Express on localhost:3000

QUESTION

Asynchronous function got jammed when used inside while loop (scraperjs)

Asked 2017-Oct-28 at 03:46

I tried to make a scraper using ruipgil/scraperjs function that is asynchronous. But when I put it inside a loop, it got jammed and won't work. I'm new to Node and the idea of asynchronous programming so please bear with me.

So, I have this code:

...

ANSWER

Answered 2017-Oct-28 at 02:40

You could try using recursion so that it doesn't get stuck in that while loop?

Source https://stackoverflow.com/questions/46985676

QUESTION

Getting 'undefined' when passing an array through Express res.render function

Asked 2017-Sep-05 at 17:38

My first attempt at a basic node app.

If I test run this file (> node index.js) with console.log(eventsArray) in it instead, the array successfully prints out data from the scraper module in the command window, so my scraper module is working.

And if I set eventsArray = ['this','is,'a','test','array']; in index.js as a test, then these strings do show up on homepage like I want after running the app with Express, so Express + Jade template are working (no need to include my ejs template here since it's working).

The problem: If I try to run the app as shown below (> node index.js), eventsArray appears to be 'undefined' when it's passed into the res.render function, so nothing shows up on localhost:3000 (homepage)

index.js:

...

ANSWER

Answered 2017-Sep-05 at 12:53

Your getEvents returns nothing, besides of this scraperjs.StaticScraper.create is async function it returns a promise.

You should return async result from getEvents:

Source https://stackoverflow.com/questions/46055444

QUESTION

Trying to run a script with Express on localhost:3000

Asked 2017-Sep-02 at 14:37

Stuck on my first attempt at a basic app. Scraper.js scrapes a URL and writes the returned array to the document obj when run alone in console, so that part works. Now all I want is an Express server to run the script whenever I open localhost:3000 but not sure how to do so.

...

ANSWER

Answered 2017-Sep-01 at 23:34

Here is a one that i've coded with promises, and also using a global variable which is daNews

Source https://stackoverflow.com/questions/46008828

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install scraperjs

Scraperjs exposes two different scrapers,.
a StaticScraper, that is light fast and with a low footprint, however it doesn't allow for more complex situations, like scraping dynamic content.
a DynamicScraper, that is a bit more heavy, but allows you to scrape dynamic content, like in the browser console. both scrapers expose a very similar API, with some minor differences when it comes to scraping.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: