crawl | Utility to crawl and diff websites for node.js | Crawler library
kandi X-RAY | crawl Summary
kandi X-RAY | crawl Summary
NOTE: This project is no longer being maintained by me. If you are interested in taking over maintenance of this project, let me know. Crawl, as it's name implies, will crawl around a website, discovering all of the links and their relationships starting from a base URL. The output of crawl is a JSON object representing a sitemap of every resource within a site, including each links outbound references and any inbound refferers. Crawl is a Node.js based library that can be used as a module within another application, or as a stand alone tool via it's command line interface (CLI).
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of crawl
crawl Key Features
crawl Examples and Code Snippets
Lines of Code : 16
License : No License
Copyvar cheerio = require('cheerio'); // Basically jQuery for node.js
var options = {
uri: 'http://www.google.com',
transform: function (body) {
return cheerio.load(body);
}
};
rp(options)
.then(function ($) {
// Process rp('http://www.google.com')
.then(function (htmlString) {
// Process html...
})
.catch(function (err) {
// Crawling failed...
});
@Override
public String handleRequest(String[] input, Context context) {
System.setProperty("webdriver.chrome.verboseLogging", "true");
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.setExperimentalOption("excludeSwi public static List crawl(String startUrl, HtmlParser htmlParser) {
String host = getHost(startUrl);
List result = new ArrayList<>();
Set visited = new HashSet<>();
result.add(startUrl);
visited.add( def crawl(url, max_urls=30):
"""
Crawls a web page and extracts all links.
You'll find all links in `external_urls` and `internal_urls` global set variables.
params:
max_urls (int): number of max urls to crawl, default is 30.
import scrapy
import logging
#base url=https://arzdigital.com/latest-posts/
#start_url =https://arzdigital.com/latest-posts/page/2/
class CriptolernSpider(scrapy.Spider):
name = 'criptolern'
allowed_domains = ['arzdigital.com']
$ scrapy crawl links -t jsonlines -o links.json
$ scrapy crawl links -t json -o links.json
import json
class TinyhouselistingsSpider(scrapy.Spider):
name = 'tinyhouselistings'
listings_url = 'https://thl-prod.global.ssl.fastly.net/api/v1/listings/search?area_min=0&measurement_unit=feet&page={}'
def start_reasync function crawl(directory, filesArray) {
const dirs = await fsPromises.readdir(directory, {
withFileTypes: true
});
//loop through all files/directories
for (let i = 0; i < dirs.length; i++) {
cons let crawl = async function(){
let browser = await puppeteer.launch({ headless:false });
const context = browser.defaultBrowserContext();
// URL An array of permissions
cCommunity Discussions
Trending Discussions on crawl
QUESTION
I am currently on the path of learning C++ and this is an example program I wrote for the course I'm taking. I know that there are things in here that probably makes your skin crawl if you're experienced in C/C++, heck the program isn't even finished, but I mainly need to know why I keep receiving this error after I enter my name: Exception thrown at 0x79FE395E (vcruntime140d.dll) in Learn.exe: 0xC0000005: Access violation reading location 0xCCCCCCCC. I know there is something wrong with the constructors and initializations of the member variables of the classes but I cannot pinpoint the problem, even with the debugger. I am running this in Visual Studio and it does initially run, but I realized it does not compile with GCC. Feel free to leave some code suggestions, but my main goal is to figure out the program-breaking issue.
ANSWER
Answered 2021-Jun-13 at 00:59The problem is here:
QUESTION
I have a AWS Glue job that was slightly modified, only the read was changed, the job runs fine however the datatypes on my columns have changed. Where I previously had BigInt, I now just have Ints. This is causing an EMR Job dependent on these files to error out due to the schema mismatch. I'm not sure what would cause this issue since the mapping did not change, so if anyone has insight that would be great here is the old & new code:
...ANSWER
Answered 2021-Jun-10 at 14:28Both spark DataFrame and glue DynamicFrame infer the schema when reading data from json, but evidently, they do it differently: sparks treats all numerical values as bigint, while glue is trying to be clever, and (I guess) looks at the actual range of values on the fly.
Some more info about DynamicFrame schema inference can be found here.
If you are going to write parquet in the end anyway, and want the schema stable and consistent, I'd say your easiest way around this is to just revert your change and go back to spark DataFrame.
You can also use apply_mapping to change the types explicitly after reading the data, but it seems like defeating the purpose of having the dynamic frame in the first place.
QUESTION
I have an AWS Glue crawler that is set-up to crawl new folders only. I tried to see if deleting a partition would cause it to re-visit the corresponding S3 folder, and it doesn't. Is there a way I can force a re-visit of a folder, short of changing the crawler to crawl all folders?
...ANSWER
Answered 2021-Jun-09 at 08:44If your partitions are "predictable", for example date based, you could completely bypass the crawlers and use partition projection. See the docs:
https://docs.aws.amazon.com/athena/latest/ug/partition-projection.html
QUESTION
Greeting, in general the problem is this, I created a web application using React JS, like a database using Firesbase Firestore. Everything worked fine until it was time to update the security rules (they were temporary, well, and time was up). It demanded to immediately change the rules, otherwise the base will stop responding after the expiration of the term. At first, I just extended the temporary rules, but it only worked once, after that all such attempts were in vain. After reading the documentation on writing security rules and looking at a couple of tutorials, I decided to write simple rules allow read: if true; allow write: if false;. In the project, the user does not interact with the base in any way, the text simply comes from the base and everything is essentially, so these rules are more than enough. I also additionally checked these rules on the emulator and everything went well. I saved the rules, but the application did not rise, I tried other options, to the extent that I simply put true everywhere and made the base completely open, but to no avail. I have already tried everything and crawled everything, but I still could not find a solution.
My app code:
...ANSWER
Answered 2021-Jun-08 at 12:01Posting this as a Community Wiki as it's based on the comments of @samthecodingman and @spectrum_10101.
The error is being generated by either testEng/test or testUa/test not actually existing, so their data will be set as undefined. So it's likely that the root cause of this issue is located somewhere else in your app.
QUESTION
I'm trying to get the input tag and use click() by using selenium.
Here is my code:
...ANSWER
Answered 2021-Jun-08 at 04:21The element that you are looking for, is in iframe. So we would have to change the driver focus in order to interact with the desire element or elements :
Iframe xpath :
QUESTION
I have a list of pages to crawl using selenium
Let's say the website is example.com/1...N (up to unknown size)
...ANSWER
Answered 2021-Jun-08 at 02:22Initialize a last_page variable to be infinity (preferably in a class variable)
And update and crawl with the following logic would be good enough
Since two threads can update last_page at the same time,
prevent higher page overwrite last_page updated by lower page
QUESTION
I am currently building a small test project to learn how to use crontab on Linux (Ubuntu 20.04.2 LTS).
My crontab file looks like this:
* * * * * sh /home/path_to .../crontab_start_spider.sh >> /home/path_to .../log_python_test.log 2>&1
What I want crontab to do, is to use the shell file below to start a scrapy project. The output is stored in the file log_python_test.log.
My shell file (numbers are only for reference in this question):
...ANSWER
Answered 2021-Jun-07 at 15:35I found a solution to my problem. In fact, just as I suspected, there was a missing directory to my PYTHONPATH. It was the directory that contained the gtts package.
Solution: If you have the same problem,
- Find the package
I looked at that post
- Add it to sys.path (which will also add it to PYTHONPATH)
Add this code at the top of your script (in my case, the pipelines.py):
QUESTION
I'm looking for a way to set-up an incremental Glue crawler for S3 data, where data arrives continuously and is partitioned by the date it was captured (so the S3 paths within the include path contain date=yyyy-mm-dd). My concern is, that if I run the crawler in the course of a day, the partition for it will be created, and will not be re-visited in subsequent crawls. Is there a way to force a given partition, that I know might still be receiving updates, to be crawled while running the crawler incrementally and not wasting resources on historic data?
...ANSWER
Answered 2021-Jun-07 at 14:00The crawler will visit only new folders with an incremental crawl (assuming you have set crawl new folders only option). The only circumstance where adding more data to an existing folder would cause a problem is if you were changing schema by adding a differently formatted file into a folder that was already crawled. Otherwise the crawler has created the partition and knows the schema, and is ready to pull the data, even if new files are added to the existing folder.
QUESTION
import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.pipelines.files import FilesPipeline
from urllib.parse import urlparse
import os
class DatasetItem(scrapy.Item):
file_urls = scrapy.Field()
files = scrapy.Field()
class MyFilesPipeline(FilesPipeline):
pass
class DatasetSpider(scrapy.Spider):
name = 'Dataset_Scraper'
url = 'https://kern.humdrum.org/cgi-bin/browse?l=essen/europa/deutschl/allerkbd'
headers = {
'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/53 7.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36'
}
custom_settings = {
'FILES_STORE': 'Dataset',
'ITEM_PIPELINES':{"/home/LaxmanMaharjan/dataset/MyFilesPipeline":1}
}
def start_requests(self):
yield scrapy.Request(
url = self.url,
headers = self.headers,
callback = self.parse
)
def parse(self, response):
item = DatasetItem()
links = response.xpath('.//body/center[3]/center/table/tr[1]/td/table/tr/td/a[4]/@href').getall()
for link in links:
item['file_urls'] = [link]
yield item
break
if __name__ == "__main__":
#run spider from script
process = CrawlerProcess()
process.crawl(DatasetSpider)
process.start()
ANSWER
Answered 2021-Jun-05 at 18:16In case if pipeline code, spider code and process launcher stored in the same file
You can use __main__ in path to enable pipeline:
QUESTION
So i have this nodejs that was originaly used as api to crawl data using puppeteer from a website based on a schedule, now to check if there is a schedule i used a function that link to a model query and check if there are any schedule at the moment.
It seems to work and i get the data, but when i was crawling the second article and the next there is always this error UnhandledPromiseRejectionWarning: Error: Request is already handled! and followed by UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch().
and it seems to take a lot of resource from the cpu and memory.
So my question is, is there any blocking in my code or anything that could have done better.
this is my server.js
...ANSWER
Answered 2021-Jun-05 at 16:26I figured it out, i just used puppeteer cluster.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install crawl
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page
