crawler | Web crawler based on Puppeteer | Crawler library

by open-data-plan TypeScript Version: v1.6.2 License: MIT

X-Ray Key Features Code Snippets Community Discussions(10)Vulnerabilities Install Support

kandi X-RAY | crawler Summary

crawler is a TypeScript library typically used in Automation, Crawler applications. crawler has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. You can download it from GitHub.

Web crawler based on Puppeteer.

Support

Quality

Security

License

Reuse

Support

crawler has a low active ecosystem.

It has 13 star(s) with 3 fork(s). There are 2 watchers for this library.

It had no major release in the last 6 months.

There are 0 open issues and 1 have been closed. On average issues are closed in 1 days. There are 8 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of crawler is v1.6.2

Quality

crawler has no bugs reported.

Security

crawler has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

License

crawler is licensed under the MIT License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

crawler releases are not available. You will need to build from source code and install.

Installation instructions are not available. Examples and code snippets are available.

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of crawler

Get all kandi verified functions for this library.

crawler Key Features

No Key Features are available at this moment for crawler.

crawler Examples and Code Snippets

No Code Snippets are available at this moment for crawler.

Community Discussions

Trending Discussions on crawler

Accessing Aurora Postgres Materialized Views from Glue data catalog for Glue Jobs

Next.js Dynamic Meta Tags with SSG Not Pre-Rendering

Javascript Set, reference breaks when passing to function

Get AWS Glue Crawler to re-visit the folder for a partition that's been deleted

Cleaning column names in pandas

How to avoid "module not found" error while calling scrapy project from crontab?

AWS Glue Incremental crawl of continually arriving data on S3

Is it better to import static or dynamic with I/O Bound application

Trying to download files without starting scrapy project but from .py file. Created Custom pipeline within python file, This error comes as metioned

Measure code coverage using Xdebug when crawls web application

QUESTION

Accessing Aurora Postgres Materialized Views from Glue data catalog for Glue Jobs

Asked 2021-Jun-15 at 13:51

I have an Aurora Serverless instance which has data loaded across 3 tables (mixture of standard and jsonb data types). We currently use traditional views where some of the deeply nested elements are surfaced along with other columns for aggregations and such.

We have two materialized views that we'd like to send to Redshift. Both the Aurora Postgres and Redshift are in Glue Catalog and while I can see Postgres views as a selectable table, the crawler does not pick up the materialized views.

Currently exploring two options to get the data to redshift.

Output to parquet and use copy to load
Point the Materialized view to jdbc sink specifying redshift.

Wanted recommendations on what might be most efficient approach if anyone has done a similar use case.

Questions:

In option 1, would I be able to handle incremental loads?
Is bookmarking supported for JDBC (Aurora Postgres) to JDBC (Redshift) transactions even if through Glue?
Is there a better way (other than the options I am considering) to move the data from Aurora Postgres Serverless (10.14) to Redshift.

Thanks in advance for any guidance provided.

...

ANSWER

Answered 2021-Jun-15 at 13:51

Went with option 2. The Redshift Copy/Load process writes csv with manifest to S3 in any case so duplicating that is pointless.

Regarding the Questions:

N/A
Job Bookmarking does work. There is some gotchas though - ensure Connections both to RDS and Redshift are present in Glue Pyspark job, IAM self ref rules are in place and to identify a row that is unique [I chose the primary key of underlying table as an additional column in my materialized view] to use as the bookmark.
Using the primary key of core table may buy efficiencies in pruning materialized views during maintenance cycles. Just retrieve latest bookmark from cli using aws glue get-job-bookmark --job-name yourjobname and then just that in the where clause of the mv as where id >= idinbookmark

conn = glueContext.extract_jdbc_conf("yourGlueCatalogdBConnection") connection_options_source = { "url": conn['url'] + "/yourdB", "dbtable": "table in dB", "user": conn['user'], "password": conn['password'], "jobBookmarkKeys":["unique identifier from source table"], "jobBookmarkKeysSortOrder":"asc"}

datasource0 = glueContext.create_dynamic_frame.from_options(connection_type="postgresql", connection_options=connection_options_source, transformation_ctx="datasource0")

That's all, folks

Source https://stackoverflow.com/questions/67928401

QUESTION

Next.js Dynamic Meta Tags with SSG Not Pre-Rendering

Asked 2021-Jun-12 at 16:29

I have spent the better part of three days trying to get a Open Graph image generator working for my Next.js blog. After getting frustrated with hitting the 50mb function size limit I changed away from an API to a function call in the getStaticProps method of my pages/blog/[slug].tsx. This is working but now the issue is with the meta tags. I am dynamically setting them using the image path from the image generation function as well as information from the respective post. When I view the page source, I see all the appropriate tags and the open graph image has been generated and the path works but none of these tags are seen by crawlers. Upon checking the source file I realized that none of the head tags are pre-rendered. I am not sure if I am not understanding exactly what SSG does because I thought it would pre-render my blog pages (including the head). This seems like a common use case, and although I found some relevant questions on SO, I haven't found anyone really answering it. Is this an SSG limitation? I have seen tutorials for dynamic meta tags and they use SSR but that doesn't seem like it should be necessary.

...

ANSWER

Answered 2021-Jun-12 at 16:29

Thanks for anyone who looked at my issue. I figured it out! The way I implemented my dark mode used conditional rendering on the whole app to prevent any initial flash. I have changed the way I do dark mode and everything is working now!

Source https://stackoverflow.com/questions/67914091

QUESTION

Javascript Set, reference breaks when passing to function

Asked 2021-Jun-11 at 12:28

tl;dr: When I pass a Set to child functions which in turn adds new values to the Set, the new values are not added to the original Set. I find this wierd since it Set is an object. Why is this so?

I am building a web crawler in Node, which I will use to visit all pages on a domain. The basic algorithm is as follows:

Visit a page.
Save the url in a collection that contains all visited links.
Extract all links not yet visited.
Repeat 1-3 for all new links.

I use a Set as a container for the links, since lookup complexity is O(1). I initalize the Set before I start visiting the links, and pass it into the function that contains the logic. The problem is that when I add the link to the Set, it is not there the next time the function is called. It appears as when I pass the Set to a function, a new object is created. How do I get around this?

...

ANSWER

Answered 2021-Jun-11 at 12:28

The problem is that when I add the link to the Set, it is not there the next time the function is called.

No, that's not the problem. The Set is working fine, and doesn't change its identitiy.

The problem is that you're adding the links to the set at the wrong time, in a recursive function. By adding an url only when you actually visit it, it can happen that when multiple pages point to the same url, that url becomes part of multiple linksNotVisited arrays (at different recursive calls), and will then be visited multiple times.

Instead, check whether a link has been visited right before loading that page:

Source https://stackoverflow.com/questions/67936781

QUESTION

Get AWS Glue Crawler to re-visit the folder for a partition that's been deleted

Asked 2021-Jun-10 at 05:41

I have an AWS Glue crawler that is set-up to crawl new folders only. I tried to see if deleting a partition would cause it to re-visit the corresponding S3 folder, and it doesn't. Is there a way I can force a re-visit of a folder, short of changing the crawler to crawl all folders?

...

ANSWER

Answered 2021-Jun-09 at 08:44

If your partitions are "predictable", for example date based, you could completely bypass the crawlers and use partition projection. See the docs:

https://docs.aws.amazon.com/athena/latest/ug/partition-projection.html

Source https://stackoverflow.com/questions/67881748

QUESTION

Cleaning column names in pandas

Asked 2021-Jun-10 at 03:50

I have a Dataframe I receive from a crawler that I am importing into a database for long-term storage.

The problem I am running into is a large amount of the various dataframes have uppercase and whitespace.

I have a fix for it but I was wondering if it can be done any cleaner than this:

...

ANSWER

Answered 2021-Jun-10 at 03:45

You can try via columns attribute:

Source https://stackoverflow.com/questions/67914330

QUESTION

How to avoid "module not found" error while calling scrapy project from crontab?

Asked 2021-Jun-07 at 15:35

I am currently building a small test project to learn how to use crontab on Linux (Ubuntu 20.04.2 LTS).

My crontab file looks like this:

* * * * * sh /home/path_to .../crontab_start_spider.sh >> /home/path_to .../log_python_test.log 2>&1

What I want crontab to do, is to use the shell file below to start a scrapy project. The output is stored in the file log_python_test.log.

My shell file (numbers are only for reference in this question):

...

ANSWER

Answered 2021-Jun-07 at 15:35

I found a solution to my problem. In fact, just as I suspected, there was a missing directory to my PYTHONPATH. It was the directory that contained the gtts package.

Solution: If you have the same problem,

Find the package

I looked at that post

Add it to sys.path (which will also add it to PYTHONPATH)

Add this code at the top of your script (in my case, the pipelines.py):

Source https://stackoverflow.com/questions/67841062

QUESTION

AWS Glue Incremental crawl of continually arriving data on S3

Asked 2021-Jun-07 at 14:00

I'm looking for a way to set-up an incremental Glue crawler for S3 data, where data arrives continuously and is partitioned by the date it was captured (so the S3 paths within the include path contain date=yyyy-mm-dd). My concern is, that if I run the crawler in the course of a day, the partition for it will be created, and will not be re-visited in subsequent crawls. Is there a way to force a given partition, that I know might still be receiving updates, to be crawled while running the crawler incrementally and not wasting resources on historic data?

...

ANSWER

Answered 2021-Jun-07 at 14:00

The crawler will visit only new folders with an incremental crawl (assuming you have set crawl new folders only option). The only circumstance where adding more data to an existing folder would cause a problem is if you were changing schema by adding a differently formatted file into a folder that was already crawled. Otherwise the crawler has created the partition and knows the schema, and is ready to pull the data, even if new files are added to the existing folder.

Source https://stackoverflow.com/questions/67869433

QUESTION

Is it better to import static or dynamic with I/O Bound application

Asked 2021-Jun-07 at 09:53

I have been working on a I/O bound application which is a web crawler for news. I have one file where I start the script which we can call "monitoring.py" and by choosing which news company I want to monitor I add a parameter e.g. monitoring.py --company=sydsvenskan which will then trigger sydsvenskan webcrawling.

What it does is basically this:

scraper.py

...

ANSWER

Answered 2021-Jun-07 at 09:53

The universal answer for performance questions is : measure then decide.

You ask two questions.

Would it be faster to use dynamic imports ?

I would think so, but in a very negligeable way. Except if the computer running this code is very constrained, the difference would be barely noticeable (on the order of <1 second at startup time, and a few dozens of megabytes of RAM).

You can test it quickly by duplicating your sydsvenskan.py file 40 times, importing each of them in your scraper.py and running time python scraper.py before and after.

And in general, prefer doing simple things. Static imports are simpler than dynamic ones.

Can PyCharm still provide code insights even if the import is dynamic ?

Simply put : yes. I tested to put it in a function and it worked fine :

Source https://stackoverflow.com/questions/67858338

QUESTION

Trying to download files without starting scrapy project but from .py file. Created Custom pipeline within python file, This error comes as metioned

Asked 2021-Jun-05 at 18:16

import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.pipelines.files import FilesPipeline
from urllib.parse import urlparse
import os

class DatasetItem(scrapy.Item):
    file_urls = scrapy.Field()
    files = scrapy.Field()

class MyFilesPipeline(FilesPipeline):
    pass



class DatasetSpider(scrapy.Spider):
    name = 'Dataset_Scraper'
    url = 'https://kern.humdrum.org/cgi-bin/browse?l=essen/europa/deutschl/allerkbd'
    

    headers = {
        'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/53       7.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36'
    }
    
    custom_settings = {
            'FILES_STORE': 'Dataset',
            'ITEM_PIPELINES':{"/home/LaxmanMaharjan/dataset/MyFilesPipeline":1}

            }
    def start_requests(self):
        yield scrapy.Request(
                url = self.url,
                headers = self.headers,
                callback = self.parse
                )

    def parse(self, response):
        item = DatasetItem()
        links = response.xpath('.//body/center[3]/center/table/tr[1]/td/table/tr/td/a[4]/@href').getall()
        
        for link in links:
            item['file_urls'] = [link]
            yield item
            break
        

if __name__ == "__main__":
    #run spider from script
    process = CrawlerProcess()
    process.crawl(DatasetSpider)
    process.start()

...

ANSWER

Answered 2021-Jun-05 at 18:16

In case if pipeline code, spider code and process launcher stored in the same file
You can use __main__ in path to enable pipeline:

Source https://stackoverflow.com/questions/67737807

QUESTION

Measure code coverage using Xdebug when crawls web application

Asked 2021-Jun-04 at 13:30

I build my crawler based on ChromeDriver Selenium , and I want to measure the code coverage of the web application when my automated crawler crawls the application.

So, my question is how I do that using Xdebug (I'm newer on it). I installed Xdebug on my PHP, but I didn't know how to start? Can anyone have an idea to give me steps for that because I didn't find any resource that help me.

...

ANSWER

Answered 2021-Jun-04 at 13:30

I don't have a direct example, but I would approach this in the following way. The code is untested, and will likely require changes to work, take this as a starting point

In any case, you want to do the following things:

Collect code coverage data for each request, and store that to a file
Aggregate the code coverage data for each of these runs, and merge them

Collecting Code Coverage for Each Request

Traditionally code coverage is generated for unit tests, with PHPUnit. PHPUnit uses a separate library, PHP Code Coverage, to collect, merge and generate reports for the per-test collected coverage. You can use this library stand alone.

To collect the data, I would do composer require phpunit/php-code-coverage and then create an auto_prepend file, with something like the following in it:

Source https://stackoverflow.com/questions/67812731

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install crawler

You can download it from GitHub.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: