crawler | crawler by python base on scrapy | Crawler library
kandi X-RAY | crawler Summary
kandi X-RAY | crawler Summary
crawler by python base on scrapy
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Parse the response from Gfzinfo
- Save text to file
- Crawl crawl
- Start the cursor
- Execute the callback function
- Start the timer
crawler Key Features
crawler Examples and Code Snippets
Community Discussions
Trending Discussions on crawler
QUESTION
I have an Aurora Serverless instance which has data loaded across 3 tables (mixture of standard and jsonb data types). We currently use traditional views where some of the deeply nested elements are surfaced along with other columns for aggregations and such.
We have two materialized views that we'd like to send to Redshift. Both the Aurora Postgres and Redshift are in Glue Catalog and while I can see Postgres views as a selectable table, the crawler does not pick up the materialized views.
Currently exploring two options to get the data to redshift.
- Output to parquet and use copy to load
- Point the Materialized view to jdbc sink specifying redshift.
Wanted recommendations on what might be most efficient approach if anyone has done a similar use case.
Questions:
- In option 1, would I be able to handle incremental loads?
- Is bookmarking supported for JDBC (Aurora Postgres) to JDBC (Redshift) transactions even if through Glue?
- Is there a better way (other than the options I am considering) to move the data from Aurora Postgres Serverless (10.14) to Redshift.
Thanks in advance for any guidance provided.
...ANSWER
Answered 2021-Jun-15 at 13:51Went with option 2. The Redshift Copy/Load process writes csv with manifest to S3 in any case so duplicating that is pointless.
Regarding the Questions:
N/A
Job Bookmarking does work. There is some gotchas though - ensure Connections both to RDS and Redshift are present in Glue Pyspark job, IAM self ref rules are in place and to identify a row that is unique [I chose the primary key of underlying table as an additional column in my materialized view] to use as the bookmark.
Using the primary key of core table may buy efficiencies in pruning materialized views during maintenance cycles. Just retrieve latest bookmark from cli using
aws glue get-job-bookmark --job-name yourjobnameand then just that in the where clause of the mv aswhere id >= idinbookmarkconn = glueContext.extract_jdbc_conf("yourGlueCatalogdBConnection")connection_options_source = { "url": conn['url'] + "/yourdB", "dbtable": "table in dB", "user": conn['user'], "password": conn['password'], "jobBookmarkKeys":["unique identifier from source table"], "jobBookmarkKeysSortOrder":"asc"}
datasource0 = glueContext.create_dynamic_frame.from_options(connection_type="postgresql", connection_options=connection_options_source, transformation_ctx="datasource0")
That's all, folks
QUESTION
I have spent the better part of three days trying to get a Open Graph image generator working for my Next.js blog. After getting frustrated with hitting the 50mb function size limit I changed away from an API to a function call in the getStaticProps method of my pages/blog/[slug].tsx. This is working but now the issue is with the meta tags. I am dynamically setting them using the image path from the image generation function as well as information from the respective post. When I view the page source, I see all the appropriate tags and the open graph image has been generated and the path works but none of these tags are seen by crawlers. Upon checking the source file I realized that none of the head tags are pre-rendered. I am not sure if I am not understanding exactly what SSG does because I thought it would pre-render my blog pages (including the head). This seems like a common use case, and although I found some relevant questions on SO, I haven't found anyone really answering it. Is this an SSG limitation? I have seen tutorials for dynamic meta tags and they use SSR but that doesn't seem like it should be necessary.
ANSWER
Answered 2021-Jun-12 at 16:29Thanks for anyone who looked at my issue. I figured it out! The way I implemented my dark mode used conditional rendering on the whole app to prevent any initial flash. I have changed the way I do dark mode and everything is working now!
QUESTION
tl;dr: When I pass a Set to child functions which in turn adds new values to the Set, the new values are not added to the original Set. I find this wierd since it Set is an object. Why is this so?
I am building a web crawler in Node, which I will use to visit all pages on a domain. The basic algorithm is as follows:
- Visit a page.
- Save the url in a collection that contains all visited links.
- Extract all links not yet visited.
- Repeat 1-3 for all new links.
I use a Set as a container for the links, since lookup complexity is O(1). I initalize the Set before I start visiting the links, and pass it into the function that contains the logic. The problem is that when I add the link to the Set, it is not there the next time the function is called. It appears as when I pass the Set to a function, a new object is created. How do I get around this?
...ANSWER
Answered 2021-Jun-11 at 12:28The problem is that when I add the link to the Set, it is not there the next time the function is called.
No, that's not the problem. The Set is working fine, and doesn't change its identitiy.
The problem is that you're adding the links to the set at the wrong time, in a recursive function. By adding an url only when you actually visit it, it can happen that when multiple pages point to the same url, that url becomes part of multiple linksNotVisited arrays (at different recursive calls), and will then be visited multiple times.
Instead, check whether a link has been visited right before loading that page:
QUESTION
I have an AWS Glue crawler that is set-up to crawl new folders only. I tried to see if deleting a partition would cause it to re-visit the corresponding S3 folder, and it doesn't. Is there a way I can force a re-visit of a folder, short of changing the crawler to crawl all folders?
...ANSWER
Answered 2021-Jun-09 at 08:44If your partitions are "predictable", for example date based, you could completely bypass the crawlers and use partition projection. See the docs:
https://docs.aws.amazon.com/athena/latest/ug/partition-projection.html
QUESTION
I have a Dataframe I receive from a crawler that I am importing into a database for long-term storage.
The problem I am running into is a large amount of the various dataframes have uppercase and whitespace.
I have a fix for it but I was wondering if it can be done any cleaner than this:
...ANSWER
Answered 2021-Jun-10 at 03:45You can try via columns attribute:
QUESTION
I am currently building a small test project to learn how to use crontab on Linux (Ubuntu 20.04.2 LTS).
My crontab file looks like this:
* * * * * sh /home/path_to .../crontab_start_spider.sh >> /home/path_to .../log_python_test.log 2>&1
What I want crontab to do, is to use the shell file below to start a scrapy project. The output is stored in the file log_python_test.log.
My shell file (numbers are only for reference in this question):
...ANSWER
Answered 2021-Jun-07 at 15:35I found a solution to my problem. In fact, just as I suspected, there was a missing directory to my PYTHONPATH. It was the directory that contained the gtts package.
Solution: If you have the same problem,
- Find the package
I looked at that post
- Add it to sys.path (which will also add it to PYTHONPATH)
Add this code at the top of your script (in my case, the pipelines.py):
QUESTION
I'm looking for a way to set-up an incremental Glue crawler for S3 data, where data arrives continuously and is partitioned by the date it was captured (so the S3 paths within the include path contain date=yyyy-mm-dd). My concern is, that if I run the crawler in the course of a day, the partition for it will be created, and will not be re-visited in subsequent crawls. Is there a way to force a given partition, that I know might still be receiving updates, to be crawled while running the crawler incrementally and not wasting resources on historic data?
...ANSWER
Answered 2021-Jun-07 at 14:00The crawler will visit only new folders with an incremental crawl (assuming you have set crawl new folders only option). The only circumstance where adding more data to an existing folder would cause a problem is if you were changing schema by adding a differently formatted file into a folder that was already crawled. Otherwise the crawler has created the partition and knows the schema, and is ready to pull the data, even if new files are added to the existing folder.
QUESTION
I have been working on a I/O bound application which is a web crawler for news. I have one file where I start the script which we can call "monitoring.py" and by choosing which news company I want to monitor I add a parameter e.g. monitoring.py --company=sydsvenskan which will then trigger sydsvenskan webcrawling.
What it does is basically this:
scraper.py
...ANSWER
Answered 2021-Jun-07 at 09:53The universal answer for performance questions is : measure then decide.
You ask two questions.
Would it be faster to use dynamic imports ?I would think so, but in a very negligeable way. Except if the computer running this code is very constrained, the difference would be barely noticeable (on the order of <1 second at startup time, and a few dozens of megabytes of RAM).
You can test it quickly by duplicating your sydsvenskan.py file 40 times, importing each of them in your scraper.py and running time python scraper.py before and after.
And in general, prefer doing simple things. Static imports are simpler than dynamic ones.
Can PyCharm still provide code insights even if the import is dynamic ?Simply put : yes. I tested to put it in a function and it worked fine :
QUESTION
import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.pipelines.files import FilesPipeline
from urllib.parse import urlparse
import os
class DatasetItem(scrapy.Item):
file_urls = scrapy.Field()
files = scrapy.Field()
class MyFilesPipeline(FilesPipeline):
pass
class DatasetSpider(scrapy.Spider):
name = 'Dataset_Scraper'
url = 'https://kern.humdrum.org/cgi-bin/browse?l=essen/europa/deutschl/allerkbd'
headers = {
'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/53 7.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36'
}
custom_settings = {
'FILES_STORE': 'Dataset',
'ITEM_PIPELINES':{"/home/LaxmanMaharjan/dataset/MyFilesPipeline":1}
}
def start_requests(self):
yield scrapy.Request(
url = self.url,
headers = self.headers,
callback = self.parse
)
def parse(self, response):
item = DatasetItem()
links = response.xpath('.//body/center[3]/center/table/tr[1]/td/table/tr/td/a[4]/@href').getall()
for link in links:
item['file_urls'] = [link]
yield item
break
if __name__ == "__main__":
#run spider from script
process = CrawlerProcess()
process.crawl(DatasetSpider)
process.start()
ANSWER
Answered 2021-Jun-05 at 18:16In case if pipeline code, spider code and process launcher stored in the same file
You can use __main__ in path to enable pipeline:
QUESTION
I build my crawler based on ChromeDriver Selenium , and I want to measure the code coverage of the web application when my automated crawler crawls the application.
So, my question is how I do that using Xdebug (I'm newer on it). I installed Xdebug on my PHP, but I didn't know how to start? Can anyone have an idea to give me steps for that because I didn't find any resource that help me.
...ANSWER
Answered 2021-Jun-04 at 13:30I don't have a direct example, but I would approach this in the following way. The code is untested, and will likely require changes to work, take this as a starting point
In any case, you want to do the following things:
- Collect code coverage data for each request, and store that to a file
- Aggregate the code coverage data for each of these runs, and merge them
Collecting Code Coverage for Each Request
Traditionally code coverage is generated for unit tests, with PHPUnit. PHPUnit uses a separate library, PHP Code Coverage, to collect, merge and generate reports for the per-test collected coverage. You can use this library stand alone.
To collect the data, I would do composer require phpunit/php-code-coverage and then create an auto_prepend file, with something like the following in it:
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install crawler
You can use crawler like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page
