ershoufang | use scrapy crawl the second-hand house price | Crawler library

by xindoo Python Version: Current License: No License

X-Ray Key Features Code Snippets Community Discussions(2)Vulnerabilities Install Support

kandi X-RAY | ershoufang Summary

ershoufang is a Python library typically used in Automation, Crawler, Selenium applications. ershoufang has no bugs, it has no vulnerabilities and it has low support. However ershoufang build file is not available. You can download it from GitHub.

This project can help you crawl second-hand houses data from (lianjia.com)[lianjia.com]. This project is a scrapy project, Scrapy is an opensource crawling framework, it's pretty to use. And I use xpath to resolve second-hand houses data from html souce code. Before you run my code, you must install scrapy on your machine, you can use pip install it very quickly. pip install scrapy. Use this command to start it, it will export data into CSV format. scrapy crawl ershoufang -o item.csv. ershoufang is the name of spider. Actually, there are three spiders in this project ershoufang yanjiao and ershoufanghz, they correspond to the second-hand houses data spider of Beijing, Beijing Yanjiao, and Hangzhou. You can also create your own spider, just need to change the URL in the file ershoufang/spider/ershoufang.py line 8. In case of be banned by lianjian, I configured somethings in settings.py. eg. disable cookie, change the bot_name, crawl only one time each minute, use different use_agent(I commented it out), etc. I don't use proxy, because I don't have proxy IP source. 在你开始运行代码前，你必须先在你电脑上安装scrapy，用pip命令很快就可以装好。 pip install scrapy. 用这条命令启动代码，把数据导出成csv格式。 scrapy crawl ershoufang -o item.csv. 为了防止被链家禁掉，我在settings.py 做了一些配置，比如禁掉cookie，改了bot_name, 每分钟只爬一次，用不同的use_agent(配置被我注释掉了)…… 我没用proxy，因为我没proxy ip资源。.

Support

Quality

Security

License

Reuse

Support

ershoufang has a low active ecosystem.

It has 39 star(s) with 22 fork(s). There are 5 watchers for this library.

It had no major release in the last 6 months.

There are 0 open issues and 1 have been closed. On average issues are closed in 496 days. There are no pull requests.

It has a neutral sentiment in the developer community.

The latest version of ershoufang is current.

Quality

ershoufang has 0 bugs and 28 code smells.

Security

ershoufang has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

ershoufang code analysis shows 0 unresolved vulnerabilities.

There are 14 security hotspots that need review.

License

ershoufang does not have a standard license declared.

Check the repository for any license declaration and review the terms closely.

Without a license, all rights are reserved, and you cannot use the library in your applications.

Reuse

ershoufang releases are not available. You will need to build from source code and install.

ershoufang has no build file. You will be need to create the build yourself to build the component from source.

ershoufang saves you 93 person hours of effort in developing the same functionality from scratch.

It has 237 lines of code, 16 functions and 9 files.

It has low code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed ershoufang and discovered the below as its top functions. This is intended to give you an instant insight into ershoufang implemented functionality, and help decide if they suit your requirements.

Parse the meteostation response .
Return a generator of start requests .
Process response results .
Called when an exception is raised .
Adds a random user - agent to the request .
Process an item .
Initialize client .

Get all kandi verified functions for this library.

ershoufang Key Features

No Key Features are available at this moment for ershoufang.

ershoufang Examples and Code Snippets

No Code Snippets are available at this moment for ershoufang.

Community Discussions

Trending Discussions on ershoufang

Loop and crawler items and save elements as json format in Python

scrapy callback doesnt work in function

QUESTION

Loop and crawler items and save elements as json format in Python

Asked 2021-Mar-28 at 16:10

Given a link from here:

I would like to loop all the counties and then all commercial districts, the save them as a txt file with json format as follows:

...

ANSWER

Answered 2021-Mar-28 at 16:10

Is this what you want?

Source https://stackoverflow.com/questions/66842871

QUESTION

scrapy callback doesnt work in function

Asked 2017-Jun-24 at 12:49

When executing the first yield it will not go into the function parse_url and when executing the second yield it will not go back the function parse and it just end. During the whole process, there are no exceptions. I don't know how to deal with this problem, I need help.

...

ANSWER

Answered 2017-Jun-24 at 12:49

If you carefully looked at the logs then you might have noticed that scrapy filtered offsite domain requests. This means when scrapy tried to ping short.58.com and jxjump.58.com, it did not follow through. You can add those domains to the allowed_domains filter in your Spider class and you will see the requests being sent.

Replace:

Source https://stackoverflow.com/questions/44734655

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install ershoufang

You can download it from GitHub.
You can use ershoufang like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: