karton | Distributed malware processing framework based on Python | Machine Learning library

We have added a select field to simple products, so that we can define the price unit and display the unit after the price on the single product page. This works fine.

Additionally we have added a select field for variable products to define the price unit of each variation. This price unit should be displayed after the variation price on the product page:

I need to get some text from a website we are using to get our data from. I finally found how, using HtmlAgilityPack and finding the Xpath I'm able to print out some text from the website.

But when I try to print the date and kind, which is coded like this:

Introduction

After working with scrapy for a couple of weeks now, i still have problems, to figure out, some xpath expressions. Mostly i have big issues with extracting data of a table and "ul and li" tags.

Example Webpage i try to get data of: https://www.karton.eu/460x310x160-mm-Postal-Shipping-Box

There is a table with is called: "Productdata" and i need every single line of, but i dont get any of it.. I tried something like: response.xpath('//*[@id="2"]/tr/td/text()').getall(), response.xpath('//table[@class="table table-striped"]/tr/td/text()').getall()

My Code

Introduction

As i have to go more deeper in crawling, i face my next problem: crawling nested pages like: https://www.karton.eu/Faltkartons

My crawler has to start at this page, goes to https://www.karton.eu/Einwellige-Kartonagen and visit every product listed in this category.

It should do that with every subcategory of "Faltkartons" for every single product contained in every category.

EDITED

My code now looks like this:

After writing my first ?recursive? spider, i face some problems, i cant get fixed the whole day..

I did research, which misstakes can cause that 301 error, but every solution i tried, didnt help me out yet.

My console output

My modified settings.py

I have to create a spider which crawls information of https://www.karton.eu/einwellig-ab-100-mm and the weight of a product which is scrapable after following the productlink to its own page.

After running my code, i get following error message:

I already checked if the url is broken or not, so in my scrapy shell i could fetch it.

Code is used:

Writing a crawler that outputs: Title, Artikelnummer, Price, Delivery Status in a .csv

https://www.karton.eu/einwellig-ab-100-mm

Problem It´s really hard to figure out, which html-tag on that webpage contains the information i need. For example: Artikelnummer: 001 How do i collect the 001? There are several more tags, i do not clearly understand to get the info of

Im working on a new project and i try to crawl link

What i did

First of all i tried to get some informations in my shell, to work things out correctly. code i wrote in my shell: response.xpath(//div[@class="product-wrapper col-xs-6 col-md-4"]/text()').get() With this code i just want to print out the title of the product, but i get some very weird output:

my first problem was something with the robots.txt so i change my settings.py user agent and now it works, i guess we can determine that the error come from that change, right? correct me if im wrong.

After a bit of research i found out that this comes from wrong formatting and you can determine this error with something like that: response.xpath('normalize-space(//div[@class="product-wrapper col-xs-6 col-md-4"]/text())') but this didnt help me at all.

What can i do now?

I have encountered the error:

java.lang.NoSuchMethodError: java.lang.String.isEmpty()Z

The installed java version on my pc is 1.8.0_91.

The funny thing is, that this error does not occur on my pc, but on other pc's I tried to run my program. The error seems to be connected to a line from a class who looks up info from an excel-sheet via apache poi 4.1.1.

The troubling line of code is this one: if(!CellContent.isBlank()){ the complete class looks like this: