metascraper | Get unified metadata from websites using Open Graph | Scraper library

by microlinkhq HTML Version: 5.45.10 License: MIT

X-Ray Key Features Code Snippets(1)Community Discussions(4)Vulnerabilities Install Support

kandi X-RAY | metascraper Summary

metascraper is a HTML library typically used in Automation, Scraper applications. metascraper has no bugs, it has no vulnerabilities, it has a Permissive License and it has medium support. You can download it from GitHub.

A library to easily scrape metadata from an article on the web using Open Graph, JSON+LD, regular HTML metadata, and series of fallbacks.

Support

Quality

Security

License

Reuse

Support

metascraper has a medium active ecosystem.

It has 2049 star(s) with 154 fork(s). There are 18 watchers for this library.

There were 5 major release(s) in the last 6 months.

There are 11 open issues and 214 have been closed. On average issues are closed in 127 days. There are 1 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of metascraper is 5.45.10

Quality

metascraper has 0 bugs and 0 code smells.

Security

metascraper has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

metascraper code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

metascraper is licensed under the MIT License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

metascraper releases are available to install and integrate.

Installation instructions, examples and code snippets are available.

It has 351615 lines of code, 0 functions and 293 files.

It has low code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of metascraper

Get all kandi verified functions for this library.

metascraper Key Features

No Key Features are available at this moment for metascraper.

metascraper Examples and Code Snippets

cors problem using nuxt in with SSR and proxy

JavaScript

Lines of Code : 32

License : Strong Copyleft (CC BY-SA 4.0)

Copy

export default {
  serverMiddleware: [
    { path: '/api/meta-scraper', handler: '~/api/meta-scraper.js' }
  ]
}

export default function (req, res, next) {
  // req is the Node.js http request object
  console.log(

Community Discussions

Trending Discussions on metascraper

How to import multiple NodeJS packages together?

cors problem using nuxt in with SSR and proxy

Async NullRefrenceException from Html Agility Pack's LoadFromWebAsync

How to get Open Graph meta from URL input on the client side

QUESTION

How to import multiple NodeJS packages together?

Asked 2022-Feb-04 at 01:41

const metascraper = require('metascraper')([
  require('metascraper-author')(),
  require('metascraper-date')(),
  require('metascraper-description')(),
  require('metascraper-image')(),
  require('metascraper-logo')(),
  require('metascraper-clearbit')(),
  require('metascraper-publisher')(),
  require('metascraper-title')(),
  require('metascraper-url')()
])

...

ANSWER

Answered 2022-Feb-04 at 01:41

The following should work:

Source https://stackoverflow.com/questions/70980096

QUESTION

cors problem using nuxt in with SSR and proxy

Asked 2020-Oct-26 at 12:44

I want to make a simple wishlist with nuxt (project for fun) the idea is simple, the user add links to the main page for items he want ( ex: https://www.amazon.fr/Bracelet-ethnique-multicolore-pierres-aventurine/dp/B088FNK7WS?pd_rd_w=VcswS&pf_rd_p=93b57529-44d6-414c-8e38-22f57431b7bd&pf_rd_r=F71JXZXPJQFHJR7E6ZCB&pd_rd_r=5c6a3226-b724-44f0-9c7c-1f7d75b21c00&pd_rd_wg=AJYnZ ), links are stored to an array, ect... for a better visual, i want to prefill the name of the item and display a picture of this item based on the open graph data of the link (if available), for this usage, i found this plugin: https://metascraper.js.org#/?id=usage (work fine) to scrap the metadata of the page i target.

My problem is the following: when i do a simple this.$axios.$get(link) i have a cors issue (No** 'Access-Control-Allow-Origin' header is present on the requested resource**.) After some research, i found this error is due to Server Side Rendering and need to be solved with the usage of a proxy. But the use of @nuxt/proxy need to define one proxy per website (ex: '/amazon/': { target: 'https://amazon.com/', pathRewrite: { '^/amazon/': '' } } )

Or, i need to scrap data from any link my user put in her wishlist. Can i still use @nuxt/proxy or i need a more complex solution ?

...

ANSWER

Answered 2020-Oct-26 at 09:05

This error is actually a browser security feature, and it's triggered when you try to access a resource from a different origin, read more here

To work around it, simply fire the scrapping request from the server-side, this way you can make your server act as a proxy. And since you are using nuxt you can easily achieve this through server-middleware, thus you can create a custom API e.g: /api/meta-scraper?target=http://www.amazon.com... and from there you can fetch the needed metadata. See the below example code.

nuxt.config.js

Source https://stackoverflow.com/questions/64529365

QUESTION

Async NullRefrenceException from Html Agility Pack's LoadFromWebAsync

Asked 2020-Oct-05 at 08:31

I'm using Html Agility Pack to get meta tags from URLs. I was using HtmlWeb's Load, but I found it to be taking too long and thought that I could probably speed up performance using an async method instead. However, when I tried to use LoadFromWebAsync I get a NullReferenceException.

Using Load

...

ANSWER

Answered 2020-Oct-05 at 08:31

The issue was that I was attempting to show the data before it was loaded. This was because I was using async but not providing something to show while it was awaiting a result. It can be fixed by doing:

Source https://stackoverflow.com/questions/64202783

QUESTION

How to get Open Graph meta from URL input on the client side

Asked 2020-Jul-07 at 08:17

When an user submit a post with an URL. I want to get the Open Graph meta of the URL without a back-end server.

I've found libraries that can extract Open Graph meta such as opengraph-io and metascraper, but they all require a Node.js server to do the job.

Is it possible to achieve this on client side only?

...

ANSWER

Answered 2020-Jul-07 at 08:17

I have found a solution. Posting this answer to help anyone who is searching for a similar solution.

https://www.opengraph.io/ provides up to 100 requests a month free. You need to signup to get the API key to place into the app_id. In just few lines of code, I get the meta data in a response object:

Source https://stackoverflow.com/questions/62634812

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install metascraper

metascraper is library to easily scrape metadata from an article on the web using Open Graph metadata, regular HTML metadata, and series of fallbacks.
Have a high accuracy for online articles by default.
Make it simple to add new rules or override existing ones.
Don't restrict rules to CSS selectors or text accessors.