colly | Elegant Scraper and Crawler Framework for Golang | Crawler library

 by   gocolly Go Version: v2.1.0 License: Apache-2.0

kandi X-RAY | colly Summary

kandi X-RAY | colly Summary

colly is a Go library typically used in Automation, Crawler applications. colly has no bugs, it has no vulnerabilities, it has a Permissive License and it has medium support. You can download it from GitHub.

Lightning Fast and Elegant Scraping Framework for Gophers. Colly provides a clean interface to write any kind of crawler/scraper/spider. With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

            kandi-support Support

              colly has a medium active ecosystem.
              It has 19706 star(s) with 1603 fork(s). There are 324 watchers for this library.
              It had no major release in the last 12 months.
              There are 144 open issues and 381 have been closed. On average issues are closed in 89 days. There are 25 open pull requests and 0 closed requests.
              It has a neutral sentiment in the developer community.
              The latest version of colly is v2.1.0

            kandi-Quality Quality

              colly has 0 bugs and 0 code smells.

            kandi-Security Security

              colly has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
              colly code analysis shows 0 unresolved vulnerabilities.
              There are 0 security hotspots that need review.

            kandi-License License

              colly is licensed under the Apache-2.0 License. This license is Permissive.
              Permissive licenses have the least restrictions, and you can use them in most projects.

            kandi-Reuse Reuse

              colly releases are available to install and integrate.
              Installation instructions, examples and code snippets are available.
              It has 5025 lines of code, 258 functions and 54 files.
              It has medium code complexity. Code complexity directly impacts maintainability of the code.

            Top functions reviewed by kandi - BETA

            kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
            Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of colly
            Get all kandi verified functions for this library.

            colly Key Features

            No Key Features are available at this moment for colly.

            colly Examples and Code Snippets

            No Code Snippets are available at this moment for colly.

            Community Discussions


            What can the go-colly library do?
            Asked 2022-Apr-07 at 21:22

            Can the go-colly library crawl all HTML tags and text content under a div tag? If so, how? I can get all texts under a div tag. Like this:



            Answered 2022-Apr-07 at 21:22

            If you looking for innerHTML it is accessible by DOM and using Html method (e.DOM.Html()).



            Getting attribute value with Go Colly
            Asked 2022-Mar-26 at 20:33

            When working with c.OnHTML in "html", how can I get the value of the href attribute inside the #id-card-1 ID?



            Answered 2022-Mar-26 at 20:33

            The ChildAttr function can use for this purpose.

            ChildAttr returns the stripped text content of the first matching element's attribute.




            What is the default mode in GoColly, sync or async?
            Asked 2022-Jan-24 at 07:58

            What is the default mode in which network requests are executed in GoColly? Since we have the Async method in the collector I would assume that the default mode is synchronous. However, I see no particular difference when I execute these 8 requests in the program other than I need to use Wait for async mode. It seems as if the method only controls how the program is executed (the other code) and the requests are always asynchronous.



            Answered 2022-Jan-24 at 07:58

            The default collection is synchronous.

            The confusing bit is probably the collector option colly.Async() which ignores the actual param. In fact the implementation at the time of writing is:



            I cannot web-scrape forbes top billionares website with colly go
            Asked 2021-Nov-15 at 13:36
            package main
            import (
             func checkError(err error){
             if err!=nil{
            func main(){
            defer file.Close()
            defer writer.Flush()
            c.OnHTML(".scrolly-table tbody tr", func(e *colly.HTMLElement) {
                        e.ChildText(".rank .ng-binding"),
                c.OnError(func(_ *colly.Response, err error) {
                    fmt.Println("Something went wrong:", err)
                c.OnRequest(func(r *colly.Request) {
                    fmt.Println("Visiting", r.URL)
                c.OnResponse(func(r *colly.Response) {
                    fmt.Println("Visited", string(r.Body))


            Answered 2021-Nov-01 at 05:23

            Make sure what is available if you disable javascript in your browser (you can do it using the developer tools). Most scrapers will only get you the textual representation of the page, while the browser will also run javascript engine against it. If the data you are trying to scrape is populated with Javascript, there is a very good chance that is the reason you can't scrape it.



            Pygame vertical collisions
            Asked 2021-Nov-05 at 20:07

            I have a code which make the player move in two dimensions, with no gravity (like Isaac or World Hardest Game). The problem are the collisions with the tiles (celle) in the map:



            Answered 2021-Nov-05 at 20:07

            The best way i know to handle 2d collision is to move the player on one axis, check for collision and then move the player on the other axis:



            how to scrape attribute in attibute with colly
            Asked 2021-Oct-21 at 13:39

            I try to scrape productId of a product but i can not. please help

            html code



            Answered 2021-Oct-21 at 13:39

            The attribute value is a raw value, and in this case, it's in JSON format, so you will need to parse the JSON in order to correctly get the data.

            For example:



            Create an array of a certain type and reuse it
            Asked 2021-Aug-31 at 01:14

            I'm having trouble creating the payload for my graphql resolver. How could I rewrite this to return a completed array?

            I'm stuck inside of c.OnHTML("article", func(e *colly.HTMLElement) {} and can't return the data outside of it.



            Answered 2021-Aug-29 at 04:57

            If you need to get full list of articles, you need to append new article into articles slice inside of your c.OnHTML function instead creating new slice.

            and return articles slice end of the News() Method.



            How can I write one after another JSON data
            Asked 2021-Jul-18 at 20:16

            I am working on a website scraper. I can send only 1 JSON data to JSON file regularly. I want to write one after another JSON data, so I need to keep hundreds of data in a single JSON file. like this



            Answered 2021-Jul-18 at 15:34

            Here is a solution which appends new Info to the list and store in file. The solution will perform properly only for relatively small list. For large lists, the overhead of writing the entire file each time may be too high. In such case i propose to change the format to ndjson. It will allow to write only the current Info struct instead of the whole list. I've also added synchronization mechanism to avoid race conditions in case you send multiple HTTP requests at the same time.

            I assumed that the identifier must be generated separately for each request, and it is not a problem if collision occur.



            type 'List' is not a subtype of type 'Map' need to conver list of map as map
            Asked 2021-Jul-10 at 11:50

            the result I got as List



            Answered 2021-Jul-10 at 11:50

            i don't think this is map

            {name: john, voted: [{5fhh54522b5: 8}, {2cg128gsc4541: 822}]}, {name: Donald, voted: [{8br55rj25ns3j: 822}, {jfej4v85552: 1}]}, {name: Abraham, voted: []}, {name: Colly, voted: []}

            they are multiple maps, not single map.

            you can use list.asMap() function to convert list of any to map,

            HI, i tried in dart pad, i can see that it is not valid json, you have to convert it like this, after that it you will be able to parse the json string easily

            voted :[ { id : '5fhh54522b5', count : '8'}, ]



            Append property from two arrays of objects
            Asked 2021-Jul-05 at 15:21

            In JavaScript, I have two arrays - arr1, arr2

            arr1 is an array of objects as shown below -



            Answered 2021-Jul-05 at 14:56

            You can use map and find in your case.


            Community Discussions, Code Snippets contain sources that include Stack Exchange Network


            No vulnerabilities reported

            Install colly

            Add colly to your go.mod file:.


            Bugs or suggestions? Visit the issue tracker or join #colly on freenode.
            Find more information at:

            Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items

            Find more libraries
          • HTTPS


          • CLI

            gh repo clone gocolly/colly

          • sshUrl


          • Stay Updated

            Subscribe to our newsletter for trending solutions and developer bootcamps

            Agree to Sign up and Terms & Conditions

            Share this Page

            share link

            Explore Related Topics

            Consider Popular Crawler Libraries


            by scrapy


            by cheeriojs


            by winstonjs


            by binux


            by gocolly

            Try Top Libraries by gocolly


            by gocollyGo


            by gocollyGo


            by gocollyHTML