urlscan | Mutt and terminal url selector | Command Line Interface library

The respons.json() method returns a dictionary. You can iterate over the keys and values of this dictionary and print.

If there is other HTML content elsewhere in your PHP script then this will also be outputted as it normally is, except in this case it will become part of the downloaded file. If you don't want that to happen then you have to stop your script with an exit(); command after you have output the content you actually want. In your script, it looks like you can probably do this just after the call to the function. (But if you have already output some HTML before this, you'll need to alter your script more substantially.)

N.B. I'm surprised you aren't getting a warning about headers being already sent? That normally happens if you try to set headers after you've already echoed some content. Check your log files. Normally you are supposed to output the headers first.

Also, unless you are wanting to keep it for some other purpose, there is no use in saving anything to urlscan.txt - it is not playing any part in your download process. And it would get overwritten every time this script is executed anyway. The headers will cause the browser to treat the output contents (i.e. anything which the PHP script sends to the regular output) as a text file - but this is not the same file as the text file on your server's disk, and its contents can be different.

You happen to be outputting similar content (via echo" $formatted_url
";) as you are adding to the urlscan file (via fwrite($fh, $formatted_url."\n");) and I think this may be confusing you into thinking that you're outputting the contents of urlscan.txt, but you aren't - your PHP headers are telling the browser to treat the output of your script (which would normally just go onto the browser window as a HTML page) as a file - but it's a) a new file, and b) actually isn't a file at all until it reaches the browser, it's just a response to a HTTP request. The browser turns it into a file on the client machine because of how it interprets the headers.

Another thing: the content you output needs to be in text format, not HTML, so you need to change the
in your echo to a \n.

Lastly, you're outputting the content-type header twice, which is nonsense. A HTTP request or response can only have one content type. In this case, text/plain is the valid MIME type, the other one is not real.

Taking into account all of the above, your code would probably be better written as:

Except that data should be a dict, not a preformatted json string, you're not doing anything wrong.

The URL is wrong in the JS snippet. It should be

You should inject your database service into your ẀebCrawler instances and not use a singleton to manage the result of your web-crawl.

crawler4j supports a custom CrawlController.WebCrawlerFactory (see here for reference), which can be used with Spring to inject your database service into a ImageCrawler instance.

Every single crawler thread should be responsible for the whole process you described with (e.g. by using some specific services for it):

decode this image, get the initiator of search and save results to database

Setting it up like this, your database will be the only source of truth and you will not have to deal with synchronizing crawler-states between different instances or user-sessions.

url.Value is a map[string][]string containing values used in query parameters or POST form. You would need it if you were trying to do something like:

You can use the UrlEncode or UrlPathEncode methods from the HttpUtility class to achieve what you need. See documentation at https://msdn.microsoft.com/en-us/library/system.web.httputility.urlencode(v=vs.110).aspx

It's important to understand however, that you should not need to encode the whole URL string. It's only the parameter values - which may contain arbitrary data and characters which aren't valid in a URL - that you need to encode.

To explain this concept, run the following in a simple .NET console application: