Support
Quality
Security
License
Reuse
kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample Here
Get all kandi verified functions for this library.
:bathtub: Clean Code concepts adapted for JavaScript
See all related Code Snippets
QUESTION
Do you need background workers or multiple threads to fire multiple Async HttpWebRequests?
Asked 2017-Jul-12 at 21:00I'm trying to call to the Google PageSpeed Insights API with mutliple input urls read from a .txt
file and to output the results to a .csv
.
I wrote a console app to try to fire these requests off, and then as they come back to add them to a list, and when they are all done, to write the list
to the .csv
file (async got a little nutty when trying to write the responses immediately to the .csv
).
My code it below, and far from optimized. I come form a JavaScript background, where I usually don't use web workers or any other managed new threads, so I was trying to do the same in C#.
WebRequest
s and write them to a collection (or output file) without using multiple threads and have them all run in parallel, not having to wait for each request to come back before handling the next one?BackgroundWorker
s are needed, what's a Clean Code way of doing this?static void Main(string[] args)
{
Console.WriteLine("Begin Google PageSpeed Insights!");
appMode = ConfigurationManager.AppSettings["ApplicationMode"];
var inputFilePath = READ_WRITE_PATH + ConfigurationManager.AppSettings["InputFile"];
var outputFilePath = READ_WRITE_PATH + ConfigurationManager.AppSettings["OutputFile"];
var inputLines = File.ReadAllLines(inputFilePath).ToList();
if (File.Exists(outputFilePath))
{
File.Delete(outputFilePath);
}
List<string> outputCache = new List<string>();
foreach (var line in inputLines)
{
var requestDataFromPsi = CallPsiForPrimaryStats(line);
Console.WriteLine($"Got response of {requestDataFromPsi.Result}");
outputCache.Add(requestDataFromPsi.Result);
}
var writeTask = WriteCharacters(outputCache, outputFilePath);
writeTask.Wait();
Console.WriteLine("End Google PageSpeed Insights");
}
private static async Task<string> CallPsiForPrimaryStats(string url)
{
HttpWebRequest myReq = (HttpWebRequest)WebRequest.Create($"https://www.googleapis.com/pagespeedonline/v2/runPagespeed?url={url}&strategy=mobile&key={API_KEY}");
myReq.Method = WebRequestMethods.Http.Get;
myReq.Timeout = 60000;
myReq.Proxy = null;
myReq.ContentType = "application/json";
Task<WebResponse> task = Task.Factory.FromAsync(
myReq.BeginGetResponse,
asyncResult => myReq.EndGetResponse(asyncResult),
(object)null);
return await task.ContinueWith(t => ReadStreamFromResponse(t.Result));
}
private static string ReadStreamFromResponse(WebResponse response)
{
using (Stream responseStream = response.GetResponseStream())
using (StreamReader sr = new StreamReader(responseStream))
{
string jsonResponse = sr.ReadToEnd();
dynamic jsonObj = Newtonsoft.Json.JsonConvert.DeserializeObject(jsonResponse);
var psiResp = new PsiResponse()
{
Url = jsonObj.id,
SpeedScore = jsonObj.ruleGroups.SPEED.score,
UsabilityScore = jsonObj.ruleGroups.USABILITY.score,
NumberResources = jsonObj.pageStats.numberResources,
NumberHosts = jsonObj.pageStats.numberHosts,
TotalRequestBytes = jsonObj.pageStats.totalRequestBytes,
NumberStaticResources = jsonObj.pageStats.numberStaticResources,
HtmlResponseBytes = jsonObj.pageStats.htmlResponseBytes,
CssResponseBytes = jsonObj.pageStats.cssResponseBytes,
ImageResponseBytes = jsonObj.pageStats.imageResponseBytes,
JavascriptResponseBytes = jsonObj.pageStats.javascriptResponseBytes,
OtherResponseBytes = jsonObj.pageStats.otherResponseBytes,
NumberJsResources = jsonObj.pageStats.numberJsResources,
NumberCssResources = jsonObj.pageStats.numberCssResources,
};
return CreateOutputString(psiResp);
}
}
static async Task WriteCharacters(List<string> inputs, string outputFilePath)
{
using (StreamWriter fileWriter = new StreamWriter(outputFilePath))
{
await fileWriter.WriteLineAsync(TABLE_HEADER);
foreach (var input in inputs)
{
await fileWriter.WriteLineAsync(input);
}
}
}
private static string CreateOutputString(PsiResponse psiResponse)
{
var stringToWrite = "";
foreach (var prop in psiResponse.GetType().GetProperties())
{
stringToWrite += $"{prop.GetValue(psiResponse, null)},";
}
Console.WriteLine(stringToWrite);
return stringToWrite;
}
Problem is this still runs slow. The original took 20 minutes, and after refactor it still took 20 minutes. It seems to be throttled somewhere, maybe by Google on the PageSpeed API. I tested it, calling calling https://www.google.com/, https://www.yahoo.com/, https://www.bing.com/ and 18 others and it runs slowly as well, having a bottleneck of only processing about 5 requests at a time. I tried refactoring to run 5 test URLs and then write to file and repeat but it only marginally sped up the process.
static void Main(string[] args) { MainAsync(args).Wait(); }
static async Task MainAsync(string[] args)
{
Console.WriteLine("Begin Google PageSpeed Insights!");
appMode = ConfigurationManager.AppSettings["ApplicationMode"];
var inputFilePath = READ_WRITE_PATH + ConfigurationManager.AppSettings["InputFile"];
var outputFilePath = READ_WRITE_PATH + ConfigurationManager.AppSettings["OutputFile"];
var inputLines = File.ReadAllLines(inputFilePath).ToList();
if (File.Exists(outputFilePath))
{
File.Delete(outputFilePath);
}
var tasks = inputLines.Select(line => CallPsiForPrimaryStats(line));
var outputCache = await Task.WhenAll(tasks);
await WriteCharacters(outputCache, outputFilePath);
Console.WriteLine("End Google PageSpeed Insights");
}
private static async Task<string> CallPsiForPrimaryStats(string url)
{
HttpWebRequest myReq = (HttpWebRequest)WebRequest.Create($"https://www.googleapis.com/pagespeedonline/v2/runPagespeed?url={url}&strategy=mobile&key={API_KEY}");
myReq.Method = WebRequestMethods.Http.Get;
myReq.Timeout = 60000;
myReq.Proxy = null;
myReq.ContentType = "application/json";
Console.WriteLine($"Start call: {url}");
// Try using `HttpClient()` later
//var myReq2 = new HttpClient();
//await myReq2.GetAsync($"https://www.googleapis.com/pagespeedonline/v2/runPagespeed?url={url}&strategy=mobile&key={API_KEY}");
Task<WebResponse> task = Task.Factory.FromAsync(
myReq.BeginGetResponse,
myReq.EndGetResponse,
(object)null);
var result = await task;
return ReadStreamFromResponse(result);
}
private static string ReadStreamFromResponse(WebResponse response)
{
using (Stream responseStream = response.GetResponseStream())
using (StreamReader sr = new StreamReader(responseStream))
{
string jsonResponse = sr.ReadToEnd();
dynamic jsonObj = Newtonsoft.Json.JsonConvert.DeserializeObject(jsonResponse);
var psiResp = new PsiResponse()
{
Url = jsonObj.id,
SpeedScore = jsonObj.ruleGroups.SPEED.score,
UsabilityScore = jsonObj.ruleGroups.USABILITY.score,
NumberResources = jsonObj.pageStats.numberResources,
NumberHosts = jsonObj.pageStats.numberHosts,
TotalRequestBytes = jsonObj.pageStats.totalRequestBytes,
NumberStaticResources = jsonObj.pageStats.numberStaticResources,
HtmlResponseBytes = jsonObj.pageStats.htmlResponseBytes,
CssResponseBytes = jsonObj.pageStats.cssResponseBytes,
ImageResponseBytes = jsonObj.pageStats.imageResponseBytes,
JavascriptResponseBytes = jsonObj.pageStats.javascriptResponseBytes,
OtherResponseBytes = jsonObj.pageStats.otherResponseBytes,
NumberJsResources = jsonObj.pageStats.numberJsResources,
NumberCssResources = jsonObj.pageStats.numberCssResources,
};
return CreateOutputString(psiResp);
}
}
static async Task WriteCharacters(IEnumerable<string> inputs, string outputFilePath)
{
using (StreamWriter fileWriter = new StreamWriter(outputFilePath))
{
await fileWriter.WriteLineAsync(TABLE_HEADER);
foreach (var input in inputs)
{
await fileWriter.WriteLineAsync(input);
}
}
}
private static string CreateOutputString(PsiResponse psiResponse)
{
var stringToWrite = "";
foreach (var prop in psiResponse.GetType().GetProperties())
{
stringToWrite += $"{prop.GetValue(psiResponse, null)},";
}
Console.WriteLine(stringToWrite);
return stringToWrite;
}
ANSWER
Answered 2017-Jul-12 at 14:16Can I run do these multiple WebRequests and write them to a collection (or output file) without using multiple threads and have them all run in parallel, not having to wait for each request to come back before handling the next one?
Sure thing.
Is there a cleaner way to do this with callbacks?
You can always iterate over the input lines and grab a collection of the tasks that are all running.
var resultTask = Task.WhenAll(
inputLines.Select(line => CallPsiForPrimaryStats(line)).ToArray());
This is analogous to using the Q
library for promises in Javascript. With .Net tasks the host machine will spin up as many of the processes as it can in parallel.
The resultTask
will be a collection of the results you can work with, much like your outputCache
.
In the code you added above, the call to .Result
in the loop will be synchronous. Nothing is happening in parallel. Be careful when waiting for all of this, you might run out of memory before it's all passed back! Might be worth streaming this to the file as they return, and having a semaphore or lock prevent them all writing to the stream at once.
Also I think the WebClient class is more idiomatic these days than hand rolling HttpWebRequest
.
If threads or BackgroundWorkers are needed, what's a Clean Code way of doing this?
This is the beauty of the Task
library and .Net's async stack. You shouldn't need to do anything with threads.
It's important to know the difference between an async/await
type call and a synchronous
call. Anywhere you see async
in the method declaration and await
in the body means that the code is freeing up the current synchronisation thread to do other work, like fire off more tasks. When you see .Result
or .Wait()
these are synchronous and are therefore blocking the main synchronisation thread. Which means no ability for easy parallelism.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
No vulnerabilities reported
Find more information at:
Save this library and start creating your kit
See Similar Libraries in
Save this library and start creating your kit
Open Weaver – Develop Applications Faster with Open Source