JavaUtils | Lots of utils for Java dev | Reflection library

It just seems as output_dir variable contains incorrect path:

standalone hive doesnt support avro. we need to install full hadoop plus hive version and start only hive metastore to fix it

I think there is a thread-safety problem. Specifically:

1. a LinkedList is not thread-safe, and
2. you are using the LinkedList object in rt.getTasks().size() without any synchronization.

This is sufficient to cause size() to return a stale value under some circumstances.

If you are going to rely on the semantics of volatile you need to do a proper analysis of the happens-before relationships for each write / read sequence that matters to thread safety. It is tricky.

My advice would be:

1. Don't use volatile. Use synchronized and/or an existing thread-safe data structure instead ... if you need to reinvent the wheel.

2. Don't reinvent the wheel. You could replace your thread pool with a single call Executors.singleThreadExecutor; see javadoc.

Your code looks good to me, so not sure where is actual problem. But rather then modifying stub class, I have done it while invoking the service method. Your code should look something like below and you could remove your code modifications to generated stub class, and it should work.

I used this code , and I am not getting any error. Output was Google

Maybe you will have to download latest chrome driver , chrome browser version and Selenium.

My config :

Chrome: Version 67.0.3396.99 (Official Build) (64-bit)
Selenium 3.12.0 (you can go for 3.13.0 also)
Chrome driver : version 2.40

Download links : Chrome_driver
Selenium 3.13.0 Browser you can just upgrade.

Taking straight from the docs,

• The first step in GC tuning is to collect statistics on how frequently garbage collection occurs and the amount of time spent GC. This can be done by adding -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps to the Java options.
• The goal of GC tuning in Spark is to ensure that only long-lived RDDs are stored in the Old generation and that the Young generation is sufficiently sized to store short-lived objects. This will help avoid full GCs to collect temporary objects created during task execution.
• Check if there are too many garbage collections by collecting GC stats. If a full GC is invoked multiple times for before a task completes, it means that there isn’t enough memory available for executing tasks.
• If there are too many minor collections but not many major GCs, allocating more memory for Eden would help. You can set the size of the Eden to be an over-estimate of how much memory each task will need. If the size of Eden is determined to be E, then you can set the size of the Young generation using the option -Xmn=4/3*E. (The scaling up by 4/3 is to account for space used by survivor regions as well.)
• In the GC stats that are printed, if the OldGen is close to being full, reduce the amount of memory used for caching by lowering spark.memory.fraction; it is better to cache fewer objects than to slow down task execution. Alternatively, consider decreasing the size of the Young generation. This means lowering -Xmn if you’ve set it as above. If not, try changing the value of the JVM’s NewRatio parameter. Many JVMs default this to 2, meaning that the Old generation occupies 2/3 of the heap. It should be large enough such that this fraction exceeds spark.memory.fraction.
• Try the G1GC garbage collector with -XX:+UseG1GC. It can improve performance in some situations where garbage collection is a bottleneck. (This helped me)

Some more parameters that helped me were,

• -XX:ConcGCThreads=20
• -XX:InitiatingHeapOcuupancyPercent=35

All GC tuning flags for executors can be specified by setting spark.executor.extraJavaOptions in a job’s configuration.

Check this out for further details.

EDIT:

In you spark-defaults.conf write,

spark.executor.JavaOptions -XX:+UseG1GC

spark.executor.extraJavaOptions -XX:ConcGCThreads=20 -XX:InitiatingHeapOcuupancyPercent=35

I tried implement the client using JAX-RPC however I got the same error. Tried modify the way the client was processing the response, but I was getting some other errors. So I gave up use third part SOAP clients and implemented a http client sending and receiving the data as String: