Hadoop

1admin@XXXXXX:~$ spark-shell --master yarn 21/11/03 15:51:51 WARN Utils: Your hostname, XXXXXX resolves to a loopback address:
2127.0.1.1; using 192.168.29.57 instead (on interface wifi0) 21/11/03 15:51:51 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 21/11/03 15:52:01 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. Spark context Web UI available at http://XX.XX.XX.XX:4040 Spark context available as 'sc' (master = yarn, app id = application_1635934709971_0001). Spark session available as 'spark'. Welcome to
3      ____              __
4     / __/__  ___ _____/ /__
5    _\ \/ _ \/ _ `/ __/  '_/    /___/ .__/\_,_/_/ /_/\_\   version 2.4.5
6      /_/
7
8Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java
91.8.0_301) Type in expressions to have them evaluated. Type :help for more information.
10
11scala>
12
13scala> 21/11/03 15:52:35 ERROR YarnClientSchedulerBackend: YARN application has exited unexpectedly with state FAILED! Check the YARN application logs for more details. 21/11/03 15:52:35 ERROR YarnClientSchedulerBackend: Diagnostics message: Uncaught exception: org.apache.spark.SparkException: Exception thrown in awaitResult:
14        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)
15        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
16        at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
17        at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
18        at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:515)
19        at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:307)
20        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply$mcV$sp(ApplicationMaster.scala:245)
21        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
22        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
23        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:780)
24        at java.security.AccessController.doPrivileged(Native Method)
25        at javax.security.auth.Subject.doAs(Subject.java:422)
26        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
27        at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:779)
28        at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:244)
29        at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:804)
30        at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:834)
31        at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala) Caused by: java.io.IOException: Failed to connect to /192.168.29.57:33333
32        at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)
33        at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
34        at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198)
35        at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
36        at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
37        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
38        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
39        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
40        at java.lang.Thread.run(Thread.java:748) Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.29.57:33333 Caused by: java.net.ConnectException: Connection refused
41        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
42        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
43        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
44        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
45        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:688)
46        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
47        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
48        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
49        at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
50        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
51        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
52        at java.lang.Thread.run(Thread.java:748)
53
5421/11/03 15:52:35 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
55

1admin@XXXXXX:~$ spark-shell --master yarn 21/11/03 15:51:51 WARN Utils: Your hostname, XXXXXX resolves to a loopback address:
2127.0.1.1; using 192.168.29.57 instead (on interface wifi0) 21/11/03 15:51:51 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 21/11/03 15:52:01 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. Spark context Web UI available at http://XX.XX.XX.XX:4040 Spark context available as 'sc' (master = yarn, app id = application_1635934709971_0001). Spark session available as 'spark'. Welcome to
3      ____              __
4     / __/__  ___ _____/ /__
5    _\ \/ _ \/ _ `/ __/  '_/    /___/ .__/\_,_/_/ /_/\_\   version 2.4.5
6      /_/
7
8Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java
91.8.0_301) Type in expressions to have them evaluated. Type :help for more information.
10
11scala>
12
13scala> 21/11/03 15:52:35 ERROR YarnClientSchedulerBackend: YARN application has exited unexpectedly with state FAILED! Check the YARN application logs for more details. 21/11/03 15:52:35 ERROR YarnClientSchedulerBackend: Diagnostics message: Uncaught exception: org.apache.spark.SparkException: Exception thrown in awaitResult:
14        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)
15        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
16        at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
17        at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
18        at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:515)
19        at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:307)
20        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply$mcV$sp(ApplicationMaster.scala:245)
21        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
22        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
23        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:780)
24        at java.security.AccessController.doPrivileged(Native Method)
25        at javax.security.auth.Subject.doAs(Subject.java:422)
26        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
27        at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:779)
28        at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:244)
29        at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:804)
30        at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:834)
31        at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala) Caused by: java.io.IOException: Failed to connect to /192.168.29.57:33333
32        at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)
33        at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
34        at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198)
35        at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
36        at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
37        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
38        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
39        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
40        at java.lang.Thread.run(Thread.java:748) Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.29.57:33333 Caused by: java.net.ConnectException: Connection refused
41        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
42        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
43        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
44        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
45        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:688)
46        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
47        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
48        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
49        at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
50        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
51        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
52        at java.lang.Thread.run(Thread.java:748)
53
5421/11/03 15:52:35 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
55spark.driver.memory              512m
56spark.yarn.am.memory             512m
57spark.executor.memory            512m
58spark.eventLog.enabled true
59spark.eventLog.dir file:////home/admin/spark_event_temp
60spark.history.fs.logDirectory hdfs://localhost:9000/spark-logs
61spark.history.fs.update.interval 10s
62spark.history.ui.port 18080
63spark.sql.warehouse.dir=file:////home/admin/spark_warehouse
64spark.shuffle.service.port              7337
65spark.ui.port                           4040
66spark.blockManager.port                 31111
67spark.driver.blockManager.port          32222
68spark.driver.port                       33333
69

1spark = SparkSession.builder.getOrCreate()
2file = "s3://bucket/file.csv"
3
4c = spark.read\
5    .csv(file)\
6    .count()
7
8print(c)
9

1spark = SparkSession.builder.getOrCreate()
2file = "s3://bucket/file.csv"
3
4c = spark.read\
5    .csv(file)\
6    .count()
7
8print(c)
9py4j.protocol.Py4JJavaError: An error occurred while calling o26.csv.
10: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
11

1spark = SparkSession.builder.getOrCreate()
2file = "s3://bucket/file.csv"
3
4c = spark.read\
5    .csv(file)\
6    .count()
7
8print(c)
9py4j.protocol.Py4JJavaError: An error occurred while calling o26.csv.
10: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
11import os
12os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages com.amazonaws:aws-java-sdk:1.7.4,org.apache.hadoop:hadoop-aws:2.7.3 pyspark-shell'
13

121/12/07 01:25:16 ERROR org.apache.spark.scheduler.AsyncEventQueue: Listener AppStatusListener threw an exception
2java.lang.NumberFormatException: For input string: "null"
3        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
4        at java.lang.Integer.parseInt(Integer.java:580)
5        at java.lang.Integer.parseInt(Integer.java:615)
6        at scala.collection.immutable.StringLike.toInt(StringLike.scala:304)
7        at scala.collection.immutable.StringLike.toInt$(StringLike.scala:304)
8        at scala.collection.immutable.StringOps.toInt(StringOps.scala:33)
9        at org.apache.spark.util.Utils$.parseHostPort(Utils.scala:1126)
10        at org.apache.spark.status.ProcessSummaryWrapper.<init>(storeTypes.scala:527)
11        at org.apache.spark.status.LiveMiscellaneousProcess.doUpdate(LiveEntity.scala:924)
12        at org.apache.spark.status.LiveEntity.write(LiveEntity.scala:50)
13        at org.apache.spark.status.AppStatusListener.update(AppStatusListener.scala:1213)
14        at org.apache.spark.status.AppStatusListener.onMiscellaneousProcessAdded(AppStatusListener.scala:1427)
15        at org.apache.spark.status.AppStatusListener.onOtherEvent(AppStatusListener.scala:113)
16        at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:100)
17        at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
18        at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
19        at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
20        at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
21        at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
22        at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
23        at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
24        at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
25        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
26        at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
27        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
28        at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1404)
29        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)
30

121/12/07 01:25:16 ERROR org.apache.spark.scheduler.AsyncEventQueue: Listener AppStatusListener threw an exception
2java.lang.NumberFormatException: For input string: "null"
3        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
4        at java.lang.Integer.parseInt(Integer.java:580)
5        at java.lang.Integer.parseInt(Integer.java:615)
6        at scala.collection.immutable.StringLike.toInt(StringLike.scala:304)
7        at scala.collection.immutable.StringLike.toInt$(StringLike.scala:304)
8        at scala.collection.immutable.StringOps.toInt(StringOps.scala:33)
9        at org.apache.spark.util.Utils$.parseHostPort(Utils.scala:1126)
10        at org.apache.spark.status.ProcessSummaryWrapper.<init>(storeTypes.scala:527)
11        at org.apache.spark.status.LiveMiscellaneousProcess.doUpdate(LiveEntity.scala:924)
12        at org.apache.spark.status.LiveEntity.write(LiveEntity.scala:50)
13        at org.apache.spark.status.AppStatusListener.update(AppStatusListener.scala:1213)
14        at org.apache.spark.status.AppStatusListener.onMiscellaneousProcessAdded(AppStatusListener.scala:1427)
15        at org.apache.spark.status.AppStatusListener.onOtherEvent(AppStatusListener.scala:113)
16        at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:100)
17        at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
18        at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
19        at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
20        at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
21        at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
22        at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
23        at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
24        at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
25        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
26        at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
27        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
28        at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1404)
29        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)
3021/12/07 01:32:15 WARN org.apache.spark.deploy.yarn.YarnAllocator: Container from a bad node: container_1638782400702_0003_01_000001 on host: monsoon-test1-w-2.us-central1-c.c.monsoon-credittech.internal. Exit status: 1. Diagnostics: [2021-12-07 
3101:32:13.672]Exception from container-launch.
32Container id: container_1638782400702_0003_01_000001
33Exit code: 1
34[2021-12-07 01:32:13.717]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
35Last 4096 bytes of prelaunch.err :
36Last 4096 bytes of stderr :
37ltChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
38        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
39        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
40        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
41        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
42        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
43        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
44        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
45        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
46        at java.lang.Thread.run(Thread.java:748)
4721/12/07 01:31:43 ERROR org.apache.spark.executor.YarnCoarseGrainedExecutorBackend: Executor self-exiting due to : Driver monsoon-test1-m.us-central1-c.c.monsoon-credittech.internal:44367 disassociated! Shutting down.
4821/12/07 01:32:13 WARN org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '$anon$2' timeout, java.util.concurrent.TimeoutException
49java.util.concurrent.TimeoutException
50        at java.util.concurrent.FutureTask.get(FutureTask.java:205)
51        at org.apache.hadoop.util.ShutdownHookManager.executeShutdown(ShutdownHookManager.java:124)
52        at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:95)
5321/12/07 01:32:13 ERROR org.apache.spark.util.Utils: Uncaught exception in thread shutdown-hook-0
54java.lang.InterruptedException
55        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
56        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
57        at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475)
58        at java.util.concurrent.Executors$DelegatedExecutorService.awaitTermination(Executors.java:675)
59        at org.apache.spark.rpc.netty.MessageLoop.stop(MessageLoop.scala:60)
60        at org.apache.spark.rpc.netty.Dispatcher.$anonfun$stop$1(Dispatcher.scala:197)
61        at org.apache.spark.rpc.netty.Dispatcher.$anonfun$stop$1$adapted(Dispatcher.scala:194)
62        at scala.collection.Iterator.foreach(Iterator.scala:943)
63        at scala.collection.Iterator.foreach$(Iterator.scala:943)
64        at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
65        at scala.collection.IterableLike.foreach(IterableLike.scala:74)
66        at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
67        at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
68        at org.apache.spark.rpc.netty.Dispatcher.stop(Dispatcher.scala:194)
69        at org.apache.spark.rpc.netty.NettyRpcEnv.cleanup(NettyRpcEnv.scala:331)
70        at org.apache.spark.rpc.netty.NettyRpcEnv.shutdown(NettyRpcEnv.scala:309)
71        at org.apache.spark.SparkEnv.stop(SparkEnv.scala:96)
72        at org.apache.spark.executor.Executor.stop(Executor.scala:335)
73        at org.apache.spark.executor.Executor.$anonfun$new$2(Executor.scala:76)
74        at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:214)
75        at org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$2(ShutdownHookManager.scala:188)
76        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
77        at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1996)
78        at org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$1(ShutdownHookManager.scala:188)
79        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
80        at scala.util.Try$.apply(Try.scala:213)
81        at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
82        at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
83        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
84        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
85        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
86        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
87        at java.lang.Thread.run(Thread.java:748)
88

1search =  SearchEngine(db_file_dir = "/tmp/db")
2conn = sqlite3.connect("/tmp/db/simple_db.sqlite")
3pdf_ = pd.read_sql_query('''select  zipcode, lat, lng, 
4                        bounds_west, bounds_east, bounds_north, bounds_south from 
5                        simple_zipcode''',conn)
6brd_pdf = spark.sparkContext.broadcast(pdf_) 
7conn.close()
8
9
10@udf('string')
11def get_zip_b(lat, lng):
12    pdf = brd_pdf.value 
13    out = pdf[(np.array(pdf["bounds_north"]) >= lat) & 
14              (np.array(pdf["bounds_south"]) <= lat) & 
15              (np.array(pdf['bounds_west']) <= lng) & 
16              (np.array(pdf['bounds_east']) >= lng) ]
17    if len(out):
18        min_index = np.argmin( (np.array(out["lat"]) - lat)**2 + (np.array(out["lng"]) - lng)**2)
19        zip_ = str(out["zipcode"].iloc[min_index])
20    else:
21        zip_ = 'bad'
22    return zip_
23
24df = df.withColumn('zipcode', get_zip_b(col("latitude"),col("longitude")))
25

1search =  SearchEngine(db_file_dir = "/tmp/db")
2conn = sqlite3.connect("/tmp/db/simple_db.sqlite")
3pdf_ = pd.read_sql_query('''select  zipcode, lat, lng, 
4                        bounds_west, bounds_east, bounds_north, bounds_south from 
5                        simple_zipcode''',conn)
6brd_pdf = spark.sparkContext.broadcast(pdf_) 
7conn.close()
8
9
10@udf('string')
11def get_zip_b(lat, lng):
12    pdf = brd_pdf.value 
13    out = pdf[(np.array(pdf["bounds_north"]) >= lat) & 
14              (np.array(pdf["bounds_south"]) <= lat) & 
15              (np.array(pdf['bounds_west']) <= lng) & 
16              (np.array(pdf['bounds_east']) >= lng) ]
17    if len(out):
18        min_index = np.argmin( (np.array(out["lat"]) - lat)**2 + (np.array(out["lng"]) - lng)**2)
19        zip_ = str(out["zipcode"].iloc[min_index])
20    else:
21        zip_ = 'bad'
22    return zip_
23
24df = df.withColumn('zipcode', get_zip_b(col("latitude"),col("longitude")))
2521/08/02 06:18:19 WARN TaskSetManager: Lost task 12.0 in stage 7.0 (TID 1814, ip-10-22-17-94.pclc0.merkle.local, executor 6): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
26  File "/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/worker.py", line 605, in main
27    process()
28  File "/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/worker.py", line 597, in process
29    serializer.dump_stream(out_iter, outfile)
30  File "/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/serializers.py", line 223, in dump_stream
31    self.serializer.dump_stream(self._batched(iterator), stream)
32  File "/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/serializers.py", line 141, in dump_stream
33    for obj in iterator:
34  File "/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/serializers.py", line 212, in _batched
35    for item in iterator:
36  File "/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/worker.py", line 450, in mapper
37    result = tuple(f(*[a[o] for o in arg_offsets]) for (arg_offsets, f) in udfs)
38  File "/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/worker.py", line 450, in <genexpr>
39    result = tuple(f(*[a[o] for o in arg_offsets]) for (arg_offsets, f) in udfs)
40  File "/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/worker.py", line 90, in <lambda>
41    return lambda *a: f(*a)
42  File "/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/util.py", line 121, in wrapper
43    return f(*args, **kwargs)
44  File "/mnt/var/lib/hadoop/steps/s-1IBFS0SYWA19Z/Mobile_ID_process_center.py", line 102, in get_zip_b
45  File "/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/broadcast.py", line 146, in value
46    self._value = self.load_from_path(self._path)
47  File "/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/broadcast.py", line 123, in load_from_path
48    return self.load(f)
49  File "/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/broadcast.py", line 129, in load
50    return pickle.load(file)
51AttributeError: Can't get attribute 'new_block' on <module 'pandas.core.internals.blocks' from '/mnt/miniconda/lib/python3.9/site-packages/pandas/core/internals/blocks.py'>
52

1    "classification": "spark-env",
2    "properties": {
3        "conda": "/home/hadoop/conda/bin"
4    }
5

1    "classification": "spark-env",
2    "properties": {
3        "conda": "/home/hadoop/conda/bin"
4    }
5[hadoop@ip-172-30-5-150 ~]$ PATH=/home/hadoop/conda/bin:$PATH
6[hadoop@ip-172-30-5-150 ~]$ conda
7usage: conda [-h] [-V] command ...
8
9conda is a tool for managing and deploying applications, environments and packages.
10

1    "classification": "spark-env",
2    "properties": {
3        "conda": "/home/hadoop/conda/bin"
4    }
5[hadoop@ip-172-30-5-150 ~]$ PATH=/home/hadoop/conda/bin:$PATH
6[hadoop@ip-172-30-5-150 ~]$ conda
7usage: conda [-h] [-V] command ...
8
9conda is a tool for managing and deploying applications, environments and packages.
10#!/usr/bin/env bash
11
12
13# Install conda
14wget https://repo.continuum.io/miniconda/Miniconda3-4.2.12-Linux-x86_64.sh -O /home/hadoop/miniconda.sh \
15    && /bin/bash ~/miniconda.sh -b -p $HOME/conda
16
17
18conda config --set always_yes yes --set changeps1 no
19conda install conda=4.2.13
20conda config -f --add channels conda-forge
21rm ~/miniconda.sh
22echo bootstrap_conda.sh completed. PATH now: $PATH
23export PYSPARK_PYTHON="/home/hadoop/conda/bin/python3.5"
24
25echo -e '\nexport PATH=$HOME/conda/bin:$PATH' >> $HOME/.bashrc && source $HOME/.bashrc
26
27
28conda create -n zoo python=3.7 # "zoo" is conda environment name, you can use any name you like.
29conda activate zoo
30sudo pip3 install tensorflow
31sudo pip3 install boto3
32sudo pip3 install botocore
33sudo pip3 install numpy
34sudo pip3 install pandas
35sudo pip3 install scipy
36sudo pip3 install s3fs
37sudo pip3 install matplotlib
38sudo pip3 install -U tqdm
39sudo pip3 install -U scikit-learn
40sudo pip3 install -U scikit-multilearn
41sudo pip3 install xlutils
42sudo pip3 install natsort
43sudo pip3 install pydot
44sudo pip3 install python-pydot
45sudo pip3 install python-pydot-ng
46sudo pip3 install pydotplus
47sudo pip3 install h5py
48sudo pip3 install graphviz
49sudo pip3 install recmetrics
50sudo pip3 install openpyxl
51sudo pip3 install xlrd
52sudo pip3 install xlwt
53sudo pip3 install tensorflow.io
54sudo pip3 install Cython
55sudo pip3 install ray
56sudo pip3 install zoo
57sudo pip3 install analytics-zoo
58sudo pip3 install analytics-zoo[ray]
59#sudo /usr/bin/pip-3.6 install -U imbalanced-learn
60
61
62

1import os
2os.listdir('./')
3

1import os
2os.listdir('./')
3gcloud dataproc jobs submit pyspark ./testing_dep.py --cluster=pyspark-monsoon --region=us-central1
4

1import os
2os.listdir('./')
3gcloud dataproc jobs submit pyspark ./testing_dep.py --cluster=pyspark-monsoon --region=us-central1
4gcloud dataproc jobs submit pyspark ./testing_dep.py --cluster=pyspark-monsoon --region=us-central1 --properties=spark.submit.deployMode=cluster
5

1import os
2os.listdir('./')
3gcloud dataproc jobs submit pyspark ./testing_dep.py --cluster=pyspark-monsoon --region=us-central1
4gcloud dataproc jobs submit pyspark ./testing_dep.py --cluster=pyspark-monsoon --region=us-central1 --properties=spark.submit.deployMode=cluster
5Job [417443357bcd43f99ee3dc60f4e3bfea] submitted.
6Waiting for job output...
722/01/12 05:32:20 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at monsoon-testing-m/10.128.15.236:8032
822/01/12 05:32:20 INFO org.apache.hadoop.yarn.client.AHSProxy: Connecting to Application History server at monsoon-testing-m/10.128.15.236:10200
922/01/12 05:32:22 INFO org.apache.hadoop.conf.Configuration: resource-types.xml not found
1022/01/12 05:32:22 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find 'resource-types.xml'.
1122/01/12 05:32:24 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1641965080466_0001
1222/01/12 05:32:42 ERROR org.apache.spark.deploy.yarn.Client: Application diagnostics message: Application application_1641965080466_0001 failed 2 times due to AM Container for appattempt_1641965080466_0001_000002 exited with  exitCode: 13
13Failing this attempt.Diagnostics: [2022-01-12 05:32:42.154]Exception from container-launch.
14Container id: container_1641965080466_0001_02_000001
15Exit code: 13
16
17[2022-01-12 05:32:42.203]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
18Last 4096 bytes of prelaunch.err :
19Last 4096 bytes of stderr :
2022/01/12 05:32:40 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: Uncaught exception: 
21java.lang.IllegalStateException: User did not initialize spark context!
22    at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:520)
23    at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:268)
24    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:899)
25    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:898)
26    at java.security.AccessController.doPrivileged(Native Method)
27    at javax.security.auth.Subject.doAs(Subject.java:422)
28    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
29    at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:898)
30    at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
31
32
33[2022-01-12 05:32:42.203]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
34Last 4096 bytes of prelaunch.err :
35Last 4096 bytes of stderr :
3622/01/12 05:32:40 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: Uncaught exception: 
37java.lang.IllegalStateException: User did not initialize spark context!
38    at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:520)
39    at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:268)
40    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:899)
41    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:898)
42    at java.security.AccessController.doPrivileged(Native Method)
43    at javax.security.auth.Subject.doAs(Subject.java:422)
44    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
45    at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:898)
46    at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
47
48
49For more detailed output, check the application tracking page: http://monsoon-testing-m:8188/applicationhistory/app/application_1641965080466_0001 Then click on links to logs of each attempt.
50. Failing the application.
51Exception in thread "main" org.apache.spark.SparkException: Application application_1641965080466_0001 finished with failed status
52    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1242)
53    at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1634)
54    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
55    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
56    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
57    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
58    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
59    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
60    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
61ERROR: (gcloud.dataproc.jobs.submit.pyspark) Job [417443357bcd43f99ee3dc60f4e3bfea] failed with error:
62Google Cloud Dataproc Agent reports job failure. If logs are available, they can be found at:
63https://console.cloud.google.com/dataproc/jobs/417443357bcd43f99ee3dc60f4e3bfea?project=monsoon-credittech&region=us-central1
64gcloud dataproc jobs wait '417443357bcd43f99ee3dc60f4e3bfea' --region 'us-central1' --project 'monsoon-credittech'
65https://console.cloud.google.com/storage/browser/monsoon-credittech.appspot.com/google-cloud-dataproc-metainfo/64632294-3e9b-4c55-af8a-075fc7d6f412/jobs/417443357bcd43f99ee3dc60f4e3bfea/
66gs://monsoon-credittech.appspot.com/google-cloud-dataproc-metainfo/64632294-3e9b-4c55-af8a-075fc7d6f412/jobs/417443357bcd43f99ee3dc60f4e3bfea/driveroutput
67
68
69
70

1import sys
2import time
3from datetime import datetime
4
5from pyspark.sql import SparkSession
6
7start_time = datetime.utcnow()
8
9spark = SparkSession.builder.appName("check_confs").getOrCreate()
10
11all_conf = spark.sparkContext.getConf().getAll()
12print("\n\n=====\nExecuting at {}".format(datetime.utcnow()))
13print(all_conf)
14print("\n\n======================\n\n\n")
15incoming_args = sys.argv
16if len(incoming_args) > 1:
17    sleep_time = int(incoming_args[1])
18    print("Sleep time is {} seconds".format(sleep_time))
19    if sleep_time > 0:
20        time.sleep(sleep_time)
21
22end_time = datetime.utcnow()
23time_taken = (end_time - start_time).total_seconds()
24print("Script execution completed in {} seconds".format(time_taken))
25

1import sys
2import time
3from datetime import datetime
4
5from pyspark.sql import SparkSession
6
7start_time = datetime.utcnow()
8
9spark = SparkSession.builder.appName("check_confs").getOrCreate()
10
11all_conf = spark.sparkContext.getConf().getAll()
12print("\n\n=====\nExecuting at {}".format(datetime.utcnow()))
13print(all_conf)
14print("\n\n======================\n\n\n")
15incoming_args = sys.argv
16if len(incoming_args) > 1:
17    sleep_time = int(incoming_args[1])
18    print("Sleep time is {} seconds".format(sleep_time))
19    if sleep_time > 0:
20        time.sleep(sleep_time)
21
22end_time = datetime.utcnow()
23time_taken = (end_time - start_time).total_seconds()
24print("Script execution completed in {} seconds".format(time_taken))
25{
26        'placement': {
27            'cluster_name': dataproc_cluster
28        },
29        'pyspark_job': {
30            'main_python_file_uri': "gs://" + compute_storage + "/" + job_file,
31            'args': trigger_params,
32            "properties": {
33                "spark.submit.deployMode": "cluster",
34                "spark.executor.memory": "3155m",
35                "spark.scheduler.mode": "FAIR",
36            }
37        }
38    }
39

1import sys
2import time
3from datetime import datetime
4
5from pyspark.sql import SparkSession
6
7start_time = datetime.utcnow()
8
9spark = SparkSession.builder.appName("check_confs").getOrCreate()
10
11all_conf = spark.sparkContext.getConf().getAll()
12print("\n\n=====\nExecuting at {}".format(datetime.utcnow()))
13print(all_conf)
14print("\n\n======================\n\n\n")
15incoming_args = sys.argv
16if len(incoming_args) > 1:
17    sleep_time = int(incoming_args[1])
18    print("Sleep time is {} seconds".format(sleep_time))
19    if sleep_time > 0:
20        time.sleep(sleep_time)
21
22end_time = datetime.utcnow()
23time_taken = (end_time - start_time).total_seconds()
24print("Script execution completed in {} seconds".format(time_taken))
25{
26        'placement': {
27            'cluster_name': dataproc_cluster
28        },
29        'pyspark_job': {
30            'main_python_file_uri': "gs://" + compute_storage + "/" + job_file,
31            'args': trigger_params,
32            "properties": {
33                "spark.submit.deployMode": "cluster",
34                "spark.executor.memory": "3155m",
35                "spark.scheduler.mode": "FAIR",
36            }
37        }
38    }
3921/12/07 19:11:27 INFO org.sparkproject.jetty.util.log: Logging initialized @3350ms to org.sparkproject.jetty.util.log.Slf4jLog
4021/12/07 19:11:27 INFO org.sparkproject.jetty.server.Server: jetty-9.4.40.v20210413; built: 2021-04-13T20:42:42.668Z; git: b881a572662e1943a14ae12e7e1207989f218b74; jvm 1.8.0_292-b10
4121/12/07 19:11:27 INFO org.sparkproject.jetty.server.Server: Started @3467ms
4221/12/07 19:11:27 INFO org.sparkproject.jetty.server.AbstractConnector: Started ServerConnector@18528bea{HTTP/1.1, (http/1.1)}{0.0.0.0:40389}
4321/12/07 19:11:28 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at ******-m/0.0.0.5:8032
4421/12/07 19:11:28 INFO org.apache.hadoop.yarn.client.AHSProxy: Connecting to Application History server at ******-m/0.0.0.5:10200
4521/12/07 19:11:29 INFO org.apache.hadoop.conf.Configuration: resource-types.xml not found
4621/12/07 19:11:29 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find 'resource-types.xml'.
4721/12/07 19:11:30 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1638554180947_0014
4821/12/07 19:11:31 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at ******-m/0.0.0.5:8030
4921/12/07 19:11:33 INFO com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl: Ignoring exception of type GoogleJsonResponseException; verified object already exists with desired state.
50
51
52=====
53Executing at 2021-12-07 19:11:35.100277
54[....... ('spark.yarn.historyServer.address', '****-m:18080'), ('spark.ui.proxyBase', '/proxy/application_1638554180947_0014'), ('spark.driver.appUIAddress', 'http://***-m.c.***-123456.internal:40389'), ('spark.sql.cbo.enabled', 'true')]
55
56
57======================
58
59
60
61Sleep time is 1 seconds
62Script execution completed in 9.411261 seconds
6321/12/07 19:11:36 INFO org.sparkproject.jetty.server.AbstractConnector: Stopped Spark@18528bea{HTTP/1.1, (http/1.1)}{0.0.0.0:0}
64
65

1import sys
2import time
3from datetime import datetime
4
5from pyspark.sql import SparkSession
6
7start_time = datetime.utcnow()
8
9spark = SparkSession.builder.appName("check_confs").getOrCreate()
10
11all_conf = spark.sparkContext.getConf().getAll()
12print("\n\n=====\nExecuting at {}".format(datetime.utcnow()))
13print(all_conf)
14print("\n\n======================\n\n\n")
15incoming_args = sys.argv
16if len(incoming_args) > 1:
17    sleep_time = int(incoming_args[1])
18    print("Sleep time is {} seconds".format(sleep_time))
19    if sleep_time > 0:
20        time.sleep(sleep_time)
21
22end_time = datetime.utcnow()
23time_taken = (end_time - start_time).total_seconds()
24print("Script execution completed in {} seconds".format(time_taken))
25{
26        'placement': {
27            'cluster_name': dataproc_cluster
28        },
29        'pyspark_job': {
30            'main_python_file_uri': "gs://" + compute_storage + "/" + job_file,
31            'args': trigger_params,
32            "properties": {
33                "spark.submit.deployMode": "cluster",
34                "spark.executor.memory": "3155m",
35                "spark.scheduler.mode": "FAIR",
36            }
37        }
38    }
3921/12/07 19:11:27 INFO org.sparkproject.jetty.util.log: Logging initialized @3350ms to org.sparkproject.jetty.util.log.Slf4jLog
4021/12/07 19:11:27 INFO org.sparkproject.jetty.server.Server: jetty-9.4.40.v20210413; built: 2021-04-13T20:42:42.668Z; git: b881a572662e1943a14ae12e7e1207989f218b74; jvm 1.8.0_292-b10
4121/12/07 19:11:27 INFO org.sparkproject.jetty.server.Server: Started @3467ms
4221/12/07 19:11:27 INFO org.sparkproject.jetty.server.AbstractConnector: Started ServerConnector@18528bea{HTTP/1.1, (http/1.1)}{0.0.0.0:40389}
4321/12/07 19:11:28 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at ******-m/0.0.0.5:8032
4421/12/07 19:11:28 INFO org.apache.hadoop.yarn.client.AHSProxy: Connecting to Application History server at ******-m/0.0.0.5:10200
4521/12/07 19:11:29 INFO org.apache.hadoop.conf.Configuration: resource-types.xml not found
4621/12/07 19:11:29 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find 'resource-types.xml'.
4721/12/07 19:11:30 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1638554180947_0014
4821/12/07 19:11:31 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at ******-m/0.0.0.5:8030
4921/12/07 19:11:33 INFO com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl: Ignoring exception of type GoogleJsonResponseException; verified object already exists with desired state.
50
51
52=====
53Executing at 2021-12-07 19:11:35.100277
54[....... ('spark.yarn.historyServer.address', '****-m:18080'), ('spark.ui.proxyBase', '/proxy/application_1638554180947_0014'), ('spark.driver.appUIAddress', 'http://***-m.c.***-123456.internal:40389'), ('spark.sql.cbo.enabled', 'true')]
55
56
57======================
58
59
60
61Sleep time is 1 seconds
62Script execution completed in 9.411261 seconds
6321/12/07 19:11:36 INFO org.sparkproject.jetty.server.AbstractConnector: Stopped Spark@18528bea{HTTP/1.1, (http/1.1)}{0.0.0.0:0}
64
6521/12/07 19:09:04 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at ******-m/0.0.0.5:8032
6621/12/07 19:09:04 INFO org.apache.hadoop.yarn.client.AHSProxy: Connecting to Application History server at ******-m/0.0.0.5:8032
6721/12/07 19:09:05 INFO org.apache.hadoop.conf.Configuration: resource-types.xml not found
6821/12/07 19:09:05 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find 'resource-types.xml'.
6921/12/07 19:09:06 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1638554180947_0013
70