sparkstreaming | 封装sparkstreaming动态调节batch time ； rocket

I want to change the Kafka topic destination to save the data depending on the value of the data in SparkStreaming. Is it possible to do so again? When I tried the following code, it only executes the first one, but does not execute the lower process.

:)

I've ended myself in a (strange) situation where, briefly, I don't want to consume any new record from Kafka, so pause the sparkStreaming consumption (InputDStream[ConsumerRecord]) for all partitions in the topic, do some operations and finally, resume consuming records.

First of all... is this possible?

I've been trying sth like this:

I created a DummySource that reads lines from a file and convert it to TaxiRide objects. The problem is that there are fields that correspond to org.joda.time.DateTime where I use org.joda.time.format.{DateTimeFormat, DateTimeFormatter} and SparkStreaming cannot serialize those fields.

How do I make SparkStreaming serialize them? My code is below together with the error.

I am now trying to put SparkStreaming and Kafka work together on Ubantu. But here comes the question.

I can make sure Kafka's working properly.

On the first terminal:

I am new to Kafka and trying to implement Kafka consumer logic in spark2 and when I run all my code in the shell and start the streaming it shows nothing.

I have viewed many posts in StackOverflow but nothing helped me. I have even downloaded all the dependency jars from maven and tried to run but it still shows nothing.

Spark Version: 2.2.0 Scala version 2.11.8 jars I downloaded are kafka-clients-2.2.0.jar and spark-streaming-kafka-0-10_2.11-2.2.0.jar

but it still I face the same issue.

Please find the below code snippet

I just need to know if a global public class variable, used in a SparkStreaming process will be considered as a broadcasted variable.

For now, I succeeded to use a pre-setted variable "inventory" into a JavaDStream transformation.

I am going through Spark Structured Streaming and encountered a problem.

In StreamingContext, DStreams, we can define a batch interval as follows :

I have written a spark structured streaming app (I'm using Scala with sbt) and now I have to create an integration test. Unfortunately I'm running into a dependency problem I can't solve. I'm using scala with sbt.

My dependency looks like the following

I am using SparkStreaming for reading data from a topic. I am facing an exception in it.

java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord Serialization stack: - object not serializable (class: org.apache.kafka.clients.consumer.ConsumerRecord, value: ConsumerRecord(topic = rawEventTopic, partition = 0, offset = 14098, CreateTime = 1556113016951, serialized key size = -1, serialized value size = 2916, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = {"id":null,"message":null,"eventDate":"","group":null,"category":"AD","userName":null,"inboundDataSource":"AD","source":"192.168.1.14","destination":"192.168.1.15","bytesSent":"200KB","rawData":"{username: vinit}","account_name":null,"security_id":null,"account_domain":null,"logon_id":null,"process_id":null,"process_information":null,"process_name":null,"target_server_name":null,"source_network_address":null,"logon_process":null,"authentication_Package":null,"network_address":null,"failure_reason":null,"workstation_name":null,"target_server":null,"network_information":null,"object_type":null,"object_name":null,"source_port":null,"logon_type":null,"group_name":null,"source_dra":null,"destination_dra":null,"group_admin":null,"sam_account_name":null,"new_logon":null,"destination_address":null,"destination_port":null,"source_address":null,"logon_account":null,"sub_status":null,"eventdate":null,"time_taken":null,"s_computername":null,"cs_method":null,"cs_uri_stem":null,"cs_uri_query":null,"c_ip":null,"s_ip":null,"s_supplier_name":null,"s_sitename":null,"cs_username":null,"cs_auth_group":null,"cs_categories":null,"s_action":null,"cs_host":null,"cs_uri":null,"cs_uri_scheme":null,"cs_uri_port":null,"cs_uri_path":null,"cs_uri_extension":null,"cs_referer":null,"cs_user_agent":null,"cs_bytes":null,"sc_status":null,"sc_bytes":null,"sc_filter_result":null,"sc_filter_category":null,"x_virus_id":null,"x_exception_id":null,"rs_content_type":null,"s_supplier_ip":null,"cs_cookie":null,"s_port":null,"cs_version":null,"creationTime":null,"operation":null,"workload":null,"clientIP":null,"userId":null,"eventSource":null,"itemType":null,"userAgent":null,"eventData":null,"sourceFileName":null,"siteUrl":null,"targetUserOrGroupType":null,"targetUserOrGroupName":null,"sourceFileExtension":null,"sourceRelativeUrl":null,"resultStatus":null,"client":null,"loginStatus":null,"userDomain":null,"clientIPAddress":null,"clientProcessName":null,"clientVersion":null,"externalAccess":null,"logonType":null,"mailboxOwnerUPN":null,"organizationName":null,"originatingServer":null,"subject":null,"sendAsUserSmtp":null,"deviceexternalid":null,"deviceeventcategory":null,"devicecustomstring1":null,"customnumber2":null,"customnumber1":null,"emailsender":null,"sourceusername":null,"sourceaddress":null,"emailrecipient":null,"destinationaddress":null,"destinationport":null,"requestclientapplication":null,"oldfilepath":null,"filepath":null,"additionaldetails11":null,"applicationprotocol":null,"emailrecipienttype":null,"emailsubject":null,"transactionstring1":null,"deviceaction":null,"devicecustomdate2":null,"devicecustomdate1":null,"sourcehostname":null,"additionaldetails10":null,"filename":null,"bytesout":null,"additionaldetails13":null,"additionaldetails14":null,"accountname":null,"destinationhostname":null,"dataSourceId":2,"date":"","violated":false,"oobjectId":null,"eventCategoryName":"AD","sourceDataType":"AD"})) - element of array (index: 0) - array (class [Lorg.apache.kafka.clients.consumer.ConsumerRecord;, size 1) at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40) ~[spark-core_2.11-2.3.0.jar:2.3.0] at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46) ~[spark-core_2.11-2.3.0.jar:2.3.0] at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100) ~[spark-core_2.11-2.3.0.jar:2.3.0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:393) ~[spark-core_2.11-2.3.0.jar:2.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_151] at java.lang.Thread.run(Unknown Source) [na:1.8.0_151]

2019-04-24 19:07:00.025 ERROR 21144 --- [result-getter-1] o.apache.spark.scheduler.TaskSetManager : Task 1.0 in stage 48.0 (TID 97) had a not serializable result: org.apache.kafka.clients.consumer.ConsumerRecord

Code for reading topic data is below -