Tags » SPARK

Appreciating the complexity of Stream processing systems

A typical streaming system has three parts to it:

  1. Input: Usually referred to as source
  2. Processing
  3. Output: Usually referred to as sink

Data flow:

Events would be received on the input side, processed (cleaned, deDuplicated, aggregated etc) and sent to the sink… 149 more words


runs through my veins
remaining eager
to take away your pain
to break your chains
sending sparks
lighting up my brain
shooting down
with the rain…

143 more words

From A spark...

.. to Be inspired 🔥

(Mark the changes in the site name)


Last Week in Stream Processing & Analytics – 23.5.2018

This is the 115th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week: 622 more words


Kindly Notice ⛔ Changed my wordpress url.↗️

from agp98.wordpress.com to inspirationalbeing.wordpress.com

I hope now we get more connected 🔥


I wanna change my wordpress Url..

Can anyone guide me in Details ?

What will I lose and What will I gain?

I need my url to be friendly to be found out


Decoding Apache Spark sourcecode at bigdatamann – part3

case: rdd.take(1)

val rdd = spark.sparkContext.parallelize(Seq(1,1000^3), 2)


The take method has the following logic:
– for rdd.take(5), read 1 partition and obtain the data… 239 more words