Tags » SPARK

Mystery of SparkSQL default partitions

Spark uses “spark.default.parallelism”, property to decide on number of partitions after shuffle. In RDD world, one RDD inherits number of partitions from the parent RDD if there are no shuffle involved. 162 more words

Sparksql

How to connect with anyone

Try this powerful experiment, with a loved one, a friend, a stranger…

“Studies say that 4 minutes of uninterrupted eye contact can increase intimacy. To test this this theory out, we brought in six pairs in different stages of their relationship and had them try it.” 6 more words

Spirituality

Create StructType Column in SparkSQL using existing Columns

While working on one of the project, I had a requirement to create complex data types from existing tables and save out data in parquet format. 134 more words

Development

Cassandra Host outside of Spark Config

As mentioned at the cassandra spark connector help page, if you want to specify cassandra host and other related configurations, you need to use spark-default.conf or specify it while doing spark submit. 161 more words

Design

What stops you?

My dream is to pursue
what I love
and follow my heart
but nobody will let me
also where do I start?
They say if I do this… 82 more words

Poetry

Advocare makes it easy to make Money at Home!


Advocare is a very unique and exciting business opportunity for many reasons…

– In demand products with amazing Sponsors like Drew Brees

– Discounts from 20% all the way to 40% 296 more words

花火如火花

It was beautiful.

What we had.

And all those who agreed

Still questions why it didn’t last.

Of passion we gave sparks,

Of love we truly cared. 62 more words

Vivid Pieces