Tags » Sparkling Water

Running python and pysparkling with Zeppelin and YARN on Hadoop

Apache Zeppelin is very useful to use cell based notebooks (similar to jupyter) to work with various applications i.e. spark, python, hive, hbase etc by using various… 403 more words

Hadoop

Sophisticated welcoming drinks - they will love it!

I’ve learnt this one from an Italian friend of mine who lives in Germany. She is very enthusiastic and knows very well how to make people feel good in her house… starting with this delicious welcoming drink! 305 more words

How To

Using Kyro library with Sparkling Water

Start sparkling water from spark shell:

$ bin/sparkling-shell

Start sparkling water from spark shell and add 3rd party jar:

$ bin/sparkling-shell --jars kryo-4.0.0.jar
Note: Make sure the jar file is accessible from this path… 402 more words
Machine Learning

Sparkling Water 2.0 Walkthrough with pysparkling

My ENV:

SPARK_HOME=/Users/avkashchauhan/tools/spark-2.0.1-bin-hadoop2.6
 H2O_HOME=/Users/avkashchauhan/src/github.com/h2oai/h2o-3
 MASTER=local-cluster

Pysparkling Command:

$$> bin/pysparkling --num-executors 2 --executor-memory 2g --driver-memory 2g --conf spark.dynamicAllocation.enabled=false
 Python 2.7.10 (default, Jul 30 2016, 18:31:42)
  on darwin
 Type "help", "copyright", "credits" or "license" for more information. 321 more words

Using Sparkling water and PySpark to log console output

Here is the command Option #1:

./pyspark --deploy-mode client --conf spark.dynamicAllocation.enabled=false --packages com.databricks:spark-csv_2.11:1.4.0 --py-files ../../sparkling-water-1.6.7/py/dist/h2o_pysparkling_1.6-1.6.7-py2.7.egg

Here is the command Option #2:

./pyspark --deploy-mode client --conf spark.dynamicAllocation.enabled=false --packages com.databricks:spark-csv_2.11:1.4.0,ai.h2o:sparkling-water-core_2.10:1.6.7 --py-files ../../sparkling-water-1.6.7/py/dist/h2o_pysparkling_1.6-1.6.7-py2.7.egg… 100 more words
H2O

Sparkling Water - Tips and Tricks

You must set SPARK_HOME to proper spark version you would want to use:

$ export SPARK_HOME=/home/ec2-user/spark-1.6.2-bin-hadoop2.6

This is how you will launch spark shell:

$ bin/spark-shell or  $SPARK_HOME/bin/spark-shell…
318 more words
Scripting

Pysparkling launch issue with Sparkling water

When trying to import “pysparkling” package with the command below, you will get the following error:

> from pysparkling import *

Here is the error:

217 more words
Machine Learning