One of my recent research tasks has been to port GCAM (Global Change Assessment Model) on to the Cube, our groups HPC cluster, and some of the… 650 more words
Iowans and Iowa institutions have played roles in nationally and internationally significant science and technology developments in the last week, but sometimes you have to know the background to understand their involvement. 1,564 more words
And now for the secret sauce–how to actually get Spark workers to run and talk to a master.
For a long time, I had my start script launching the master. 977 more words
So the next thing you need to do to get Spark running in Grid Engine is to set up your queue, job class, and your parallel environment. 707 more words
Apache Spark has become a pretty big thing where I work. We were originally approached about running it on our HPC cluster about 3 years ago, and, knowing little to nothing about Big Data clusters, I agreed to set it up and get it rolling. 384 more words