Sunday 31 January 2016

Simplest way for Running Spark on Windows machine

I have posted this solution as an answer to one of the stackoverflow question.

You can run spark jobs in windows machine. But, it needs few additional things. There are couple of files which are required in Hadoop Home.

1) winutils.exe
2) winutils.dll


Steps:

1) Download latest version of Hadoop from hadoop website.
2) Download winutils.exe and winutils.dll from below link. 
3) Copy winutils.exe and winutils.dll from that folder to your $HADOOP_HOME/bin.

4) or at the command, and add HADOOP_HOME/bin to PATH in environment variables. You can go to Advanced System settings and choose environment variables and do this. 

After this run the Spark jobs. 


No comments:

Post a Comment