Apr 21, '202 min readApache Spark, Big data, Hadoop, macOS

Install Apache Spark on macOS

This short guide will assume that you already have already homebrew, xcode-select and java installed on your macOS. If not, run the following commands on your terminal.

$ xcode-select –install
$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
$ brew cask install java

Once you are sure that everything is correctly installed on your machine, you have to follow these steps to install Apache Spark

Step 1: Install scala

brew install scala@2.11

Keep in mind you have to change the version if you want to install a different one

Step 2: Install Spark

brew install apache-spark

Step 3: Add environment variables

Add the following environment variables to your .bash_profile or .zshrc

export SPARK_HOME=/usr/local/Cellar/apache-spark/2.4.5/libexec           export PATH="$SPARK_HOME/bin/:$PATH"

Keep in mind you have to change the version to the one you have installed

Step 4: Review binaries permissions

For some reason, some installations are not give execution permission to binaries. Review it and give them if necessary.

chmod +x /usr/local/Cellar/apache-spark/2.4.5/libexec/bin/*

Keep in mind you have to change the version to the one you have installed

Step 5: Verify installation

If everything worked fine you will be able to open a spark-shell running the following command

spark-shell

This should open a shell as follows

$ spark-shell 
20/04/21 12:32:33 WARN Utils: Your hostname, mac.local resolves to a loopback address: 127.0.0.1; using 192.168.1.134 instead (on interface en1)
20/04/21 12:32:33 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
20/04/21 12:32:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://192.168.1.134:4040
Spark context available as 'sc' (master = local[*], app id = local-1587465163183).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.5
      /_/
         
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_162)
Type in expressions to have them evaluated.
Type :help for more information.

scala>

We are done! You have successfully installed Apache Spark on macOS

You can also find me on Twitter if you’d like to read similar technical tricks!