Install Apache Spark on macOS
This short guide will assume that you already have already
java installed on your macOS. If not, run the following commands on your terminal.
$ xcode-select –install $ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)" $ brew cask install java
Once you are sure that everything is correctly installed on your machine, you have to follow these steps to install Apache Spark
Step 1: Install scala
Step 2: Install Spark
brew install apache-spark
Step 3: Add environment variables
Add the following environment variables to your
Step 4: Review binaries permissions
For some reason, some installations are not give execution permission to binaries. Review it and give them if necessary.
Step 5: Verify installation
If everything worked fine you will be able to open a
spark-shell running the following command
This should open a shell as follows
$ spark-shell 20/04/21 12:32:33 WARN Utils: Your hostname, mac.local resolves to a loopback address: 127.0.0.1; using 192.168.1.134 instead (on interface en1) 20/04/21 12:32:33 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 20/04/21 12:32:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Spark context Web UI available at http://192.168.1.134:4040 Spark context available as 'sc' (master = local[*], app id = local-1587465163183). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.4.5 /_/ Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_162) Type in expressions to have them evaluated. Type :help for more information. scala>
We are done! You have successfully installed Apache Spark on macOS
You can also find me on Twitter if you’d like to read similar technical tricks!