Environment README
- This is a very complex environment
- You will see some scary WARNings and ERRORs - Everything will be OK, I promise!
- We will verify every step of the environment with either a UI or Command Line
- This repo contains code that is constantly evolving, so there may be TODOs, broken code, and dead ends
- Feel free to pick up anything that is marked TODO and submit a pull request - we love contributors!
- At some point, you will exit out of the Docker Container by accident - you can recover here
- Everything you experience in this codebase is available for you in a public Docker Image on Docker Hub here
- Copy and paste the ENTIRE command line
Here is an example command that will scroll of the screen because it's super long and I'm too lazy to figure out that weird \ thing in linux and make things more readable. Hello there!! Don't forget about me. There is more to this command than you can see because you are not scrolling far enough to the right.
- Do not copy and paste the prompt
spark-sql> SELECT nothing FROM prompt;
- Windows ONLY: Copy and Pasting with Putty
- Use the
whichcommand if you are trying to locate a script from the$PATH
which start-all-services.sh
### EXAMPLE OUTPUT ###
/root/pipeline/bin/service/start-all-services.sh
- We will often
tail -f <some-file>.logto verify a successful script startup - The script is running in the background
- The
tailis running in the foreground - You can - and will need to -
Ctrl-Cout of atailwithout disrupting the background script
- Docker throws this WARNING upon initial
docker run - Ignore this
- You will see this a lot
- If your command prompt gets stuck on this message, just hit
<enter>a few times
- Every Spark Application binds to an admin port for the admin UI - including Zeppelin which is a long-running Spark Application (similar to Spark Streaming)
- Spark will start at port 4040 and +1 until it finds an open port - throwing a WARN on every attempt
- When you start Zeppelin, for example, this will bind to the first available port after 4040
- We usually start Zeppelin first, so this usually binds to port 4040, but every Spark job after will throw a scary WARN.
- Ignore this!
- Spark SQL always creates a tiny
derby.logfile when it first starts up - before finding the real Hive Metastore - Sometimes this file is created in the same path where you run your Spark Application (ie.
pipeline-spark-shellSpark Shell) - Ignore this!
Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
- You likely have started too many long-running Spark Apps like SparkShell, PySparkShell, PySpark (Jupyter/iPython Notebook), Zeppelin Notebook, Spark Streaming, etc.
- Below is a sample screenshot of the Spark Admin UI when you are in this state
- You need to kill some of the long-running Spark Apps through the Spark Admin UI - usually one of the PySpark (Jupyter/iPython) apps
Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
- The optimized Hadoop libraries specific to this OS and Hardware have not been installed
- Ignore this!
WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
- This warning appears to indicate a problem with the Hive Metastore
- I assure you that the Hive Metastore is running properly
- Ignore this!
- Haven't bothered to track this warning down
- Ignore this!
- Related to the resource-starvation issue
- Every iPython/Jupyter notebook you run will acquire and hold 1 CPU
- Notebooks need to be run from top to bottom.
- If you run them out of order - or jump between notebooks - your variable assignments may clobber each other
- Keep an eye on this
- Most Scala-based notebooks operate by generating inner classes for each cell executed within the notebook
- The names of these inner classes, over many executions of the notebook/cell, get very long and exceed the OS filename length
- To recover, just restart the notebook kernel/interpreter using the
Interpretermenu option in Zeppelin - Note: You will lose all notebook-scoped variable assignments in Zeppelin