The purpose of this project was to test the deployment of a Spark ML model with Spark Streaming and Apache Kafka in AWS. This project has two components. The first component is the simulatorAWS. This is the second component which essentially does the following:
- Using Spark Streaming, consume records from a Kafka topic where the strings represent an instance to be predicted.
- Parse and transform the records.
- Make the prediction using an already trained Random Forest classifier from Spark ML.
- Publish the records back to Kafka.
This project also includes the code to train the model which closely resembles the example in Chapter 4 of
Ryza, Sandy, et al. Advanced Analytics with Spark. O'Reilly, 2017.