Marketplace Scraper is a web scraping tool designed to extract listings from online marketplaces such as Facebook Marketplace and Kijiji. It allows users to specify search parameters such as year, make, model, city, and keywords to filter the search results and save them to a CSV file.
- Scrapes listings from Facebook Marketplace and Kijiji.
- Filters search results by year, make, model, city, and keywords.
- Saves filtered listings to a CSV file.
- Error logging for encountered issues during scraping.
- Easy setup and installation using a virtual environment.
-
Download Python 3.10.9: You need to download Python 3.10.9 from the official Python website.
-
Install Python: After downloading the installer, follow the instructions provided to install Python on your system. Make sure to select the option to add Python to your PATH during the installation process. You can watch this video if you need help to install Python 3.10.9 https://www.youtube.com/watch?v=ASBwcqw2yZI
-
Configure PATH (if necessary): If Python is not automatically added to your PATH during installation, you can manually configure it. Follow the appropriate instructions for your operating system:
- Windows: Follow these instructions.
- macOS/Linux: Edit the
.bashrcor.bash_profilefile in your home directory and add the following line:Replaceexport PATH="$PATH:/path/to/python3.10.9/bin"
/path/to/python3.10.9with the actual path where Python is installed.
-
Check Python Installation: Open a terminal or command prompt and run the following command to verify that Python 3.10.9 is installed correctly:
python3 --version
-
Download Source Code Download the repo from https://github.com/moya1295/parts_scraper. Extract in your desired folder.
-
Double Click Install.py Double click install.py file to complete the installation process. once finished you will see a folder created by the name virt. that means installation is complete. This may take 10-15 mins.
-
Once the installation is complete, you can run the
main.pyscript to start scraping listings from the desired marketplace. -
Follow the prompts to select the spider, enter the required information (year, make, model, city, keywords), and start the scraping process.
-
The output CSV file will be saved in the
outputdirectory with the specified filename.
- When prompted, enter
1to select the Facebook spider or2to select the Kijiji spider.
- You will be asked to enter the year, make, model, city, and two optional keywords (if any) to filter the search results.
- For example, if you want to search for a 2010 Toyota Corolla in Toronto, you would enter
2010,Toyota,Corolla, andTorontoas input. - For example, if you want to search for 2020 Honda Civic Front Bumper in Montreal, you would type keywords as year=2020, make=honda, model=civic, city=montreal, keyword_1=front, keyword_2=bumper.
- After the spider finishes scraping, the results will be saved in a CSV file named based on the provided input parameters.
- You can find the output CSV file in the
outputdirectory.
- Any errors encountered during the scraping process will be logged in the
errors.logfile located in the project directory.
- Press
Enterto exit the program after the scraping process is complete.