hire-edge

This repository was initially created and built during my time as an EDGE Fellow for the College of Natural Science (Fall 2026). The original purpose was to create an automated pipeline for identifying hiring trends in data science to provide the BS in Data Science program with a competitive "edge" when our students enter the workforce. Since that initial creation, I've been using this project as a mechanism to explore web-scraping and streamlit-based dashboard development in Python. I've also used this as an experimental space to test how AI-assistive tools like Claude Code and GitHub Copilot can enhance the development process and, as such, there are contributions to the code based that come from these AI tools in addition to my own work.

The project is open source and available on GitHub for anyone interested in using it as a reference or starting point for their own data science job market analysis projects.

Project Overview

This Python-based tool scrapes, analyzes, and visualizes data science job postings to paint a comprehensive picture of the current career landscape. It helps identify:

• In-demand skills and technologies - What skills employers are actively seeking
• Geographic distribution - Where data science jobs are located
• Sector trends - Which industries are hiring data scientists
• Experience requirements - What levels of experience are in demand

Features

• Data Collection: LinkedIn scraping via Playwright, Adzuna API, and JSearch API (OpenWeb Ninja) integration
• Data Analysis: Extract insights about skills, locations, sectors, and experience levels
• Post-processing: Re-derive sector, experience level, skills, and degree requirements from existing CSVs without re-scraping, enabling heuristic improvements over time
• Interactive Dashboard: Streamlit-based visualization dashboard with filtering capabilities
• Export Functionality: Download filtered data for further analysis

Requirements

• Python 3.9+
• uv for environment and dependency management
• See pyproject.toml for the full dependency list

Installation

1. Clone the repository:

git clone https://github.com/devinsilvia/hire-edge.git
cd hire-edge

2. Install dependencies with uv:

uv sync

3. Install the Playwright browser binary:

uv run playwright install chromium

Optionally verify the browser setup:

uv run python setup_browser.py

4. Create a .env file (optional):

cp .env.example .env

Edit .env to customize scraping parameters if needed.

Quick Start

Want to explore the dashboard right away without scraping any data? The repository includes an example dataset so you can skip straight to visualization:

uv sync
uv run python run.py dashboard  # select job_postings_jsearch_example.csv in the sidebar

The example dataset contains 124 real job postings across 23 fields and is sufficient for exploring all dashboard features and experimenting with heuristic changes.

For a full data collection workflow, see the Usage section below.

---

Usage

Step 1: Scrape Job Postings

LinkedIn scraping uses a two-phase workflow — first collect job URLs, then scrape each one:

uv run python run.py collect-urls
uv run python run.py scrape-from-urls

This creates a CSV file in the data/ directory with job postings. Splitting the phases lets you review or edit the URL list before scraping begins, retry only failed URLs without re-collecting, and track failures separately in data/failed_job_urls.txt.

Step 2: Analyze Data (Optional)

Print summary statistics to the console:

uv run python run.py analyze

Step 3: Launch the Dashboard

Start the Streamlit dashboard to visualize the insights:

uv run python run.py dashboard

The dashboard will open in your default web browser (typically at http://localhost:8501).

Dashboard Features

The interactive dashboard provides:

• Key Metrics: Total jobs, companies, locations, and sectors
• Skills Analysis: Bar chart of most in-demand technical skills
• Geographic Distribution: Pie chart showing job distribution by location
• Sector Analysis: Breakdown of jobs by industry sector
• Experience Levels: Distribution of experience requirements
• Job Titles: Common job title variations
• Skills by Sector: Sector-specific skill requirements
• Filtering: Filter by location, sector, experience level, and degree required
• Data Source Selector: Switch between available job posting CSV files
• Data Export: Download filtered results as CSV

Project Structure

hire-edge/
├── src/
│   ├── __init__.py          # Package initialization
│   ├── scraper.py           # Job scraping functionality (LinkedIn + Adzuna + JSearch)
│   ├── analyzer.py          # Data analysis module
│   └── dashboard.py         # Streamlit dashboard
├── scripts/
│   ├── adzuna_field_explorer.py  # Explore raw Adzuna API fields
│   └── jsearch_test.py           # Ad-hoc JSearch API exploration
├── data/
│   └── job_postings_jsearch_example.csv  # Example dataset (committed for testing)
├── tests/                   # Test directory
├── run.py                   # CLI entry point (argparse)
├── pyproject.toml           # Project dependencies (uv)
├── .env.example            # Example environment configuration
├── .gitignore              # Git ignore rules
└── README.md               # This file

Known Limitations

• LinkedIn guest sessions are typically limited to ~60–70 unique jobs per search page
• LinkedIn applies rate limits even when the sign-in modal is dismissed automatically
• JSearch free tier may return 504 timeouts when JSEARCH_PAGES_PER_REQUEST is set above 1

Future Enhancements

• Additional job board integrations (Indeed, Glassdoor) — subclass BrowserScraper or APIScraper as appropriate
• Sentiment analysis of job descriptions
• Salary distribution and employment type charts in the dashboard (data is collected from Adzuna and JSearch; visualization is planned)
• Temporal trend analysis (tracking changes over time)
• Machine learning to predict job market trends
• Email alerts for new matching positions

Contributing

Contributions are welcome! There are three natural entry points depending on your interests:

Data collection — add new job board integrations by subclassing BrowserScraper or APIScraper in src/scraper.py, or improve existing scrapers. This typically requires API keys or a working browser environment.

Heuristics and analysis — tune the sector, experience_level, degree_required, and skills heuristics in src/scraper.py, or improve the JobAnalyzer class in src/analyzer.py. No API keys needed: the committed example dataset works as a test bed for reprocess_csv and the analyzer without any scraping.

Dashboard and visualization — improve charts and filters in src/dashboard.py. Also requires no API keys or scraping. Just run:

uv sync
uv run python run.py dashboard  # select job_postings_jsearch_example.csv in the sidebar

The BACKLOG.md file lists concrete ideas across all three tracks — salary/employment-type charts, skill extraction gaps, job title clustering, and more.

Please feel free to submit a Pull Request for any of the above.

License

This project is licensed under the MIT License. It originated as an educational project during the EDGE Fellowship program and is open source for anyone to use as a reference or starting point.

Author

Created by Devin Silvia as part of the EDGE Fellowship for the College of Natural Science to support BS in Data Science students.

Contact

For questions or feedback about this project, please open an issue on GitHub.