IDEAM Rainfall Data Fetcher 🌧️

A Python tool to fetch and process rainfall data from Colombia's IDEAM (Institute of Hydrology, Meteorology and Environmental Studies) public API. This tool automatically filters for actual rainfall events and creates multiple data aggregations.

Features ✨

• Smart Filtering: Only downloads data when it's actually raining (valorobservado > 0)
• Multiple Aggregations: Creates raw, 30-minute, and daily rainfall aggregations
• Public Data: Uses Colombia's open government data API (no API keys required)
• Easy to Use: Simple Python class with comprehensive documentation
• Flexible Output: Saves data as CSV files with timestamps

Installation 📦

1. Clone this repository:

git clone https://github.com/yourusername/ideam-rainfall-data.git
cd ideam-rainfall-data

2. Install dependencies:

pip install -r requirements.txt

Quick Start 🚀

Basic Usage

from src.ideam_rainfall import IDEAMRainfallFetcher

# Create fetcher instance
fetcher = IDEAMRainfallFetcher()

# Run complete analysis (last 7 days)
raw_df, agg_30min_df, agg_daily_df = fetcher.run_full_analysis()

Advanced Usage

from src.ideam_rainfall import IDEAMRainfallFetcher

# Create fetcher with custom settings
fetcher = IDEAMRainfallFetcher()

# Get data for last 14 days
raw_df = fetcher.get_rainfall_data_with_rain_only(days_back=14)

# Create custom aggregations
raw_df, agg_30min_df, agg_daily_df = fetcher.create_aggregations(raw_df)

# Save to custom directory
saved_files = fetcher.save_all_files(
    raw_df, agg_30min_df, agg_daily_df, 
    output_dir="./my_rainfall_data"
)

# Show summary statistics
fetcher.show_summary_stats(raw_df, agg_30min_df, agg_daily_df)

Command Line Usage

python src/ideam_rainfall.py

Data Output 📊

The tool generates three types of CSV files:

1. Raw Data (ideam_lluvia_raw_YYYYMMDD_HHMM.csv)

• Minute-by-minute rainfall records
• Only includes records where it's actually raining
• Contains station information, coordinates, and rainfall values

2. 30-Minute Aggregated (ideam_lluvia_30min_YYYYMMDD_HHMM.csv)

• Rainfall summed over 30-minute intervals
• Only includes intervals with actual rainfall
• Useful for detailed temporal analysis

3. Daily Aggregated (ideam_lluvia_diaria_YYYYMMDD_HHMM.csv)

• Daily rainfall totals per station
• Only includes days with actual rainfall
• Perfect for daily weather analysis

Data Schema 📋

Common Columns

• codigoestacion: Station code
• nombreestacion: Station name
• departamento: Department/State
• municipio: Municipality/City
• latitud: Latitude coordinate
• longitud: Longitude coordinate
• unidadmedida: Measurement unit (mm)

Raw Data Columns

• fechaobservacion: Observation timestamp
• valorobservado: Observed rainfall value (mm)
• valor_numeric: Numeric rainfall value
• fecha_dt: Parsed datetime

30-Minute Aggregated Columns

• intervalo_30min: 30-minute interval timestamp
• lluvia_30min_mm: Total rainfall in 30-minute period (mm)

Daily Aggregated Columns

• fecha: Date
• lluvia_diaria_mm: Total daily rainfall (mm)

API Information 🔗

This tool uses Colombia's open government data API:

• Base URL: https://www.datos.gov.co/resource/s54a-sgyg.json
• Data Source: IDEAM (Institute of Hydrology, Meteorology and Environmental Studies)
• Authentication: None required (public API)
• Rate Limiting: Built-in delays between requests

Examples 📚

Example 1: Get Recent Rainfall Data

from src.ideam_rainfall import IDEAMRainfallFetcher

fetcher = IDEAMRainfallFetcher()

# Get last 3 days of rainfall data
raw_df = fetcher.get_rainfall_data_with_rain_only(days_back=3)

if raw_df is not None:
    print(f"Found {len(raw_df)} rainfall records")
    print(f"Total rainfall: {raw_df['valor_numeric'].sum():.1f} mm")
    print(f"Stations: {raw_df['codigoestacion'].nunique()}")

Example 2: Analyze Specific Station

# Filter data for a specific station
station_data = raw_df[raw_df['codigoestacion'] == '2801500078']

print(f"Station: {station_data['nombreestacion'].iloc[0]}")
print(f"Records: {len(station_data)}")
print(f"Total rainfall: {station_data['valor_numeric'].sum():.1f} mm")
print(f"Max intensity: {station_data['valor_numeric'].max():.1f} mm")

Example 3: Custom Output Directory

# Save files to custom directory
fetcher = IDEAMRainfallFetcher()
raw_df, agg_30min_df, agg_daily_df = fetcher.run_full_analysis(
    days_back=7, 
    output_dir="./rainfall_analysis_2024"
)

Configuration ⚙️

Adjustable Parameters

• days_back: Number of days to look back from today (default: 7)
• output_dir: Directory to save CSV files (default: current directory)
• base_url: IDEAM API endpoint (default: official endpoint)

API Request Strategies

The tool tries multiple approaches to fetch data:

1. Medium sample with server-side rain filtering
2. Large sample with server-side rain filtering
3. Date range with server-side rain filtering
4. All precipitation data with client-side filtering

Requirements 📋

• Python 3.7+
• requests >= 2.25.0
• pandas >= 1.3.0
• numpy >= 1.20.0

Contributing 🤝

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

License 📄

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments 🙏

• IDEAM (Institute of Hydrology, Meteorology and Environmental Studies) for providing the public API
• Colombia's Open Government Data initiative for making this data freely available
• Python community for the excellent libraries used in this project

Changelog 📝

v1.0.0

• Initial release
• Basic rainfall data fetching
• 30-minute and daily aggregations
• CSV output functionality
• Comprehensive documentation

Support 💬

If you encounter any issues or have questions, please:

1. Check the Issues page
2. Create a new issue with detailed information
3. Include your Python version and error messages

---

Made with ❤️ for Colombia's weather data community