The Stormsync data collector is an open-source Go project designed for collecting hail, wind, and tornado Storm Reports from the National Weather Center. The core functionality involves collecting data from designated URLs, de-duplicating entries, and forwarding valid reports to a Kafka topic for downstream consumption.
| Report Type | URL |
|---|---|
| Wind Report | https://www.spc.noaa.gov/climo/reports/today_wind.csv |
| Hail Report | https://www.spc.noaa.gov/climo/reports/today_hail.csv |
| Tornado Report | https://www.spc.noaa.gov/climo/reports/today_torn.csv |
System Requirements:
- Go:
version 1.22.x
- Clone the repository:
$ git clone https://github.com/stormsync/collector
- Change to the collector directory:
$ cd collector/cmd/collect
- Build the app:
$ go build -o app
Run using the command below:
KAFKA_ADDRESS="my.kafka.ip-or-hostname:PORT" \ KAFKA_USER="my-kafka-username" \ KAFKA_PASSWORD="my-kafka-password" \ TOPIC="kafka-topic-to-send-reports-to" \ POLLING_INTERVAL_IN_SECONDS="60" \ REDIS_ADDRESS="my-redis-ip-or-hostname:PORT" \ REDIS_USER="my-redis-user" \ REDIS_PASSWORD="my-redis-password" \ ./app
If you do not use the environment variables or do not have them set in the environment you will see the following:
time=2024-05-21T11:02:59.097-05:00 level=ERROR msg="ENV variable required" Name=KAFKA_ADDRESS time=2024-05-21T11:02:59.097-05:00 level=ERROR msg="ENV variable required" Name=KAFKA_USER time=2024-05-21T11:02:59.097-05:00 level=ERROR msg="ENV variable required" Name=KAFKA_PASSWORD time=2024-05-21T11:02:59.097-05:00 level=ERROR msg="ENV variable required" Name=TOPIC time=2024-05-21T11:02:59.097-05:00 level=ERROR msg="ENV variable required" Name=POLLING_INTERVAL_IN_SECONDS time=2024-05-21T11:02:59.097-05:00 level=ERROR msg="ENV variable required" Name=REDIS_ADDRESS time=2024-05-21T11:02:59.097-05:00 level=ERROR msg="ENV variable required" Name=REDIS_USER time=2024-05-21T11:02:59.097-05:00 level=ERROR msg="ENV variable required" Name=REDIS_PASSWORD
Run the test suite using the command below from the root directory of the repo:
$ go test ./...
Contributions are welcome! Here are several ways you can contribute:
- Report Issues: Submit bugs found or log feature requests for the `` project.
- Submit Pull Requests: Review open PRs, and submit your own PRs.
- Join the Discussions: Share your insights, provide feedback, or ask questions.
- Fork the Repository: Start by forking the project repository to your local account.
- Clone Locally: Clone the forked repository to your local machine using a git client.
git clone ../
- Create a New Branch: Always work on a new branch, giving it a descriptive name.
git checkout -b new-feature-x
- Make Your Changes: Develop and test your changes locally.
- Commit Your Changes: Commit with a clear message describing your updates.
git commit -m 'Implemented new feature x.' - Push to local: Push the changes to your forked repository.
git push origin new-feature-x
- Submit a Pull Request: Create a PR against the original project repository. Clearly describe the changes and their motivations.
- Review: Once your PR is reviewed and approved, it will be merged into the main branch. Congratulations on your contribution!
This project is protected under the GNU Affero General Public License v3.0