Postmortem & rebuild server #59
Description
[DigitalOcean] New Ticket # 767564 : Networking disabled: citadel
Hi there,
We are sorry to report that we have detected what appears to be a large flood of traffic from one or more of your servers that is disrupting the normal traffic flow for other users.
I got owned again :( early signs point to unsecured Elasticsearch -- right before New Relic cut out there was a huge CPU spike caused by some process I don't recognize running under the elasticsearch user.
- Find out how they got in
- find out how to stop it from happening again
- Get rid of the droplet
- Spin up new server (Could be an opportunity to switch to AWS)
- reconnect DNS
- rebuild server (bright side: good full test of Ansible)
- bonus: Get elasticsearch off the server and switch to Amazon's new hosted ES