Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

During stressful times (like outages) people involved with response, troubleshooting, and recovery also often mis-remember the events as they happened, sometimes unconsciously neglecting critical facts and timings of observations, assumptions, etc.

This is a great reason to use IRC or some other loggable, text-based medium. After everything's green and you're doing the failure analysis, you can look at the logs to see what happened and when. IRC also makes it easier to collaborate if things are broken in the middle of the night when everyone is at home.



Why we have central monitoring systems that aggregate system data together. Makes trend analysis and correlation cross systems very useful. Unfortunately its a home grown solution, not sure what's available off the shelf.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: