I Almost Died for a Full Sentry Database
A tale of bad logging practices, overengineering, and a near-death experience.
https://it-notes.dragas.net/2024/12/28/i-almost-died-for-a-full-sentry-database/
(Repost, one year later)
#ITNotes #IT #SysAdmin #HorrorStories #Sentry #CloudComputing #Cloud #Repost
@stefanoStefano Marinelli I had similar experiences*, pre- and post-cloud. Devs casually coding “horror stories” and refusing to even considering double-checking. Last job in Italy, an alarm went off at 2am on Christmas night; that time I sent a very long and detailed email with half of the universe in cc. Here in the UK I spent 3 weeks with two other colleagues working days, nights and weekends to figure out weird timeouts. When we explained the problem what the devs should have fixed, the CTO thanked us but said that they didn't have time; too many important features to add.
*) I even fainted in a similar way :) but I wasn't on call that time.
If you have a fediverse account, you can quote this note from your own instance. Search https://hachyderm.io/users/jtheo/statuses/115803162821569619 on your instance and quote it. (Note that quoting is not supported in Mastodon.)
