Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Slide 54 is insanely important. At my current place we've invested tons in instrumentation and metrics. Every IO operation is guarded with a semaphore and access is monitored and exposed. Every cache in the system logs everything from hit and miss rate to volume etc. Every async operation logs operation trails. We also ship a ton of tools for everything from bread and butter stuff like stack dumping, pinpointing long running threads etc. to more sophisticated stuff like being able to break in to a page rendering and profiling various element hierarchies, either over time or following specific users.

As I said, we've invested so much in getting the entire stack monitored very granularly, but every time our customers run in to issues it pays dividends.

For reference, we run on the JVM which is very pleasant. A ton of tools interop with it and basic things like MBeans being integrated in to JVM makes monitoring a lot easier.



How much overhead does this setup incur?


Run time or development time? Runtime is mostly negligible except for places with heavy congestion (caches + rendering) where you can turn the heaviest operations on and off as needed -- in our case the call to `System.getCurrentTimeMillis()` is usually too expensive to do unless we really need it.

In terms of development, it's not a lot of work continuously, but it took some while to write up, test and benchmark standard libraries for eg monitorable semaphores. In general though, what it does is raise the entry threshold for new developers a bit as you need to have a feel for when something requires monitoring and when it doesn't.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: