Ustream as a company likes to challenge employees to think outside of the box and fuse in creativity. We believe that being surrounded by unique, innovative people helps develop new ideas.
In February, we hosted an internal Hackathon to give engineers the freedom to work on things they love and to hack solutions that could solve problems to find better ways of doing things at our company.
(To learn more, check out this previous blogpost)
We hoped that in a few months, one or two of the Hackathon projects would become a reality and make it into the product as a feature or a completely new application.
Some of our developers needed a tool to make their life easier. Instead of purchasing an already available product, they gathered together and developed their own tool, now called Openduty. (You can see the source on Github)
The OpenDuty project is an alarm aggregation and dispatching service built for system administrators and support teams on call duty. When an incident happens regarding our server or service, this tool provides a proper alerting system for quick notification. This development will automatically route actionable alerts when a problem occurs and determine whom to contact.
Till the end of the Hackathon contest an internal SMS gateway, a Pagerduty compatible REST API, and a basic admin dashboard was developed. The team succeeded to send automated SMS and phone call notifications for a given alert, escalate it according to a calendar, and see logs according to the failures in the system.
In 2 months they have build a Beta version and started implementing new bells and whistles.
Today, Openduty collects alerts from monitoring tools, provides an overall summary of the alerts and allows users to build notification rules. It provides also the freedom to set up multiple levels of coverage, so if the “primary” on-call person doesn’t respond to an alert in a timely fashion, it’s automatically escalates to a “secondary” person, and so on. The system integrates with GTalk, Push notification, Phone and Text Message (operates through Twilio).
The idea came from Bálint Csergő (Deathowl) who works as an Infrastructure engineer at Ustream Hungary. He and his team constantly work on new ideas to make Ustream’s service better.
We welcome any feedback or ideas regarding this project!