Who gets our 3 a.m. call?

Posted 7:34 p.m. Jun 18, 2009 — Updated 7:34 p.m. Jun 18, 2009

Do you remember the hypothetical “3 a.m. call” debate during the presidential election? Who could best handle a crisis in the middle of the night? Clinton or Obama?

Well, at WRAL.com, we know exactly who to call when things go awry at 3 a.m. We have men, four men to be exact, who are up to the challenge when the website goes down.

Our team of software developers, led by Jason Priebe, carry a pager so they can quickly respond when a problem arises. But more than that, they are a group of talented guys who make the website sharper and stronger for viewers like you.

I asked Jason to tell me about his team and what they do behind the scenes to make WRAL.com work so well:

Kelly: Tell me about your team. What do they do?

Jason: We have an excellent team of software developers: Rusty Kroboth, Peter Lawrence and Gary Whitmore. All of these guys have strong technical backgrounds. Both Rusty and Peter came to us with years of experience in "big media" Web publishing. Gary graduated in 2007 and joined us that year.

I joined WRAL OnLine in 1996, left to work for a startup in RTP in 2001, and returned to Capitol Broadcasting Company’s New Media division in 2005 to build out the infrastructure to bring WRAL.com back in house.

The technical staff of CBC New Media is responsible for building and maintaining a number of sites, including WRAL.com, WRALSportsFan.com, localtechwire.com, HighSchoolOT.com, ncwanted.com, and others.

That entails building and deploying the HTML/CSS/Javascript for the sites in our publishing system, building interactive applications for the sites (like contests, search pages, mailing list signups, etc.), building automation to manage content, managing media workflows (audio, video, and image publishing), among many other tasks. In addition, we manage all of our own server and network infrastructure (hardware and operating systems).

Given the importance of sites like wral.com to the company and to the community, we have 24/7 support responsibilities. If a system goes down in the middle of the night, or a newsroom editor needs help on a Saturday afternoon, one of us is going to get paged about it. We share that pager duty so that it's not too massive of a burden.

Because we all take shifts on call, we all have to be cross-trained. All of our team members serve as software developers, database admins, and system administrators. For a small team like ours, it's not practical to have specialists. As a result, we are very nimble to respond to new challenges -- a software developer doesn't have to wait for systems engineers to install the new tools he needs for a project -- he just does it himself.

Kelly: People who visit WRAL.com probably don't know your names, but they know your work. What are some of the projects your team has worked on that viewers would recognize?

Jason: Recently, we've spent significant development resources on our Out & About section, where we provide a calendar of events and venues around the Triangle. What we hope we've done is to build a nice platform on which we can build and expand this service. This project involved some large modifications to our core publishing system and a number of integrations with various feed providers.

We're currently preparing an overhaul to localtechwire.com that will bring the site a whole new look and some nice new interactive features. For example, the new site will make extensive use of tagging on its stories. This will help visitors find related content quickly.

Some of our biggest projects are harder to point to. For example, we recently overhauled our video encoding system to make it easier to schedule live video streams and to create new ones on the fly. I can't point to a particular page on the site where you would see the results, but overall, we hope that it will make the WRAL.com newsroom faster to respond to breaking news video.

Kelly: What are some challenges you face and how do you try to overcome them?

Jason: There's the constant struggle of trying to keep up with increasing demand for the content we're providing. Every year, we continue to see higher and higher page view numbers. This puts more and more demand on our server infrastructure. We have to find creative ways to scale that infrastructure in a cost-effective way. To meet that challenge, we like to use open-source solutions like Linux and Apache to avoid costly licensing fees (and because of their proven track record of reliability).

Another big challenge is finding a balance between maintaining all the existing services while continuing to build and expand new ones. We have more than 200 different automated processes running at all times, gathering data, images, audio, and video from various sources, reformatting it, and packaging it for delivery on our various platforms. Every time we build a new site or new section on WRAL.com, we add to that list. It's critical that these processes run with minimal intervention from our team.

We work hard to build monitoring systems that can distinguish between temporary failures and critical failures in our various systems. Good monitoring helps us stretch our team's resources.

Kelly: What's the best thing about working on this site?

Jason: It's a lot of fun to be in an exciting environment like WRAL. Nothing is ever slow here, and working in the newsroom when big news is happening is amazing. Not all software developers can say that about their work environments.

I hope I'm speaking for the team here, but what I like best is that the work we do touches so many people throughout our community. I'm proud that the information we provide helps people make decisions every day, whether it's for work, for play, how they vote, or even about their families' safety.

Kelly: What would you like viewers to know or understand better about the work you do?

Jason: I want them to know that our team is deeply committed to providing a high-quality product that is available when they need it, responds quickly, and is available on all current platforms: Windows, Mac, and Linux (and all the recent versions of the major browsers).

You'd be surprised how many people assume that any problems they see are due to us building a "Windows/Internet Explorer only site." Nothing could be further from the truth. In fact, two of our developers use Macs every day, and most of us use Firefox as our primary browser. We pride ourselves on using standards-based development techniques, and we do as much cross-platform testing as we can.

Suggest A Correction