So you want to be a First Responder (Scribe)? You've come to the right place! You don't need to be a senior team member to be a First Responder — anyone can do it providing you have the requisite knowledge.

Purpose#

The purpose of the First Responder is to be the first person on the scene of an incident. You triage the situation, declare the incident, create the incident Slack channel, and then serve as scribe — maintaining a timeline of key events during the incident, documenting actions, and keeping track of any follow-up items that will need to be addressed.

It's important for the rest of the command staff to be able to focus on the problem at hand, rather than worrying about documenting the steps.

Your job as First Responder is to listen to the call and to watch the incident Slack channel, keeping track of context and actions that need to be performed, documenting these in Slack as you go. You should not be performing any remediations, checking graphs, or investigating logs. Those tasks will be delegated to the subject matter experts (SMEs) by the Incident Commander.

Prerequisites#

Before you can be a First Responder, it is expected that you meet the following criteria. Don't worry if you don't meet them all yet, you can still continue with training!

  • Excellent verbal and written communication skills.

Responsibilities#

Read up on our Different Roles for Incidents to see what is expected from a First Responder, as well as what we expect from the other roles you'll be interacting with.

Training Process#

There is no formal training process for this role, reading this page should be sufficient for most tasks. Here's a list of things you can do to train though:

  • Read the rest of this page, particularly the sections below.

  • Participate in incident response simulations and drills.

    • Shadow a drill to see how it's run.
    • Be the First Responder (scribe) for multiple drills.

Scribing#

Scribing is more art than science. The objective is to keep an accurate record of important events that occurred on the call, so that we can look back at the timeline to see what happened. But what exactly is important? There's no overwhelming answer, and it really comes down to judgment and experience. But here are some general things you most definitely want to capture as scribe:

  • The result of any polling decisions.
    • This is not "9 people voted yay, 3 voted nay".
    • It is "Polled for if we should do rolling restart. is proceeding with restart."
  • Any follow-up items that are called out as "We should do this..", "Why didn't this?..", etc.
    • This is not "Why isn't the on-call SME on the call?"
    • This is "TODO: Why didn't we get paged for this earlier?"

Incident Call Procedures and Lingo#

The Steps for First Responder provide a detailed description of what you should be doing during an incident.

Here are some examples of phrases and patterns you should use during incident calls.

Note Important Actions#

During a call, you will hear lots of discussion happening. You should not be documenting all of this in the Slack channel. You only want to document things which will be important for the final timeline. It's not always obvious what this might be, and it's usually a matter of judgment. You generally want to note any actions the IC has asked someone to perform, along with the result of any polling decisions.

Polled for decision on whether to perform rolling restart. We are proceeding with restart. [USER_A] to execute.

Some actions might seem important at the time, but end up not being. That's OK. It's better to have more info than not enough, but don't go overboard.

Note Follow-up Actions#

Sometimes during the call, someone will either mention something we "should fix", or the IC will specifically ask you to note a follow-up item. You can do this in Slack by simply prefixing with "TODO" — this will make it easier to search for later.

TODO: Why did we not get paged for the fall in traffic on [X] cluster?

The postmortem owner will find these after and raise ClickUp tickets for them.

End of Call Notification#

When the IC ends the call, you should post a message into Slack to let everyone know the call is over, and that they should continue discussion elsewhere.

Call is over, thanks everyone. Follow up in Slack.