There are several main roles for our incident response teams at Polygon. Certain roles only have one person per incident (e.g. IC), whereas other roles can have multiple people (e.g. Subject Matter Expert, SME). It's all about coming together as a team, working the problem, and getting a solution quickly.

The diagram below shows a generic incident response role hierarchy that is common across the industry. We use it as a framework to describe our own approach — not every role maps one-to-one at Polygon, and the sections below explain how we've adapted each role to fit our team structure and operational reality.

Incident Response Structure

During larger complex incidents, the role structure may be adjusted to account for the creation of sub-teams. Read about how we handle complex incidents for more information.

Flexible Structure

It is not intended that every role be filled by a different person for every incident. For example, if the incident is small enough in scope, the Deputy might also take on the responsibilities of the Scribe and Internal Liaison for that specific incident. The structure should be flexible and scale based on the size and scope of the incident.

Incident Commander (IC)#

What is it?#

An Incident Commander acts as the single source of truth of what is currently happening and what is going to happen during a major incident. They come in all shapes, sizes, and colors.

Why have one?#

As any software system grows in size and complexity, things break and cause incidents. The Incident Commander is needed to help drive major incidents to resolution.

What are the responsibilities?#

  1. Help prepare for major incidents,
    • Setup communications channels for major incidents.
    • Funnel people to these communications channels when there is a major incident.
    • Train team members on how to communicate during major incidents and train other Incident Commanders.
  2. Drive major incidents to resolution,
    • Get everyone on the same communication channel.
    • Collect information from team members for their services/area of ownership status.
    • Collect proposed repair actions and then recommend repair actions to be taken.
    • Delegate all repair actions, the Incident Commander is NOT a resolver.
    • Be the single authority on system status
  3. Communication during major incidents,
    • Timely reminders to Customer Liaison to draft external communication messages when needed.
    • Reviewing, giving feedback and approving draft external communication
    • Asking Customer Liaison to post the reviewed draft external communication
    • Asking all responders if there are strong objections to posting the external communication is a good practice but is not mandatory
    • Approving the removal of an ephemeral investigation message if there has been no customer impact and the major incident turns out to be a false alarm
    • If Customer Liaison is not available to post on the status page, the IC is responsible for delegating somebody to post publicly in their stead.
  4. Postmortem,
    • Creating the initial template right after the incident so people can put in their thoughts while fresh.
    • Assigning the postmortem after the event is over, this can be done after the call.
    • Work with Team Leads/Managers on scheduling preventive actions.

Who are they?#

The IC is a senior member of the product team facing the incident. Who specifically depends on the affected product:

  • Polygon PoS Chain — Adam Dossa or Mudit Gupta (CTO)
  • Agglayer — Taylan or Mudit Gupta (CTO)
  • Other products — The senior engineering lead for the affected product, or Mudit as the catch-all escalation

If the designated IC is unavailable, the most senior Subject Matter Expert on the call should step into the IC role until a proper IC can be reached.

How can I become one?#

Take a look at our Incident Commander training guide.

IC Fallback

We do not currently run a formal Deputy role. If the IC needs to step away or transition to an SME role, they should explicitly hand off IC responsibilities to another senior responder on the call. This handoff must be announced clearly in the incident channel — never assume someone else has taken over.

Scribe#

What is it?#

A Scribe documents the timeline of an incident as it progresses and makes sure all important decisions and data are captured for later review.

Why have one?#

The Incident Commander will need to focus on the problem at hand and the subject matter experts will need to focus on resolving the incident. It is important to capture a timeline of events as they happen so that they can be reviewed during the postmortem to determine how well we performed, and so we can accurately determine any additional impact that we might not have noticed at the time.

What are the responsibilities?#

The Scribe is expected to:

  1. Ensure the incident call is being recorded.
  2. Note in Slack important data, events, and actions, as they happen. Specifically:
    • Key actions as they are taken (Example: "prod-server-387723 is being restarted to attempt to remove the stuck lock")
    • Status reports when one is provided by the IC (Example: "We are in S0, block production has halted due to a stuck lock on the block producer, X is restarting the service, next checkin in 3 minutes")
    • Any key callouts either during the call or at the ending review (Example: "Note: (Bob B) We should have a better way to determine stuck locks.")

Who are they?#

Anyone can act as a Scribe during an incident, and are chosen by the Incident Commander at the start of the call. At Polygon, the First Responder on duty typically fills the Scribe role — they're already maintaining the incident timeline, posting status updates to status.polygon.technology, and logging key events in the incident channel. For larger incidents, the IC may designate a separate Scribe to free the First Responder for other coordination tasks.

How can I become one?#

Follow our Scribe training guide. If you're on the First Responder rotation, scribing is part of the role. For others, let the Incident Commanders know you'd like to be considered for scribing during the next incident.

Subject Matter Expert#

What is it?#

A Subject Matter Expert (SME), sometimes called a "Resolver", is a domain expert or designated owner of a component or service that is part of the Polygon infrastructure or service stack.

Why have one?#

The IC and Deputy are not all-knowing super beings. When there is a problem with a service, an expert in that service is needed to be able to quickly help identify and fix issues.

What are the responsibilities?#

  1. Being able to diagnose common problems with the service.
  2. Being able to rapidly fix issues found during an incident.
  3. Concise communication skills, specifically for CAN reports:
    • Condition: What is the current state of the service? Is it healthy or not?
    • Actions: What actions need to be taken if the service is not in a healthy state?
    • Needs: What support does the resolver need to perform an action?

Who are they?#

Anyone who is considered a "domain expert" can act as a resolver for an incident. At Polygon, SMEs are pulled in from the relevant on-call rotation:

  • DevOps (@DevOpsOnCall) — Node operations, infrastructure, deployments, chain recovery
  • Applications (@ApplicationsOnCall) — Portal, Staking, managed services
  • Smart Contracts (@SmartContractsOnCall) — Bridge, staking contracts, protocol-level issues
  • Agglayer (@AgglayerOnCall) — Agglayer service, cross-chain settlement
  • SecOps (@SecOpsOnCall) — Security incidents, vulnerability response

If the IC is unavailable, the most senior SME on the call should step into the IC role until a proper IC can be reached.

How can I become one?#

Take a look at our Subject Matter Expert training guide. You should also discuss with your team and service owner to determine what the requirements are for your particular service.

Customer Liaison#

What is it?#

A person responsible for interacting with customers and partners, either directly or via public communication channels. At Polygon, this responsibility is shared between two roles:

  • Communication Commander — A member of the marketing team who handles public-facing communications on X, Telegram, and Discord during major incidents.
  • BD OnCall (@BDOnCall) — Handles direct outreach to T0 customers and key partners, managing relationship-sensitive communications that go beyond public updates.

The First Responder also plays a role here, managing status page updates on status.polygon.technology and Partner Comms Bot notifications.

Why have one?#

All of the other roles will be actively working on identifying the cause and resolving the issue. We need roles focused purely on customer and partner interaction so that it can be done properly, with the due care and attention it needs.

What are the responsibilities?#

  1. Drafting external communication messages when needed, picking the appropriate template, either when asked by the IC or at own initiative.
  2. Asking for more information / clarification if needed for clear communication.
  3. Regularly notify the IC of partner and customer reports — who is affected, what they're seeing, and any specific escalations. This can include providing specific customer references or examples for investigation purposes.
  4. Post any publicly facing messages regarding the incident (X, Telegram, Discord, status page) once approved by the IC.
  5. Removing an ephemeral investigation message once approved by the IC.
  6. Provide customers with the external message from the postmortem once it is completed.

Who are they?#

The Communication Commander is drawn from the Marketing team. The BD OnCall is drawn from the Business Development team. Both are paged as part of the major incident response process.

How can I become one?#

Follow our Customer Liaison training guide, and discuss with your team lead about joining the Communication Commander or BD OnCall rotation.

Where can I find out more about external communications?#

Review our external communication guidelines.

Internal Liaison#

What is it?#

A person responsible for interacting with internal stakeholders. Whether it's notifying an internal team of the incident, or mobilizing additional responders within the organization.

Why have one?#

For larger incidents, we may have multiple teams across the organization involved. Having a dedicated liaison to mobilize those teams and bring them up to speed frees up the rest of the responders to handle the incident.

What are the responsibilities?#

  1. Page SME's or other on-call engineers as instructed by the Incident Commander (e.g. @DevOpsOnCall, @SmartContractsOnCall, @AgglayerOnCall).
  2. Notify other teams within the organization (e.g. Finance, Legal, Marketing, Executive team) as instructed by the Incident Commander.
  3. Liaise with stakeholders and provide status updates as necessary.
  4. Interact with internal stakeholders to answer their questions, to keep the primary call distraction free.

Who are they?#

Anyone designated by the Incident Commander during incident response. At Polygon, the First Responder or IC typically handles internal mobilization by paging the relevant on-call groups. For larger incidents, a separate Internal Liaison may be designated to free up the IC and First Responder.

How can I become one?#

Follow our Internal Liaison training.