PagerDuty Integration

This guide will show how to setup an integration between Indent and PagerDuty. Once done, your outage responders will be able to securely obtain auto-approved cloud and infrastructure access. For example, access servers based on their SRE on-call status.

PagerDuty + Indent Integration

• Grant incident responders secure, auto-approved, access to cloud apps and infrastructure based on their on-call status in PagerDuty.
• Record thorough access logs for compliance and auditing.
• Auto-revoke access after a set amount of time, or once the responder is no longer on-call.
• Use on-call status in PagerDuty to temporarily change status in GitHub Teams, Google Groups, Okta Groups, AWS IAM, and many other services.

How it Works

• When an engineer requests elevated privileges during an incident, Indent will check with the PagerDuty API to see if that person is on-call.
• IF the engineer is on-call, Indent will auto-approve the request based on rules you set.
• IF the engineer is NOT on-call, Indent will notify another team member or a manager for approval.
• The service that checks this will be deployed by you, as an AWS Lambda.
• To achieve the Principle of Least Privilege, access grants default to be time-based and ephemeral.

Requirements

note

This page assumes you completed the Indent Quickstart. (5 min setup)

• Indent’s Quickstart (5 min setup)
• PagerDuty Account with an on-call schedule
• PagerDuty API Key
• AWS S3
• AWS Lambda
• GitHub Actions

Support

If you need help with this integration, please start a conversation with our support engineers.

Integration Walkthrough

Configuring PagerDuty

• Go to your PagerDuty account, and navigate to Integrations → API Access Keys.
• Click the + Create New API Key button and follow the on screen prompts.
• Copy the API key and hold onto it as a secret for the next step.

Installing Indent

1. Cloning the repo

• Click on the Use This Template button to clone the GitHub repo for the Indent service.
• In the new repo go to Settings → Secrets → Actions.
• Click New Repository Secret and name it PAGERDUTY_KEY.
• Paste the value for your PagerDuty API Key and click the Add secret button to save.
• Leave this tab open for adding the secrets mentioned in the next steps.

2. Configuring the S3 bucket

• Go to AWS S3 and select an existing bucket or create a new one
  • Most of the default settings are good, but some recommended values are included here.

Recommended AWS S3 values

• Name — easily identifiable name for the bucket, such as indent-deploy-state-123.
• Region — where you plan to deploy the Lambda, like us-west-2.
• Bucket versioning — if you want to have revisions of past deployments, otherwise pick disabled.
• Default encryption — enable for server-side encryption for deployment files.

• This integration currently assumes your region is us-west-2, and you want S3 encryption turned on. If you choose other settings, update your main.tf values accordingly.

3. Configuring AWS Credentials

• Go to AWS IAM → Add Users and create a new user for deploys, such as indent-terraform-deployer.
• Configure the service account's access:
  • Credential type — select Access key - Programmatic access.
  • Permissions — click Attach existing policies directly and select AdministratorAccess.
  • Follow the prompts until the account is created.
• Add the resulting values as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to GitHub Secrets.

4. Customize Secure Auto-approvals

You may want secure auto-approvals to behave differently based on a few conditions. For instance, is the response team dealing with sensitive data or are they fixing a more general incident? A shorter 3 hour duration might make sense for sensitive data, while an 18 hour duration might be ideal for more mundane incidents.

The following steps are optional, and by default approvals will check with every schedule on your account.

Change approval duration

• Add a GitHub Secret named AUTO_APPROVAL_DURATION.
  • Set the value to the default number of hours you would like an approval to last.
  • For example, a value of 3 would default to a 3 hour duration.

Similarly, you can set secure auto-approvals to only apply to specific on-call schedules.

Specify auto-approved schedules

• Visit your PagerDuty dashboard, and go to People → On-Call Schedules
  • Make a note of the schedule name(s) you want to select.
  • Note: The name of a schedule is case-sensitive, so kubes is different than Kubes.
• Add a GitHub Secret named AUTO_APPROVAL_PAGERDUTY_SCHEDULES
  • Set the value to a comma separated list of the schedules you want to target.
  • For example, a value of DBA, API would narrow the auto-approvals to only consider those schedules.

Lastly, adding labels to resources can help restrict auto-approvals to a smaller set of resources.

Filter with labels

• Visit your Indent Admin page.
  • For the rule you want to modify, click Edit → Add label
  • Enter a label name and value.
  • For example, a label name might be Env and value Prod.
  • Click Done → Save
• Navigate to Resources and click on a resource you want the rule to target.
  • Click Edit add an Env label.
  • CLick Add label → Save to save your changes.
  • Repeat this for each resource you want the rule to target.

5. Deploy the Webhook

• On your Indent dashboard go to Integrations → Catalog → PagerDuty.
  • Copy the webhook secret, and in a new tab add it to your GitHub Secrets as INDENT_WEBHOOK_SECRET.
  • Leave this PagerDuty + Indent integrations page open until the final step.
• In a new tab open main.tf from your GitHub repo, and change the empty value for backend to the name you chose in the step 2. Configuring the S3 bucket.
• From your repo navigate to GitHub Actions → The latest job, and copy the URL shown in Terraform Log.
  • Note: If your most recent job failed, relaunch the job and wait for it to succeed.
  • On your PagerDuty + Indent integrations page, paste the URL you just copied.

6. Add policies

• On your Indent dashboard go to Admin → Policies → Webhook Policies → + New.
  • Expand the webhooks field, and select your PagerDuty webhook.
  • Note: the Condition Builder section is where you can optionally add supplementary conditions for which resources the PagerDuty integration applies.
  • Save your changes.
• On your Indent dashboard go to Resources.
  • Click the Pull from integrations button, and select Indent Bot.
• On your Indent dashboard go to Admin → Config.
  • Edit an existing rule, or create a new rule.
  • Customize which resources you want to be targetted in the When someone requests access to... section.
  • Enable auto approval by adding the bot user or [email protected] to the list of Reviewers.
  • Select one of the elements under Kinds of Resources, and add any resources you want available for on-demand access.
  • Save your changes.

7. Test that it works

• Identify a person who is currently on-call.
• Submit an access request for that person, to the example resource
  • If you would like a reminder on how to make a request, visit the Indent Quickstart guide.
• Verify that the PagerDuty bot auto-approves access.
• Add resources like AWS IAM, Okta Groups, and other services that your on-call engineers might need to request access to during an incident.

Summary

Congrats! You added the PagerDuty + Indent integration. Your SREs can now request secure on-demand server access based on their on-call rotation. Next, try installing Google Groups or adding a GitHub Teams integration for secure access to sensitive repos.

How to Uninstall

• Go to your PagerDuty account, and navigate to Integrations → API Access Keys.
• Click Disable or Remove for the key you created for the PagerDuty + Indent integration.
• On your Indent dashboard go to Integrations → Webhooks, and select your PagerDuty webhook.
  • Click the Delete button to remove the integration.

Questions

Where do I view the code I'm deploying?

The code is in the initial Indent APIs GitHub repo that you cloned.

How do I create a PagerDuty API Key?

PagerDuty has detailed docs on how to find and create an API Key. This guide needs a General access REST API key.

Where can I find a list of all of the secrets?

A full list of secrets used by this integration are listed here.

Name	Value
INDENT_WEBHOOK_SECRET	Get this from your Indent App or an Indent Webhook in the Dashboard
PAGERDUTY_KEY	Create an API access key for programatically getting on-call schedule participants.
AWS_ACCESS_KEY_ID	Your Programmatic AWS Access Key ID
AWS_SECRET_ACCESS_KEY	Your Programmatic AWS Secret Access Key
AWS_SESSION_TOKEN	Optional: Your AWS Session Token. Note: If you use an AWS Session ID you will need to update it for each deployment once the session expires

How do I redeploy the webhook?

The repo you created from a template auto-deploys to AWS when you push or merge PRs to the main branch. You can manually redeploy the webhooks by re-running the latest GitHub Action job.

Why the bot failing to approve my requests?

It might just be that the PagerDuty Bot hasn't appeared yet. Try visiting Resources, clicking Pull from integrations, and selecting Indent Bot. If that fails to create a PagerDuty Bot resource, try the following.

• Navigate to Integrations → Resources → New.
• Create a new resource with Kind indent.v1.bot and a name like PagerDuty Auto-approval Bot.

If the bot exists, try contacting support.

How do I ask for help?

If you have questions or need help with your integration, try chatting with Indent Support.