Token scanning

GitHub scans public repositories for known token formats to prevent fraudulent use of credentials that were committed accidentally.

As a service provider, you can partner with GitHub so that your token formats are included in our token scanning. Whenever a match of your token format is found, a payload is sent to an HTTP endpoint of your choice.

This article describes how a service provider can partner with us to be part of the token scanning system on GitHub.

The token scanning process

The following diagram summarizes the token scanning process, with any matches sent to a service provider's verify endpoint.

Flow diagram showing the process of scanning for a token and sending matches to a service provider's verify endpoint

Joining the token scanning system on GitHub

  1. Contact us to get the process started.
  2. Identify the relevant tokens you want to scan for, and create regular expressions to capture them.
  3. Create a token alert service which accepts webhooks from GitHub that contain the token scanning message payload.
  4. Implement signature verification in your token alert service.
  5. Implement token revocation and user notification in your token alert service.

Contact us to get the process started

To get the enrollment process started, send us an email at token-scanning@github.com.

We will send you details on the token scanning system, and you will need to agree to our terms of participation before proceeding.

Identify your tokens and create regular expressions

To scan for your tokens, we need the following pieces of information for each token that you want us to look for:

  • A unique, human readable name for the token type. We'll use this to generate the Type value in the message payload later.
  • A regular expression which finds the token type. Be as precise as possible, because this will reduce the number of false positives.
  • The URL of the endpoint for us to send the messages to. This does not have to be unique for each token type.

Send this information to token-scanning@github.com.

Create a token alert service

Create a public, internet accessible HTTP endpoint at the URL you provided to us. When a match of your regular expression is found, we will send a HTTP POST message, which follows the example below, to your endpoint.

POST / HTTP/1.1
Host: HOST
Accept: */*
Content-Type: application/json
GITHUB-PUBLIC-KEY-IDENTIFIER: 3ec68716d6df3f7cd532ac97e55420cb1c14375245
GITHUB-PUBLIC-KEY-SIGNATURE: MEUCICop4nvIgmcY4+mBG6Ek=
Content-Length: 0123

[
  {
    "token": "X-Header-Bearer: as09dalkjasdlfkjasdf09a",
    "type": "ACompany_API_token",
    "url": "https://github.com/octocat/Hello-World/commit/123456718ee16e59dabbacb1b4049abc11abc123"
  }
]

The message body is a JSON array that contains one or more objects which have the following contents. We use an array because we might batch multiple token matches in one message.

  • Token: The value of the match.
  • Type: The type name for your token match. This is what you provided when sending us the regular expression for the match.
  • URL: The public commit URL where the match was found.

Implement signature verification in your token alert service

We strongly recommend you implement signature validation in your token alert service to ensure that the messages you receive are genuinely from GitHub, and not malicious.

You can retrieve the GitHub token scanning public key from https://api.github.com/meta/public_keys/token_scanning.

Assuming you receive the following message, the code snippets below demonstrate how you could perform signature validation.

Sample message sent to verify endpoint

POST / HTTP/1.1
Host: HOST
Accept: */*
content-type: application/json
GITHUB-PUBLIC-KEY-IDENTIFIER: 3ec68716d6df3f7cd532ac97e55420cb1c14375245
GITHUB-PUBLIC-KEY-SIGNATURE: MEUCICop4nvIgmcY4+mBG6Ek=
Content-Length: 0000

[{"token": "some_token", "type": "some_type", "url": "some_url"}]

Validation sample in Go

package main

import (
  "crypto/ecdsa"
  "crypto/sha256"
  "crypto/x509"
  "encoding/asn1"
  "encoding/base64"
  "encoding/json"
  "encoding/pem"
  "errors"
  "fmt"
  "math/big"
  "net/http"
  "os"
)

func main() {
  payload := `[{"token": "some_token", "type": "some_type", "url": "some_url"}]`

  kID := "3ec68716d6df3f7cd532ac97e55420cb1c143752450d65bf916d41ca25d9dfc4"

  kSig := "MEUCICop4nvIgmcY4+mB+i5GlUYGNL20Qrlrx3RvrysilFHeAiEAg7yJ8KqEUcUadBaxepp3COhTUrk4feZ9TTb/xdBG6Ek="

  // Fetch the list of GitHub Public Keys
  req, err := http.NewRequest("GET", "https://api.github.com/meta/public_keys/token_scanning", nil)
  if err != nil {
    fmt.Printf("Error preparing request: %s\n", err)
    os.Exit(1)
  }

  req.Header.Add("Authorization", "Bearer "+os.Getenv("GITHUB_PRODUCTION_TOKEN"))

  resp, err := http.DefaultClient.Do(req)
  if err != nil {
    fmt.Printf("Error requesting GitHub signing keys: %s\n", err)
    os.Exit(2)
  }

  decoder := json.NewDecoder(resp.Body)
  var keys GitHubSigningKeys
  if err := decoder.Decode(&keys); err != nil {
    fmt.Printf("Error decoding GitHub signing key request: %s\n", err)
    os.Exit(3)
  }

  // Find the Key used to sign our webhook
  pubKey, err := func() (string, error) {
    for _, v := range keys.PublicKeys {
      if v.KeyIdentifier == kID {
        return v.Key, nil

      }
    }
    return "", errors.New("specified key was not found in GitHub key list")
  }()

  if err != nil {
    fmt.Printf("Error finding GitHub signing key: %s\n", err)
    os.Exit(4)
  }

  // Decode the Public Key
  block, _ := pem.Decode([]byte(pubKey))
  if block == nil {
    fmt.Println("Error parsing PEM block with GitHub public key")
    os.Exit(5)
  }

  // Create our ECDSA Public Key
  key, err := x509.ParsePKIXPublicKey(block.Bytes)
  if err != nil {
    fmt.Printf("Error parsing DER encoded public key: %s\n", err)
    os.Exit(6)
  }

  // Because of documentation, we know it's a *ecdsa.PublicKey
  ecdsaKey, ok := key.(*ecdsa.PublicKey)
  if !ok {
    fmt.Println("GitHub key was not ECDSA, what are they doing?!")
    os.Exit(7)
  }

  // Parse the Webhook Signature
  parsedSig := asn1Signature{}
  asnSig, err := base64.StdEncoding.DecodeString(kSig)
  if err != nil {
    fmt.Printf("unable to base64 decode signature: %s\n", err)
    os.Exit(8)
  }
  rest, err := asn1.Unmarshal(asnSig, &parsedSig)
  if err != nil || len(rest) != 0 {
    fmt.Printf("Error unmarshalling asn.1 signature: %s\n", err)
    os.Exit(9)
  }

  // Verify the SHA256 encoded payload against the signature with GitHub's Key
  digest := sha256.Sum256([]byte(payload))
  keyOk := ecdsa.Verify(ecdsaKey, digest[:], parsedSig.R, parsedSig.S)

  if keyOk {
    fmt.Println("THE PAYLOAD IS GOOD!!")
  } else {
    fmt.Println("the payload is invalid :(")
    os.Exit(10)
  }
}

type GitHubSigningKeys struct {
  PublicKeys []struct {
    KeyIdentifier string `json:"key_identifier"`
    Key           string `json:"key"`
    IsCurrent     bool   `json:"is_current"`
  } `json:"public_keys"`
}

// asn1Signature is a struct for ASN.1 serializing/parsing signatures.
type asn1Signature struct {
  R *big.Int
  S *big.Int
}

Validation sample in Ruby

require 'openssl'
require 'net/http'
require 'uri'
require 'json'
require 'base64'

payload = <<-EOL
[{"token": "some_token", "type": "some_type", "url": "some_url"}]
EOL

payload = payload

signature = "MEUCICop4nvIgmcY4+mB+i5GlUYGNL20Qrlrx3RvrysilFHeAiEAg7yJ8KqEUcUadBaxepp3COhTUrk4feZ9TTb/xdBG6Ek="

key_id = "3ec68716d6df3f7cd532ac97e55420cb1c143752450d65bf916d41ca25d9dfc4"

url = URI.parse('https://api.github.com/meta/public_keys/token_scanning')

request = Net::HTTP::Get.new(url.path)
request['Authorization'] = "Bearer #{ENV['GITHUB_PRODUCTION_TOKEN']}"

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = (url.scheme == "https")

response = http.request(request)

parsed_response = JSON.parse(response.body)

current_key_object = parsed_response["public_keys"].find { |key| key["key_identifier"] == key_id }

current_key = current_key_object["key"]

openssl_key = OpenSSL::PKey::EC.new(current_key)

puts openssl_key.verify(OpenSSL::Digest::SHA256.new, Base64.decode64(signature), payload.chomp)

Implement token revocation and user notification in your token alert service

Finally, you can enhance your token alert service to revoke the exposed tokens and notify the affected users. How you implement this in your token alert service is up to you, but we recommend considering any tokens that GitHub sends you messages about as public and compromised.