Wild Card Certs

Presenter Notes

Lab Network

Network

Presenter Notes

DNS

Public Records

  • example.com
    Normal Public Domain
  • home.example.com
    Public Record that points to ISP's Dynamic IP or CNAME to a DYNdns entry. A real nice service is Duck DNS - (https://www.duckdns.org/)
  • *.app.example.com
    A DNS entry that will take any entry and point to a single place

Internal

  • [SomeHost].home.example.com
    Private subdomain only resolvable inside lab. All hostnames fall into this domain also.
  • [SomeApp].app.example.com
    Application domain that is both known inside and publically. That way a device can use the same URL to access the resource regardless on which network you are on

Presenter Notes

Subdomain makes SSH easier

1Host *.home.example.com
2  ProxyJump home.example.com

Presenter Notes

Why Certs

  • Browser bypasses are getting more annoying
  • Browsers are starting to limit how long long certs can be valid
    ie: All Certs made after September 2020 can valid for no more than 398 days

Presenter Notes

LetsEncrypt

Born in 2012 paved the road for browsers to be less tollerant of wrong certs

  • Free DV (Domain Validation) Certs
  • Automatic (Kinda has to be since it is free)
  • Secure
  • Transparent
  • Open Protocol (ACME)
  • Started supporting Wild Card Certs January of 2018

Presenter Notes

ACME Protocol RFC-8555

acme

Presenter Notes

ACME Clients and Servers

  • Many Client Options accross many languages (https://letsencrypt.org/docs/client-options/)
  • CertBot is the client we are using
  • Server Side ACME implemenation Boulder and Pebble
  • Pebble is a light server built mostly for client testing

Presenter Notes

Why Wildcard Certs

  • Do not want to manage a bunch of certs for each domain
  • Handy for ephermal hosts / containers
  • keep the complexities on a single host
  • build process only has to copy files, no need for validation cycle
  • No network shenagangans to route the traffic to the host being challenged
  • Some home ISPs (Cox and Comcast) actully block TCP/80 traffic inbound

Presenter Notes

Certbot is far from lightweight

 1==================================================================
 2Package                    Arch    Version         Repo     Size
 3==================================================================
 4Installing:
 5certbot                    noarch  2.5.0-1.fc37    updates  17 k
 6Installing dependencies:
 7dbus                       aarch64 1:1.14.6-1.fc37 updates 7.7 k
 8dbus-broker                aarch64 33-1.fc37       updates 168 k
 9dbus-common                noarch  1:1.14.6-1.fc37 updates  15 k
10--------------------8<--------------------------8<------------------
11device-mapper              aarch64 1.02.175-9.fc37 fedora  136 k
12device-mapper-libs         aarch64 1.02.175-9.fc37 fedora  169 k
13systemd-networkd           aarch64 251.14-2.fc37   updates 571 k
14systemd-resolved           aarch64 251.14-2.fc37   updates 262 k
15
16Transaction Summary
17==================================================================
18Install  39 Packages

Presenter Notes

Using CertBot

Pros

  • First Class support for API changes and new features
  • Easy set up (Packages for most Platforms)

Cons

  • Install ~40 packages

Presenter Notes

Challenge Types

  • HTTP-01 challenge
    Access FROM the internet needed to the site on BOTH TCP80 & TCP443 for challengs
  • DNS-01 challenge (no network access to the resource needed)
  • TLS-SNI-01 -> Deprecated March 2019 (This ment 80/443 BOTH needed)
  • TLS-ALPN-01 -> Low / No Support and No WildCard Support

Presenter Notes

HTTP-01

http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN>

Presenter Notes

DNS-01

Pros

  • You can use this challenge to issue certificates containing wildcard domain names.
  • It works well even if you have multiple web servers.

Cons:

  • Keeping API credentials on your web server is risky.
  • Your DNS provider might not offer an API.
  • Your DNS API may not provide information on propagation times.

Presenter Notes

Manual Auth Hook

--manual-auth-hook passes two ENV Variables:

  • $CERTBOT_DOMAIN
  • $CERTBOT_VALIDATION

Presenter Notes

Authhook Example

 1#!/bin/sh
 2
 3KEY="SomeKey"
 4SECRET="SomeSecret"
 5
 6APISITE="api.godaddy.com"
 7
 8BASEDOMAIN=`echo ${CERTBOT_DOMAIN} | awk -F. '{OFS="."; print $(NF-1),$(NF)}'`
 9SUBDOMAIN=`echo ${CERTBOT_DOMAIN} | sed -e 's/'${BASEDOMAIN}'//' -e 's/\.$//'`
10
11if [ -z ${SUBDOMAIN} ] ; then
12    DNSRECORD="_acme-challenge"
13else
14    DNSRECORD="_acme-challenge.${SUBDOMAIN}"
15fi
16
17UpDateDNS(${BASEDOMIN} ${DNSRECORD} ${CERTBOT_VALIDATION})

Initial Creation

1certbot certonly --manual \
2-d *.example.com  \
3-d example.com \
4--preferred-challenges dns \
5--manual-auth-hook /etc/letsencrypt/CertBotGoDaddyAuthHook.sh

Presenter Notes

Check Status

Running certbot certificates command

 1- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 2Found the following certs:
 3  Certificate Name: example.com
 4    Serial Number: 4ed92b171a54e8ef6ecc98b8c9d13dd08b8
 5    Key Type: RSA
 6    Domains: *.example.com example.com
 7    Expiry Date: 2023-08-18 12:23:26+00:00 (VALID: 89 days)
 8    Certificate Path: /etc/letsencrypt/live/example.com/fullchain.pem
 9    Private Key Path: /etc/letsencrypt/live/example.com/privkey.pem
10  Certificate Name: test.example.com
11    Serial Number: fa4806d9253983a3cbf5cbe0df606de4483d
12    Key Type: RSA
13    Domains: *.test.example.com
14    Expiry Date: 2023-08-18 12:51:05+00:00 (INVALID: TEST_CERT)
15    Certificate Path: /etc/letsencrypt/live/test.example.com/fullchain.pem
16    Private Key Path: /etc/letsencrypt/live/test.example.com/privkey.pem
17- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Presenter Notes

CertBot DB

/etc/letsencrypt/renewal/example.com.conf

 1# renew_before_expiry = 30 days
 2version = 1.21.0
 3archive_dir = /etc/letsencrypt/archive/example.com
 4cert = /etc/letsencrypt/live/example.com/cert.pem
 5privkey = /etc/letsencrypt/live/example.com/privkey.pem
 6chain = /etc/letsencrypt/live/example.com/chain.pem
 7fullchain = /etc/letsencrypt/live/example.com/fullchain.pem
 8
 9# Options used in the renewal process
10[renewalparams]
11account = 4a6d9352e2fc12db24d8d48208528151
12pref_challs = dns-01,
13authenticator = manual
14manual_auth_hook = /etc/letsencrypt/CertBotGoDaddyAuthHook.sh
15server = https://acme-v02.api.letsencrypt.org/directory

Presenter Notes

API Limits

Prod

  • The Certificates per Registered Domain limit is 50 per week.
  • The Duplicate Certificate limit is 5 per week.
  • The Failed Validations limit is 5 per hour.

Staging

  • The Certificates per Registered Domain limit is 30,000 per week.
  • The Duplicate Certificate limit is 30,000 per week.
  • The Failed Validations limit is 60 per hour.

Presenter Notes

Staging API

  • Production API - https://acme-v02.api.letsencrypt.org/directory
  • Staging API - https://acme-staging-v02.api.letsencrypt.org/directory

Overide Default API URL

1ACMEURL="https://acme-staging-v02.api.letsencrypt.org/directory"
2certbot --server ${ACMEURL} -d blah.example.com --manual

Stage: (also can be done by --test-cert)

Presenter Notes

Putting Certs to Use

Presenter Notes

Delivering Certs to Hosts

Three use cases for Ansible to solve

  • Load balancer
    Multuple domains on each loadbalancer
  • App Hosts
    single domin in seemingly random locations
  • haproxy
    When app is NOT on TCP/80 or TCP/443 and redirect to correct port. There need to have a cert so it can redirect https:// traffic comming into TCP/443

Presenter Notes

Inventory Load Balancer

Multiple certs that land into a single directory

 1loadbalancers:
 2  hosts:
 3    loadbalancer1.home.example.com:
 4      domains:
 5      - example.com
 6      - app.example.com
 7      - home.example.com
 8      certdir: /etc/nginx/certs
 9    loadbalancer2.home.example.com:
10      domains:
11      - example.com
12      - app.example.com
13      - home.example.com
14      certdir: /etc/nginx/certs

Presenter Notes

Inventory for Stand Alone Apps

Because every application seems to put cert files into seeming random places

 1appcerts:
 2  hosts:
 3    nessus.home.example.com:
 4      domain: home.example.com
 5      certfile: /opt/nessus/com/nessus/CA/servercert.pem
 6      keyfile: /opt/nessus/var/nessus/CA/serverkey.pem
 7      service: nessusd
 8      haproxy:
 9        enabled: true
10        cert_dest: /etc/haproxy/haproxy.pem
11    logs.home.example.com:
12      domain: home.example.com
13      certfile: /opt/splunk/etc/auth/splunkweb/cert.pem
14      keyfile: /opt/splunk/etc/auth/splunkweb/privkey.pem
15      service: splunk
16      haproxy:
17        enabled: true
18        cert_dest: /etc/haproxy/haproxy.pem
19    media.home.example.com:
20      domain: example.com
21      certfile: /etc/pki/tls/certs/example.crt
22      keyfile: /etc/pki/tls/certs/example.key
23      service: httpd

Presenter Notes

Ansible Playbook

Uses inventory

 1- name: Update App Certs
 2  hosts: appcerts
 3  become: true
 4  gather_facts: false
 5
 6  tasks:
 7    - name: Copy {{ domain }} cert to Host
 8      ansible.builtin.copy:
 9        src: /etc/letsencrypt/live/{{ domain }}/fullchain.pem
10        dest: "{{ certfile }}"
11      notify: Restart Service
12
13    - name: Copy {{ domain }} key to Host
14      ansible.builtin.copy:
15        src: /etc/letsencrypt/live/{{ domain }}/privkey.pem
16        dest: "{{ keyfile }}"
17      notify: Restart Service
18
19    - name: Create haproxy cert if enabled
20      ansible.builtin.assemble:
21        src: /etc/letsencrypt/live/{{ domain }}
22        dest: "{{ haproxy.cert_dest }}"
23        owner: "{{ haproxy.owner | default('root') }}"
24        group: "{{ haproxy.group | default('root') }}"
25        mode: "{{ haproxy.mode | default('0600') }}"
26        regexp: '(fullchain|privkey)\.pem$'
27        remote_src: false
28      when: (haproxy is defined) and (haproxy.enabled|bool == true)
29      notify: Restart Haproxy
30
31  handlers:
32    - name: Restart Service
33      service:
34        name: "{{ service }}"
35        state: restarted
36
37    - name: Restart Haproxy
38      service:
39        name: haproxy.service
40        state: restarted
41
42- name: Update Loadbalancer Certs
43  hosts: loadbalancers
44  gather_facts: false
45
46  vars:
47    service: nginx
48
49  tasks:
50    - name: Copy Cert to Host
51      ansible.builtin.copy:
52        src: /etc/letsencrypt/live/{{ item }}/fullchain.pem
53        dest: "{{ certdir }}/{{ item }}.crt"
54      notify: Restart Service
55      with_items: "{{ domains }}"
56
57    - name: Copy Chain Cert to Host
58      ansible.builtin.copy:
59        src: /etc/letsencrypt/live/{{ item }}/chain.pem
60        dest: "{{ certdir }}/{{ item }}.int"
61      notify: Restart Service
62      with_items: "{{ domains }}"
63
64    - name: Copy Private Key to Host
65      ansible.builtin.copy:
66        src: /etc/letsencrypt/live/{{ item }}/privkey.pem
67        dest: "{{ certdir }}/{{ item }}.key"
68      notify: Restart Service
69      with_items: "{{ domains }}"
70
71  handlers:
72    - name: Restart Service
73      service:
74        name: "{{ service }}"
75        state: restarted

Presenter Notes

Questions?

Presenter Notes

Cheat Sheet

 1Create New Cert:
 2ACMEURL="https://acme-v02.api.letsencrypt.org/directory"
 3certbot --server ${ACMEURL} -d blah.example.com --manual certonly
 4certbot --server ${ACMEURL} -d blah.example.com --manual --preferred-challenges dns certonly
 5
 6Stage:
 7ACMEURL="https://acme-staging-v02.api.letsencrypt.org/directory"
 8
 9Show Certs:
10certbot certificates
11
12Renew Certs:
13export CERTNAME=mycertname.com
14certbot renew
15certbot renew --cert-name ${CERTNAME} --dry-run
16certbot renew --force-renewal --cert-name ${CERTNAME} --dry-run
17certbot renew --manual --preferred-challenges dns --manual-auth-hook /etc/letsencrypt/CertBotGoDaddyAuthHook.sh
18
19Create PK12 File:
20DOMAIN="example.dev"
21CERT="/etc/letsencrypt/live/${DOMAIN}/fullchain.pem"
22KEY="/etc/letsencrypt/live/${DOMAIN}/privkey.pem"
23
24openssl pkcs12 -export -in ${CERT} -inkey ${KEY} -out ${DOMAIN}.p12

Presenter Notes