Restoring a service from a backup


Please pick the date that works for you for a session starting at 2pm and ending at 4pm:

  • 2020-09-10T22:00:00Z
  • 2020-09-17T22:00:00Z
  • 2020-09-24T22:00:00Z
  • 2020-10-01T22:00:00Z
  • 2020-10-08T22:00:00Z
  • 2020-10-15T22:00:00Z

0 voters

This is an online disaster recovery :volcano: exercise to practice restoring a service from a backup. It will happen on a shared tmux session and anyone willing to attend is welcome. However, only one person will get to actually type the commands. If you’re interested, please register your SSH public key :old_key: at and let me know your account name.

The duration of the session is likely to be around two hours, however most of the time will be waiting for provisioning to complete. It is recommended to plan for something easily interruptible to keep you busy in the meantime: there won’t be someone talking the whole time to entertain the audience :roll_eyes:

An Enough instance will be prepared for the occasion at, with a Mattermost instance running within a VPN.

Disaster recovery


  • Three OpenStack regions @ OVH
    • GRA5 for the VM running tmux
    • SBG5 for the Enough instance
    • DE1 for restoring the backup
  • Manually create a virtual machine @ GRA5
  • ssh to the virtual machine @ GRA5
  • Setup the DNS
    • enough --domain service create bind
    • update glue record @ gandi with the IP of bind-host
    • wait for propagation otherwise it will fail because letsencrypt fails

  • Setup the VPN
cat > ~/.enough/<<EOF
  - client0
  • enough --domain service create --host bind-host openvpn
  • enough --domain service create --host bind-host backup
  • apt-get install openvpn
  • cd /etc/openvpn/client
  • tar -zxvf ~/.enough/
  • systemctl start openvpn@client0
  • ping bind-host private IP

  • Setup the chat:
cat > ~/.enough/ <<EOF
openstack_network: internal
certificate_authority: ownca
  - name: chat-volume
    size: 5
encrypted_device_mount_point: /srv
openstack_network: internal
  • enough --domain service create chat
  • enough --domain openstack server reboot chat-host
  • enough --domain service create chat
  • enough --domain ssh chat-host

@nqb which time / date would work best for you in September ? I’ll check with the other participant. If there is no trivial solution I’ll launch a date poll.


If possible, on a Friday afternoon, starting at 2:00pm CEST.

Do you have an estimate duration ?

1 Like

It takes about two hours but requires only occasional interactions so you should have something else, easily interruptible, to keep you busy. Probably 30 minutes max to get started, explanations included. Then about 30 minutes max with 15 minutes interval waiting for commands to complete. So that would be about 10 minutes work, 15 minutes wait, times 3. That’s a pessimistic scenario.

I added a date poll in the topic. The other participant will be back from vacation September 3rd and I’ll invite him here.