Restic Python Backup to S3 Compatible Storage

Restic Python Backup to S3 Compatible Storage

Posted:  April 17, 2021

Restic Python Backup to S3 Compatible Storage


In my quest to get my servers fully automated, I realized I needed a reliable way to backup and restore my sites and databases.  Including a way to either browse the backups, or restore them to any other machine I have access to.

At first I thought… mmm…  duplicity seems nice, and while generally it is a viable solution, it did not quite have the OOOMPH I was looking for in my backups, so, in steps Restic, and Python.

This post is going to be mostly instructions on how to use my backup/restore binary’s and scripts, so if you need some more information about restic, please see here:

Some of the backups are multi-threaded.

Description & Requirements

So, first things first, you are going to need a S3 compatible cloud object storage provider.  Something like Amazon’s S3 ( ) will suffice, however, I would highly recommend giving Wasabi ( ) a chance (you won’t be disappointed)

Once you are signed up, you will need to gather some info to configure your backups.

  • Your S3 API Key
  • Your S3 API Secret
  • Your S3 API Endpoint (just the FQDN)
  • Your S3 Region
  • Your S3 Bucket

Got all that?   Great, don’t forget that I have set this up for my web servers, so you should probably gather up your MySQL admin info (either username and password, or the defaults file path), the starting path to your servers applications, as well, the name you wish to use for these backups and the “hash” you want to use to encrypt the backups.

Also note, this is created for Ubuntu based web servers primarily, so make sure your server is at least Ubuntu 18.04.  You also need to make sure your server has at least Python 3.6 installed.

Installing & Updating

At this point, you can go ahead and download the binary.  Yes, I did compile it to be a single binary, however, I did have to compile it separately for Ubuntu 18.04 and Ubuntu 20.04, so make sure you grab the right one for your distro.

Once you have it downloaded to your server, change the permissions on it to make it executable, rename it to kpbr, and move it to your /usr/bin/ directory.    This will make it that much easier to run it, especially when you need it for a cron job 🙂

Now, let’s get everything setup and ready to use.  Run the following command and pay attention.   kpbr -a setup  (NOTE: you’ll be doing this when I update the app too…   same download link, same command…).  It will get your server ready with the required modules and software to run.  (I’ll list that all out below…)

Once the setup is completed, it will automatically ask you to configure the app.  Just follow the prompts and answer the questions to finish… please do note, the hashes, keys & secrets are indeed secret, so you won’t be seeing them on the screen as you are typing or copying/pasting the information.

Backing Up and Restoring

Now that you are configured you can run your first backup.  Simply run the command: kpbr -a backup --what all to backup all web applications and databases on your server.

Now, if you need to restore an app or database, you have a few options.   You can do an automated restore, you can restore to a different location, or you can even mount the backups as a network drive and browse the files, you can also restore a backup from another machine if you want.  Just run the following command: kpbr -a restore and pay attention, you will be prompted to select what exactly it is you want to do.

Now for the nitty gritty of it:  I have the backups tested on both flavors of Ubuntu Server and Desktop, as well in the equivalent versions in WSL on Windows 10.   I also have the restores tested on the above as well.   I have tested the restores on Mac OS, however, have found there are too many differences in the Python versions and the modules for it to be able to support it (please I do not readily have access to a Mac), however, I have setup a bash script to allow you to be able to mount the backups as a network drive on a Mac.

The full readme, and command set is below, along with the repository of the source code.   If you do have any issues with this, please feel free to file an issue at the repo, just make sure you give me as much detail as possible.  I’ll at least need to know what OS you were using, along with what command you were running.


Operating System

    • Ubuntu 18.04+
    • Windows 10
      • Works and tested in WSL Ubuntu 18.04 & Ubuntu 20.04
      • Minor visual glitch with the select menus that go away if you go up and down once

Server Software & Binaries

    • NOTE: Do not run kpbr -a setup on Mac or Windows, instead please look up the proper documentation to install the proceeding packages for your system.
    • Python3.6+
      • Python dateutil, boto3, enquiries, mysqldb, invoke, & requests modules
      • for backing up & restore
    • restic
      • for backing up & restore
      • for restore/mount script
    • aws-cli
      • for restore selection
      • for restore/mount script
    • jq
      • for restore/mount script


For ubuntu, download the release binary associated to your flavor from here:

rename it to kpbr, make it executable, and move it to /usr/bin/

Once there, run sudo kpbr -a setup, it will install or update everything your server will need to run.

    • All in one command – Ubuntu 18.04: sudo wget -O kpbr && sudo chmod +x kpbr && sudo mv kpbr /usr/bin/ && sudo kpbr -a setup
    • All in one command – Ubuntu 20.04: sudo wget -O kpbr && sudo chmod +x kpbr && sudo mv kpbr /usr/bin/ && sudo kpbr -a setup


Download and unzip the from here: You may need to make the 2 files executable.

Once expanded open and configure it. For the BACKUP_REPOS, it is a JSON string array. Follow the example to create more nodes.

USAGE – kpbr

    kpbr -a [setup|config|backup] [ARGS]

    Required Arguments Per Action:

                Will prompt you to configure or re-configure once the setup has been completed.

                Will ask you questions to configure the app.  Please pay attention carefully.

            --what [all|db|database|acct|account|app|other]
                    NONE - also performs a full database backup
                    --db or --database THE DATABASE NAME | ALL - REQUIRED 
                    --acct or --account THE ACCOUNT NAME - REQUIRED 
                    --acct or --account THE ACCOUNT NAME - REQUIRED 
                    --app THE APP NAME - REQUIRED 
                    --paths can also be passed for all|db/database|acct/account|app as well to add additional paths to backup

            NONE: Everything will be prompted for you.
                Backup Location:
                    Local: this is the local server's backup.  It will pull everything needed from the configured config file
                    Remote: this is a remote servers backup.  You will have to enter the S3 API Key, S3 API Secret, S3 Endpoint, S3 Bucket, S3 Region, backup name/server, and the backup hash.   Make sure you have all this information handy, it is all required.
                Type of Backup:
                    Application: Otherwise known as the web application, set of files to be restored.
                        Addition Prompts:
                            Account: The account to be restored
                            Application: The application in the chosen account to be restored
                            Snapshot: The specific backup point you would like to restore.
                    Database: The database to be restored
                        Addition Prompts:
                            Database: The account to be restored
                            Snapshot: The specific backup point you would like to restore.
                Restore Method:
                    Automatic: This will overwrite the originals, so make sure you really want to do this.
                    Manual: This will restore the files to the specified path.  This includes the mysqldump if Database has been chosen as the type.
                        Addition Prompts:
                            Restore Path: The files will be restored to this path and be left alone for you to do with them as you see fit. The default is `/tmp/restore`
                    Browse: This will mount the backup as a network drive, allowing you to browse the files as you see fit.  The default mount point is `/tmp/backup-mount` however you can specify whatever path you see fit.
                        Addition Prompts:
                            Mount Point: The mount point that the backup will be mounted to, allowing you to browse the files as you see fit.  The default mount point is `/tmp/backup-mount`


You can mount a backup as a networked drive.

Configure by opening and entering your AWS Key and Secret, the S3 Bucket, S3 Region, and S3 Endpoint (for aws this is

Then add your server and server hash to the json nodes. name = the server or backup name, hash = the server hash

Once configured run the script then browse to /tmp/backup-mount/ on your machine. Inside it will contain the backups you selected. snapshots is the timestamped snapshots of the specified backup, with a symlink to the latest


Feel free to fork this, and if you run into any issues with it please fire up and issue at the repo so I can track it.

Thanks for looking and Happy Hosting!

Kevin Pirnie

Kevin Pirnie

20+ Years of PC and server maintenance & over 15+ years of web development/design experience; you can rest assured that I take every measure possible to ensure your computers are running to their peak potentials. I treat them as if they were mine, and I am quite a stickler about keeping my machines up to date and optimized to run as well as they can.

Our Privacy Policy

Revised: June 8, 2021

Thank you for choosing to be part of my website at (“Company”, “I”, “me”, “mine”). I am committed to protecting your personal information and your right to privacy. If you have any questions or concerns about this privacy notice, or my practices with regards to your personal information, please contact me at .

When you visit my website (the “Website”), and more generally, use any of my services (the “Services”, which include the Website), I appreciate that you are trusting me with your personal information. I take your privacy very seriously. In this privacy notice, I seek to explain to you in the clearest way possible what information we collect, how I use it and what rights you have in relation to it. I hope you take some time to read through it carefully, as it is important. If there are any terms in this privacy notice that you do not agree with, please discontinue use of my Services immediately.

This privacy notice applies to all information collected through my Services (which, as described above, includes our Website), as well as, any related services, sales, marketing or events.

Please read this privacy notice carefully as it will help you understand what I do with the information that I collect.


Information automatically collected

In Short: Some information – such as your Internet Protocol (IP) address and/or browser and device characteristics – is collected automatically when you visit my Website.

I automatically collect certain information when you visit, use or navigate the Website. This information does not reveal your specific identity (like your name or contact information) but may include device and usage information, such as your IP address, browser and device characteristics, operating system, language preferences, referring URLs, device name, country, location, information about how and when you use our Website and other technical information. This information is primarily needed to maintain the security and operation of our Website, and for our internal analytics and reporting purposes.

Like many businesses, I also collect information through cookies and similar technologies.

The information I collect includes:
Log and Usage Data. Log and usage data is service-related, diagnostic, usage and performance information our servers automatically collect when you access or use our Website and which we record in log files. Depending on how you interact with us, this log data may include your IP address, device information, browser type and settings and information about your activity in the Website (such as the date/time stamps associated with your usage, pages and files viewed, searches and other actions you take such as which features you use), device event information (such as system activity, error reports (sometimes called ‘crash dumps’) and hardware settings).

Device Data. I collect device data such as information about your computer, phone, tablet or other device you use to access the Website. Depending on the device used, this device data may include information such as your IP address (or proxy server), device and application identification numbers, location, browser type, hardware model Internet service provider and/or mobile carrier, operating system and system configuration information.

Location Data. I collect location data such as information about your device’s location, which can be either precise or imprecise. How much information I collect depends on the type and settings of the device you use to access the Website. For example, I may use GPS and other technologies to collect geolocation data that tells me your current location (based on your IP address). You can opt out of allowing me to collect this information either by refusing access to the information or by disabling your Location setting on your device. Note however, if you choose to opt out, you may not be able to use certain aspects of the Services.


In Short: I process your information for purposes based on legitimate business interests, the fulfillment of my contract with you, compliance with my legal obligations, and/or your consent.

I use personal information collected via my Website for a variety of business purposes described below. I process your personal information for these purposes in reliance on my legitimate business interests, in order to enter into or perform a contract with you, with your consent, and/or for compliance with my legal obligations. I indicate the specific processing grounds I rely on next to each purpose listed below.

For other business purposes. I may use your information for other business purposes, such as data analysis, identifying usage trends, determining the effectiveness of our promotional campaigns and to evaluate and improve my Website, products, marketing and your experience. I may use and store this information in aggregated and anonymized form so that it is not associated with individual end users and does not include personal information. I will not use identifiable personal information without your consent.


In Short: I only share information with your consent, to comply with laws, to provide you with services, to protect your rights, or to fulfill business obligations.


In Short: I may use cookies and other tracking technologies to collect and store your information.

I may use cookies and similar tracking technologies (like web beacons and pixels) to access or store information. Specific information about how I use such technologies and how you can refuse certain cookies is set out in our Cookie Notice.


In Short: We may transfer, store, and process your information in countries other than your own.

My servers are located in the United States of America, unless otherwise requested by my clients. If you are accessing my Website from outside, please be aware that your information may be transferred to, stored, and processed by me in my facilities and by those third parties with whom I may share your personal information (see “WILL YOUR INFORMATION BE SHARED WITH ANYONE?” above), in and other countries.

If you are a resident in the European Economic Area, then these countries may not necessarily have data protection laws or other similar laws as comprehensive as those in your country. I will however take all necessary measures to protect your personal information in accordance with this privacy notice and applicable law.


In Short: I keep your information for as long as necessary to fulfill the purposes outlined in this privacy notice unless otherwise required by law.

I will only keep your personal information for as long as it is necessary for the purposes set out in this privacy notice, unless a longer retention period is required or permitted by law (such as tax, accounting or other legal requirements). No purpose in this notice will require me keeping your personal information for longer than 6 months.

When I have no ongoing legitimate business need to process your personal information, I will either delete or anonymize such information, or, if this is not possible (for example, because your personal information has been stored in backup archives), then I will securely store your personal information and isolate it from any further processing until deletion is possible.


In Short: I aim to protect your personal information through a system of organizational and technical security measures.

I have implemented appropriate technical and organizational security measures designed to protect the security of any personal information I process. However, despite our safeguards and efforts to secure your information, no electronic transmission over the Internet or information storage technology can be guaranteed to be 100% secure, so I cannot promise or guarantee that hackers, cybercriminals, or other unauthorized third parties will not be able to defeat my security, and improperly collect, access, steal, or modify your information. Although I will do my best to protect your personal information, transmission of personal information to and from my Website is at your own risk. You should only access the Website within a secure environment.


In Short: I do not knowingly collect data from or market to children under 18 years of age.

I do not knowingly solicit data from or market to children under 18 years of age. By using the Website, you represent that you are at least 18 or that you are the parent or guardian of such a minor and consent to such minor dependent’s use of the Website. If I learn that personal information from users less than 18 years of age has been collected, I will deactivate the account and take reasonable measures to promptly delete such data from my records. If you become aware of any data I may have collected from children under age 18, please contact me at .


In Short: You may review, change, or terminate your account at any time.

If you are a resident in the European Economic Area and you believe I am unlawfully processing your personal information, you also have the right to complain to your local data protection supervisory authority. You can find their contact details here:

If you are a resident in Switzerland, the contact details for the data protection authorities are available here:

Cookies and similar technologies: Most Web browsers are set to accept cookies by default. If you prefer, you can usually choose to set your browser to remove cookies and to reject cookies. If you choose to remove cookies or reject cookies, this could affect certain features or services of my Website.


Most web browsers and some mobile operating systems and mobile applications include a Do-Not-Track (“DNT”) feature or setting you can activate to signal your privacy preference not to have data about your online browsing activities monitored and collected. At this stage no uniform technology standard for recognizing and implementing DNT signals has been finalized. As such, I do not currently respond to DNT browser signals or any other mechanism that automatically communicates your choice not to be tracked online. If a standard for online tracking is adopted that I must follow in the future, I will inform you about that practice in a revised version of this privacy notice.


In Short: Yes, if you are a resident of California, you are granted specific rights regarding access to your personal information.

California Civil Code Section 1798.83, also known as the “Shine The Light” law, permits my users who are California residents to request and obtain from me, once a year and free of charge, information about categories of personal information (if any) I disclosed to third parties for direct marketing purposes and the names and addresses of all third parties with which I shared personal information in the immediately preceding calendar year. If you are a California resident and would like to make such a request, please submit your request in writing to me using the contact information provided below.

If you are under 18 years of age, reside in California, and have a registered account with the Website, you have the right to request removal of unwanted data that you publicly post on the Website. To request removal of such data, please contact us using the contact information provided below, and include the email address associated with your account and a statement that you reside in California. I will make sure the data is not publicly displayed on the Website, but please be aware that the data may not be completely or comprehensively removed from all my systems (e.g. backups, etc.).


In Short: Yes, I will update this notice as necessary to stay compliant with relevant laws.

I may update this privacy notice from time to time. The updated version will be indicated by an updated “Revised” date and the updated version will be effective as soon as it is accessible. If I make material changes to this privacy notice, I may notify you either by prominently posting a notice of such changes or by directly sending you a notification. We encourage you to review this privacy notice frequently to be informed of how I am protecting your information.


If you have questions or comments about this notice, you may email me at or by post to:

Kevin C. Pirnie

22 Orlando St.
Feeding Hills, MA 01030
United States of America


Based on the applicable laws of your country, you may have the right to request access to the personal information I collect from you, change that information, or delete it in some circumstances. To request to review, update, or delete your personal information, you may email me at or by post to:

Kevin C. Pirnie

22 Orlando St.
Feeding Hills, MA 01030
United States of America