In this ADDcast, Andrew Katsigiannis, Director of IT and Customer Support at ADD Systems, and Kaizad Madon, ADD Energy/E360 Product Manager at ADD Systems, discuss backup best practices to ensure viability when you need them most. They discuss the difference between virtual machine and database backups, how often each should be run, and how they fit in an overall cybersecurity strategy.
You can subscribe to the ADDcast at any of the following locations, or read the transcript below.
Apple Podcasts / iTunes: https://podcasts.apple.com/us/podcast/add-systems-addcast/id1604865113
Spotify: https://open.spotify.com/show/1atj72i5AnIwTJDBFD9DzX
Amazon / Audible: https://music.amazon.com/podcasts/df7c9f4a-8a3f-431f-8787-50eee85b2bcd
Castbox: https://castbox.fm/channel/id4744519?country=us
You can also search “ADD Systems” wherever you listen to your favorite podcasts!
Brian Cohen: Welcome to ADDcast. I’m ADD Systems Multimedia Specialist, Brian Cohen. Joining me today are Andy Katsigiannis, Director of IT and Customer Support, and Kaizad Madon, ADD Energy/E360 Product Manager at ADD Systems. Gentlemen, thank you both for joining me today.
Andy Katsigiannis: Thank you, Brian.
Kaizad Madon: Thank you for having us.
Brian: Andy, can you start by telling everyone about your role here at ADD Systems?
Andy: Sure. I am the Director of IT and Customer Support. On a day-to-day basis, I oversee the operations department and the customer support department, as well as get my hands dirty in any projects that come along my way.
Brian: And Kaizad, same question to you.
Kaizad: I serve as the product manager for the E360 product. I also still work on the, I guess, desktop-based version of the E3 product, and I work extensively and very intimately with the operations group on a very regular basis, I would say.
Andy: Very true. Kaizad is a significant asset for our company.
Brian: Excellent. And I’d like to start by asking this first question to both of you – why are backups, especially VM or SQL backups, so important? Andy, we could start with you.
Andy: Okay, so a VM backup, that’s another acronym for a virtual machine backup. It protects the entire server, including the operating system, applications, configurations, etc. A SQL backup, which is for the SQL database, focuses on the database itself, the business data, transactions, records, etc. Together with both of those, they ensure how we can recover both the system and the data that runs through it.
Kaizad: The SQL backup, if you want a simple analogy. Now, by SQL, when I say it doesn’t necessarily mean SQL Server, but any database, consider that as a very fancy file-level backup. So, like Andy said, if you want to recover something, you don’t need to recover the whole operating system or the whole machine. You can recover parts of it, namely in this case, the database or certain database logs, and get tables and columns and whatever data you want back from it. So it saves you from doing a full restore and losing other stuff that you might have done on the system, while just getting you the data that you need for a more specific point in time, if you will.
Andy: A good way to think of it is like that. So a virtual machine backup restores the vehicle, the entire vehicle, the full server environment, where the SQL backup, it just restores the fuel of that vehicle so the data that powers the application. So in order to have a sophisticated and successful backup, you really need both to get your business back up and running quickly.
Brian: So how often should VM backups be run?
Andy: At minimum once a day. Many critical systems are backed up several times daily. The more frequent data changes, the more frequent the backup should be made to minimize the data loss in just a few hours, or even few minutes, depending on how many backups are taken on a daily basis.
Brian: And Kaizad, is that the same timeline for SQL backups?
Kaizad: At a minimum, yes. At least once a day, you should be doing them. That would be…so SQL has the option of different levels of backup. They have the full backup, an incremental backup, and the log backup, depending on how a recovery model is set for the database. So speaking in terms of Microsoft SQL Server, there are two recovery models, simple and full. Simple only allows you to do full backups, which means you can’t take parts of a backup and can’t do increments. You take the whole thing, and the next backup, again, takes the whole thing. For a full recovery model, you have to do full backups, and then you do incrementals on top of it, or incrementals or differentials, depending on who you’re talking to. Terminology changes. And then you do your transaction log backups. Full backups should be at least once a day, and incrementals and logs based on what your retention periods and RPOs (Recovery Point Objectives) are. In our case for critical systems, we have our fulls on a nightly basis. We have incrementals running every four hours, and the transaction logs range anywhere between 15 minutes to 30 minutes.
Brian: Great. And this next question will be for both of you again. How do you test backups, and what happens if backups aren’t tested regularly? Andy, we could start with you here.
Andy: Sure. Thanks, Brian. A backup that isn’t tested is just a theory. So here at ADD Systems, we routinely perform test restores. We do monthly restores and then we do quarterly restores. Both require different levels of verification to make sure that the backups work. Otherwise, you might just discover during an outage that the backups you have been performing all this time are just not usable.
04:58
Kaizad: Backups without testing and verification are pretty much useless. If you can’t use them, there’s no point in backing stuff up. We’re just wasting our time and resources doing it, and testing from backup to backup varies. So sometimes you have to go back into the application, make sure the application is running at a VM level. You would go make sure the OS (operating system) is running, all the applications are there. At a database level, you would, the developer or a DBA would go into the database to make sure that the data that they expect to be in those tables is in good shape and the database itself is not corrupted, so you could run checks on the database after the restore as well.
Brian: Andy, how do backups help during, let’s say, a ransomware attack?
Andy: That’s super important. Backups are really your final line of defense if ransomware encrypts your systems. If you have a good, valid backup, they can use that to restore from secure, off-site backups. The same applies to power outages, storms, hardware failures. Essentially, backups let us rebuild quickly and avoid major disruptions. So here at ADD Systems, we have our data backed up to five separate locations, with the importance of not getting encrypted should a ransomware event occur.
Brian: And Kaizad, what are the biggest mistakes companies make when it comes to backups?
Kaizad: I’d say one of the biggest ones is assuming that your backups are good. Second is you back it up, but you don’t know exactly how to restore it, or you run into trouble restoring it. A good example of that is, for SQL Server backups that use a full recovery model, it’s pretty straightforward to perform a backup of a full and incremental, and transaction logs, but what a novice DBA or a developer might not realize is, you can’t just restore it automatically. You have to roll the logs in manually, so you have to start from your full, then do the incremental, and then roll all the logs in up to the point you need it.
Going back to the prior question of ransomware, right? How often do you do your backups? It leads into that to say, let’s say your backup runs at 2 a.m. every night. If you get hit by a ransomware attack, let’s say at 10 p.m. and they start encrypting your data, the only way to get back is your last good backup, which could be the night before at 2 a.m. or if you have incrementals running during the day, you could recover your entire day without losing any work for anyone. So it’s good to have backups, but it’s also good to have regular backups, and as the more frequent your backups, the quicker your recovery and the less wasted time for everybody involved, mainly the people that are doing the actual work during the day,
Andy: I agree with that. And to add on to some of the mistakes that some companies make is, to your point, Kaizad, not testing, assuming backups work without any type of verification, which is what we discussed previously. And then a couple more things, like keeping backups in the same place as your production data, makes both vulnerable, right? To malware, to storms, to anything physical. And lastly, kind of falling in the same idea of what we’ve been talking about, treating backups as this, set-it-and-forget-it type of thing. Backups need attention. Backups need monitoring. Backups need validation. Backups need regular review. Backups need to be able to have some good people knowing how to actually do the restores, just like Kaizad said.
Kaizad: They should never be considered an afterthought, right? They have to be within your initial design of when you implement something new. It can’t be, yeah, we’re going to start using this, and then we’ll figure out how to back it up. It can’t be an afterthought. And like Andy said, you don’t want to keep your backups in the same place as the production data. You either move them off-site, either physically air-gapped or electronically air-gapped (physically or logically separated from the primary system), or make them essentially the electronic air gap is an immutable property that gets applied, so you can’t delete or change the data until that backup set expires. So that’s also critical with the backups – make sure nobody else can change them.
Brian: So, Andy, how do backups fit into a company’s overall cybersecurity strategy?
Andy: That’s a crucial part of it. I mean, of the bigger picture, if you think about it. Like you mentioned before, backups are the final line of defense, right? Cybersecurity isn’t just about preventing attacks; it’s also about recovery. Backups are what let a company bounce back when something goes wrong, alongside firewalls and multifactor authentication, monitoring, backups are the last but most dependable safety layer. And to the previous point of having the backups in multiple locations and not just having it in someone else’s cloud, right? We want to make sure that we back up locally, back up to the cloud, and then back up to at least at another location.
Kaizad: I think if we want to take our specific examples, we have everything backed up on site. That gets replicated to another site, which then gets replicated, and then we have month ends and year ends replicated all over again. So it’s essentially the five copies that we have of our backups, and they are critical. I mean, literally, like Andy said, it is the last line of defense. If everything else fails, without backups, your company’s not coming back to life. You’ve pretty much lost everything. You have to either redo it or you’re going out of business.
Brian: So obviously, very important. Now this last question I’d like to ask you both, what questions should leadership be asking IT when it comes to backups?
Andy: That’s a really good question. So sometimes the brass of the company really don’t know the scheduling, but they should know on a high level, right? So some of the questions that they should be asking their IT groups or their MNSPs (Managed Network Service Providers) are, how often are backups running, and are they tested? How long would it take to recover a critical system? Where are the backups stored? Right? Where is our private data stored? Are they stored somewhere in someone else’s cloud? If so, is that being protected from ransomware? And then the last and most important thing is, who’s checking these backups on a daily basis? Who’s checking the virtual machine backups? Who’s checking the SQL Server backups on a daily basis?
Kaizad: Yeah. I mean, those are all valid. It’s also not when your backups run, but how long they run as well.
Andy: Good point.
Kaizad: Because if your backups are going to run for like 12 hours in a day, it might eat into business hours, and that’s going to cause business disruptions. The other critical piece, like Andy said, is, you know, where are they stored, and is that also protected? You know, if you’re storing it in some unknown cloud, what cybersecurity policies do they have? What security policies they have in place? Also, the key is, you know, you have somebody doing your backups on a daily basis. Make sure that when that person is out, somebody can look at the instructions, pick it up, and follow it for a backup or a restore, documenting what happens, how it happens. Procedure is critical, especially when you have to recover something.
Brian: Well, gentlemen, I appreciate you both taking the time to speak with me today on this.
Andy: Hey, Brian, thanks. Anytime.
Kaizad: Thank you for having us.
Brian: To keep up with the latest happenings at ADD Systems, visit addsys.com/blog, or connect with us on social media by following ADD Systems on LinkedIn, Facebook, Instagram or X. If you have any questions about ADDcast, feel free to reach out to us at addcast@addsys.com. Thanks for listening and have a great day.
Leave a Reply