In many jurisdictions, there is a legislative or accounting requirement to store data for a minimum of five years. Easier said than done. That's why multiple data backups are necessary.
Ignoring the complexities involved in protecting our data from internal and external attacks, let’s look at the problems associated with successfully storing data (whether structured or unstructured) in the short and long-term.
I’ve been a PC user since the earlier 1980s and (knock on wood) have never experienced a cyber attack where my data was compromised. Sure, I’m not a viable target for concentrated attacks and am savvy enough to ignore phishing emails and certainly never click on links from unknown parties. However, I believe it’s down to overall security awareness as there are daily but blatant attempts from ‘hackers’ with poor vocabularies. It’s true that ransomware, viruses, and malware are threats to data integrity but protecting ourselves is not rocket science and is often achieved by using basic tools, many of them free.
As a non-enterprise user, I simply use Windows Defender, CCleaner, a variety of browser plugins (adblockers, Ghostery, Privacy Badger, HTTPS Everywhere) and a VPN to protect myself online. If feeling paranoid, I’ll even use Tor or a privacy-focused Linux distro.
My data issues, and I’m sure I’m not alone, lie in media failure, whether optical media (CD, DVD and Blu-Ray) or hard disc drives (HDDs). Memory cards, SSDs, memory sticks and magnetic media can also become problematic. In fact, the inspiration for the story lies in yet another 4Tb HDD failure (the dreaded clicking noise that indicates platter damage) earlier this week. It’s still under warranty but the annoyance factor is undeniable, considering it was purchased less than a year ago.
All Storage Formats Are Vulnerable
A cliché at this point but data is the lifeblood of a company; few can deny the importance of storing data, given the rise of data analytics, personalized and annoying marketing and other ‘advances’ designed to gain additional insights into customers, citizens, and markets. Essential data also requires storage, whether it’s to meet compliance or accounting requirements, retain proprietary documentation, or simply keep track of business operations and goals.
Unfortunately, despite our reliance on data, there is no single storage method that will allow us to maintain it indefinitely. It’s not a new problem, given that even episodes of popular TV shows such as Doctor Who and The Honeymooners have been lost forever over the years. Due to the cost of film stock, many shows were taped over once aired. Archivists and libraries face these issues all the time, but the business community also needs to consider the implications. How can your company protect its data and maintain a reliable data archiving process? It’s a tough one as, regardless of desired storage type, all have weaknesses.
LTO Tape Data Backups
Tape backup is the go-to solution for global enterprises and online analysis frequently highlights the potential cost savings per terabyte over other methods. It’s true that a 1Tb HDD retails for around $50 and an equivalent size tape is ten times cheaper. Why doesn’t everyone just use tape then? Because the cost of a single LTO-5 (three generations behind – an LTO-8 drive costs more) tape drive starts at around $1,000 and a cheap RAID/SAS controller card is another $200. Media cost varies by LTO type but regardless of type, correct storage is essential. Electromagnetic interference and humidity can destroy tapes. Tape works well for long-term storage if and only if you are prepared for the initial high investment. $1,000-plus represents a lot of hard drives.
HDDs
My current preferred medium. I tend to avoid the cheapest consumer-level drives but am too cost-conscious to select enterprise-level solutions. My choice: drives aimed at 24/7 video surveillance systems. Of the six 4Tb purchased, two failed within the 3-year warranty term. While I have drives much older (an 80Gb Hitachi, for example) that still work, it’s clear that larger capacity drives will fail, it’s just a matter of when.
Backblaze, in a detailed ongoing survey, clearly demonstrate that hard drive failure is expected. While these results are based on data center 24/7 usage, it’s generally assumed that hard drive failure in a typical office environment is likely after four years of usage. Power surges, vibration, and deterioration during the life span of the are all common problems of HDDs. There is a reason long-term warranties are not available on consumer drives.
Optical Media for Data Backups
Whether CD, DVD, or Blu-ray discs, good optical media for data backups is challenging to find. Bargain hunters who purchase cheap media are disappointed to discover that discs are unreadable after just a few uses. In optical media, you get what you pay for and even the best media is corrupted by sunlight, humidity, incorrect storage (near magnetic sources, for example) and scratching.
M-Disc media is expensive and claims a lifespan of 1,000 years although I’ve yet to verify that one. As the data is engraved onto advanced metals, it would seem to have potential. An M-Disc burner retails under $100, so it’s an affordable option. Unfortunately, as capacity grows in optical media, speed suffers. Expect write speeds of 6x at best. As with all optical media, it could take some time to find the media that works best with your burner and players, due to firmware incompatibility or variations in media quality. Fake optical media is very common.
Flash-based Storage
Whether it’s SSDs or memory sticks, the biggest issues here are cost and insufficient capacity when compared to HDDs or tape. In addition, the electrical charge decays over time in all such storage, and the likelihood of failure increases according to write cycles.
Ignoring memory cards as an option for long-term storage, what’s the best option to ensure data integrity in the medium to long-term? Well, you could outsource data storage requirements and make a qualified third party responsible for all the headaches associated with data storage, backup, verification, and archiving, i.e. use the cloud. Let someone else worry about data degradation, bit rot, and other terms associated with the process.
Whatever type of storage you decide on, it’s necessary to define a backup process. Will you focus on regular drive images, scheduled incremental backups, real-time mirroring, RAID, NAS or just an external drive? The choice is yours.
In conclusion, all our data storage devices and media have known flaws. Environmental conditions must be considered, and correct storage procedures are necessary to ensure data is retained without degradation. A good practice is to use the 3-2-1 method, where there is one original data source and three copies (involving at least two storage types, the cloud and tape, for example).
Data recovery from corrupt or damaged storage is expensive and is generally unnecessary if a stable backup process is in place.
Regardless of the backup/storage methods used, data verification is key. The only way to verify your backups is to restore them. A failure to restore a backup in an emergency could have serious implications for your business. There are too many variables in play to ignore what should be a standard preventative maintenance task, performed as an integral part of the disaster recovery process. The truth is we are limited by current storage technologies, at least until DNA makes them obsolete. I hope its sooner rather than later.