Photo Organisation

My approach to organising and storing my photographs.

Sunday 3 April 2022

  34 minute read  

Categories:

• Photo

Introduction

This article details my approach to organising and storing my digital photographic collection on my computers.

My photographic collection essentially contains two types of media:

1. Image and video files taken on a digital camera.
2. Scans from photographic film.

My collection is primarily centred around still images, and my organisational approach is currently only focussed on this type of media. Video information may be stored with the still images, or I may have moved that to a different structure depending on the use case. I will have to revisit the handling of video media in the future; it places great demands on storage space.

I categorise my handling of my photographic media into three types:

1. Masters: This is my principle version of an image, the ones that I work from and use.
2. Backups: Backups are used to recover images that I may lose through computer, brain, or other types of failure.
3. Archives: These are copies of images that I do not consider I need to work on, and thus do not necessarily keep in the Masters collection, but nevertheless I like to retain in case I find I do need these images in the future.

Most of my images are shot on a DSLR as a RAW file and accessed via a RAW file editor such as Adobe Lightroom or ON1 Photo RAW. Thus I tend to browse and access the photographs via one of these two applications, and my organisational structure is based around this.

Masters

I keep the master version of my photographs and videos on a SSD in my main workstation. Currently they are placed in a dedicated a whole 1TB disk partition. All of my photographic master files are thus stored under a common root directory (e.g. /PICTURES).

Under this root folder for my Masters area I then create a subdirectory for each year. Under each year subdirectory I create further subdirectories for each photographic project. A photographic project is just a group of photographs that have some relation to each other. It might, for example be:

• Photographs from a trip.
• Photographs relating to an activity such as pictures of goods to be advertised on eBay.
• Photographs that accompany one of these web articles (the pictures in this article were assigned their own project).

Each project folder is named with:

• An identifier code in the format “YYPPP” where PPP is a number that increases with each project folder I create.
• A short name identifying the project.

For example:

• 22002 Scotland Winter

For every year the first project folder is a general purpose folder that holds any odd picture that doesn’t deserve its own specific project folder. This is named:

• YY001 Miscellaneous

Thus my folder structure might take the form:

  PICTURES/
    2020/
    2021/
    2022/
      22001 Miscellaneous/
        22001001.DNG
        22001002.DNG
      22002 Scotland Winter/
        22002001.DNG
        22002002.DNG
        22002003.DNG
        22002004.DNG
        22002005.DNG
      22003 Photo Article/
        22003001.DNG
        22003002.DNG
        22003003.DNG

Images placed inside a project folder are renamed with a number based on the project identifier code followed by a 3-digit index code (###), giving a format (as seen in the above example):

• YYPPP###

In Lightroom I have created a catalogue that has this PICTURES root directory as the catalogue’s root directory. In ON1 Photo RAW I also get it to catalogue this PICTURES root directory. I find it easier to keep all my photographs in one place in my computer.

When I import a photographic image file into my computer I tend to convert it to DNG. I do this because:

• I like the idea of having an open documented format for my files rather than a proprietary undocumented format that software developers have had to reverse engineer.
• For B&W film scans, I found ON1 Photo RAW cannot handle monochrome TIFF files.
• I like to keep my colour film scans in the same format as my B&W scans.
• DNG compresses film scans to a file size noticeably smaller than TIFF can achieve.

For most pictures I take with a camera this is academic as my Pentax DSLR uses DNG as its RAW format anyway.

For images in other formats, such as PNG, I tend to leave in its original format. Same for any videos I store here.

For images that were scanned from film, the film archives are numbered and catalogued according to numbers assigned to the image in the Masters area. For example, this number is written on slides and also on any index contact sheets.

Backups

For me, I consider a backup to be a duplicate copy of a file that mitigates:

• A failure of hardware.
• Loss or corruption of a file due to making a mistake.
• Loss or corruption of a file due to malware.

Unlike archives, which I consider permanent storage of a version of a file, I consider backups to have a more temporary nature. The more often the backup is made, the less work is likely to be lost if one of the above issues occurs. However, the more often a backup is made the more storage space is consumed. This limits how many/how long backups can be retained.

The one thing I have found is that the more manual effort that is involved in making a backup, the less likely it is to happen. I consider some form of automated backup essential.

I now maintain two separate types of backup:

• Automated online backups. These backups run automatically every night the computer is on (or at the start of the next day the computer was off). This is my primary backup.
• Manual monthly backups. These backups are created manually, roughly every month. These backups are stored on an offline external hard disk in my firesafe.

Backup Software

I use Macrium Reflect Home 8 to make my backups. I bought the version with 4 client licenses and run a client on each PC I need to keep backed up.

The main features of Macrium Reflect that appeal to me are:

• Seems dependable and focussed on the task of handling backups.
• Supports mounting a backup image as a hard drive.
• Allows creation of a bootable disk with the recovery software and the backup images together.
• Macrium Image Guardian may provide some protection against ransomware.

A key feature for me is the creation of a bootable disk. I was amazed this was not a standard feature in all backup software. Imagine the situation. You need to recover from a serious computer failure. You find your disk with a backup image, but then need to find a separate USB boot disk with the recovery software. You eventually find that disk only to discover you have forgotten to update it for some time and it no longer supports the latest backup image format or new hardware! It seems to me the best solution is to have a single disk that can boot into the recovery environment and also hosts the backup image files. Macrium Reflect makes it easy to achieve this.

Another feature I have been grateful for is the ability to mount a backup image as a disk. A while ago I had a major failure of my workstation. Deader than a dead parrot. It would take me a few days to debug the issue and get the hardware back up and running. In the mean time I had stuff to do, and the files I needed were stored on the workstation. What I was able to do was fire up the laptop, mount as a drive a recent workstation backup image file held on my NAS, then browse this drive for the files I needed and copy them onto my laptop hard drive. When I got the workstation working again I restored it from a recent backup image on the NAS.

The Macrium Image Guardian feature is interesting. It claims to provide protection from ransomware by preventing the OS from modifying/deleting a backup image. This can only be done from within the Macrium Reflect software itself. Whether it would really prevent ransomware from deleting the backups, I don’t know, but every layer of defence is useful.

Automated Online Backups

My primary backups are my automated online backups which are stored on my NAS.

On each of my PCs I have set Macrium Reflect to run an automated backup job, backing up my computer disk partitions to the NAS. This automated backup is set to:

• Create a full backup of the PC at the start of each month.
• Create a differential backup of the PC every week.
• Create a daily incremental backup of the PC each night.

Essentially it means that for at least a week I can recover a version of a file that existed on a specific day. Then for the current month I can get a version that is at most a week old.

I have configured my NAS to have a dedicated share for backups. I have allocated 8TB to this share and this means I can store about 6 months of the monthly backup images. My big concern with online backups is that they could be vulnerable to a ransomware, potentially corrupting my backups as well as the original files. To mitigate this I try and restrict access to the backup images on the NAS:

User Account	Access to Backup Share
Public access	No access
My personal user account	Read only access
Backup software account	Read/write access

The idea is that I can recover data from the backup, but can’t easily delete it. Only the backup software has read/write access to these files. Thus if I suffer a ransomware attack it can’t see the backup images without specific user privileges. It is likely to only run under my own user privileges and thus not have write access to the backup share. Well that’s the theory anyway. Security is all about layers.

The biggest problem with this idea: Microsoft. Microsoft Windows does not allow you to access a server with more than one set of user privileges. So I found that if the backup software had recently run, when I browsed the backup share, I would have the same access the backup software had and thus able to delete backup images. Useless! At heart, Windows is not a multi-user Operating System. The workaround is to use the fact that a server can be accessed by its domain name or its IP address. Although these are identifiers for the same physical server Windows sees them as separate servers. So I set the Macrium Reflect software to access the backup share using the NAS IP address. I would normally browse the NAS’ shares using its domain name and thus received limited backup share access rights of my personal user account. The Macrium Reflect software is able to access the share with its backup user account with full write access. You need to ensure the NAS is assigned a static IP on your network for this to work.

The one area of these backups I keep reconsidering is whether to encrypt these backups. Currently I do, so that if my network is compromised the data in the backup cannot be read. However, should I forget the password I have lost my backups. I probably should not encrypt my backups as most of the data is in plain text on my PCs, and anything sensitive is stored in an encrypted container in my computers.

Manual Monthly Backups

To protect against ransomware corrupting both my computers and any backups it finds on my network, I maintain a second set of offline backups. These are held on USB hard disks stored in my firesafe.

Approximately once a month:

• Take appropriate external hard disk out of the firesafe and connect it to my workstation.
• Use the Macrium Reflect tools to burn a new recovery image to the disk.
• Run a manual backup of my workstation to the disk (after deleting any previous backup image).
• Connect the disk to my other computers in turn and run a manual backup of that computer to the disk (after deleting any previous backup image).
• Once all computers are backed up I return this disk to the firesafe.

I use a set of 2.5" USB3 external hard drives. In order to provide a number of versions of my files stretching back in time, I use a set of 7 hard disks that can give me 6 months of separate backups. Each hard disk is used twice a year. The disk set is:

• January / July
• February / August
• March / September
• April / October
• May / November
• June
• December

I actually have 2 December disks. One I keep at home and one at a relative’s house. Just before Christmas I do a full backup on this December drive. I then take it to my relative’s over Christmas and swap it with the other drive that I store there. This way at least I have one form of offsite backup, even if it becomes 1 year out of date.

I also encrypt these backups also, but again this runs the risk of forgetting a password results in lost backups. I suspect I should move to not encrypting the backups stored in the firesafe, and only encrypt offsite backups.

The main challenge of this scheme is that I forget to make these manual backups as they take a little effort. I can go several months until I remember.

Archives

I consider an archive to be a file that i wish to retain indefinitely, but don’t need ready access to it, and in fact don’t anticipate having to access it at all. Thus it is really for:

• Files I don’t need but wish to keep “Just in case.”
• A copy of a file I have an active master for, but wish a 2nd version in a format that can be reliably accessed in years time.

Currently I host the following types of archive:

• Original RAW image files created by my cameras.
• Original RAW scans of film media.
• A copy of the master image of my photographs.
• A snapshot final image of my photographs.

Archive Storage

I store my archives on multiple types of storage media. The types of storage media are described below:

• NAS

  I have a NAS in my house for hosting shared files. On this NAS I have created a share “Archive” which is used to host online archives.

• Offline External Drive

  I use external USB hard drives to host offline archives; one for photographs and one for videos. These hard drives are stored in my data fire safe when not in use.

• Optical Media

  An archive is written to a DVD or a Blu-ray disk. Most would consider this to be an obsolete media but I like them as they are read-only; there is less danger of files being deleted/corrupted by malware, and similarly no danger of my deleting the files through a blunder (which I have done in the past with hard disks). They are also cheap so I can occasionally create a new copy and store the old copies in a shoebox in my attic.
  
  Generally I use Taiyo Unden 8x DVD+R disks and Panasonic BD-R and BD-R DL. I am not sure I can buy Taiyo Unden DVD disks going forward, so will need to reconsider my options there. The Blu-rays I have to buy from shops in Japan using eBay. I believe Panasonic Blu-rays are still considered one of the most reliable disks you can get.

RAW Archives

I archive the original RAW images taken by cameras or produced by my film scanner.

I use the import feature of Lightroom or ON1 Photo RAW to copy the RAW images from the SD-card/scan directory into my photo Masters directory where the master version will be catalogued. I have set these programs up so that they make a second copy of the original file to a special Import directory on my workstation’s temporary drive.

Periodically, once I have imported a sizeable volume of images, I will then copy these RAW image archives to their permanent archive areas.

• NAS:

  My NAS has a share called “Archive”. In this share is a subdirectory called “RAW”. This directory is used to store the RAW backup files.

  

  In this RAW archive area I have created a directory for each year. In these directories I then create a subdirectory for each month in the year in the format “YYMM”

  For camera RAW files, I copy the RAW file from the temporary backup area into the sub-directory for the month that the RAW image was originally taken in.

  //NAS/Archive/
    RAW/
      2020/
      2021/
      2022/
        2201/
        2202/
          IMGP7268.DNG
          IMGP7269.DNG
          IMGP7270.DNG
          IMGP7271.DNG
        2203/
  

  For file scans I use a slightly different approach. In this case I use the date the film image was originally taken in, not the date the scan file was created, to signal the subdirectory that RAW file will be stored in. Thus if the film image was originally taken in November 1995, the scan file will be stored in directory “RAW/1995/9511”.

• External Hard Disk:

  I have a USB hard disk that I use to store the RAW images in. This disk normally resides in the firesafe.

  The organisation of the files on this hard disk follows the same arrangement as for the NAS, except the disk top level directory is equivalent to the //NAS/archive/RAW directory:

  DISK/
    2020/
    2021/
    2022/
      2201/
      2202/
        IMGP7268.DNG
        IMGP7269.DNG
        IMGP7270.DNG
        IMGP7271.DNG
      2203/
  

  I normally write these files immediately after writing the files to NAS. Once done I can then delete the RAW files from the temporary RAW area.

• Optical Media

  My final step is to write the RAW images to a DVD or Blu-ray drive.

  My first step is to write the files to DVD+R drives for that year. The first DVD I use in year 2022 is labelled “RAW202201”, the second “RAW202202”, etc. The files are written in their monthly sub-directory, using the disk root directory to represent the year.

  RAW202203/
    2202/
      IMGP7268.DNG
      IMGP7269.DNG
      IMGP7270.DNG
      IMGP7271.DNG
    2203
      IMGP7272.DNG
      IMGP7273.DNG
  

  Files are written in order of ascending date until the disk is full, or all files have been written. I always select the order by date options in my burning software to make sure my files are definitely ordered by ascending date before I select the ones to add to the disk. I do not finalise these disks so that I can add more files to a disk at a later date.

  Once a DVD is full I’ll add a new DVD to the set. Once about 6 DVDs are full I’ll create a new Blu-ray and reburn the RAW files to it. The Blu-rays also are labelled in sequence “RAW202201”, “RAW202202”, etc.. Essentially the Blu-rays form their own independent set of archives with file ordering carrying over from one disk to another. When all files in an original DVDs have now been re-burned to a Blu-Ray, the original DVDs are thrown into a shoebox in my attic.

  Once a year, generally in the New Year, I reburn these RAW archives into a fresh set of disks. I generally use 50GB BD-R DL disks for this to reduce the number of disks I store. Again the disks are labelled in the previous sequence with the files in order crossing each disk boundary. On these archives I perform burn quality scan using the application Opti Drive Control. I reburn any disk where this scan does not report an excellent burn. Any original DVD or Blu-ray disks are tossed into that shoebox in the attic. You never know when they might come in handy.

Masters Archives

The master version of my images resides on the workstation in the Masters area and accessed using my RAW software applications. However occasionally I like to make an archive of the edited master images. Typically I will do this once I have completed all of my edits for a project, or all of the edits for all projects in a year. Essentially I am trying to create a snapshot of the edited images, but retaining the full quality of the master file.

I keep one copy of these master archives on my NAS. My NAS has a share called Archive. In this share is a subdirectory called Masters. This directory is used to store the Masters archive files.

Under this subdirectory I essentially mimic the file structure of the Masters area on my workstation disk. Each year is a subdirectory. Under this each project has its own subdirectory. The name of this subdirectory is the same as my workstation Masters area. In reality, I tend to start by just copying the whole project area from the workstation Masters area to the correct year subdirectory on the NAS Masters archive directory. This will copy the original RAW files, but may also include sidecar files containing the edits that have been made.

//NAS/archive/
  Masters/
    1993/
    1995/
      95001 Miscellaneous
        95001001.dng
        95001001.on1
        95001002.dng
        95001002.on1
      95007 Mera Peak
        95007001.dng
        95007001.on1
        95007002.dng
        95007002.on1
        95007003.dng
        95007003.on1
        95007004.dng
        95001004.on1
    1997/

If the project was edited in Lightroom, then I will export as a catalogue the year directory for the projects being archived.

//NAS/archive/
  Masters/
    1993/
    1995/
      1995.lrcat
      95001 Miscellaneous
        95001001.dng
        95001001.on1
        95001002.dng
        95001002.on1
      95007 Mera Peak
        95007001.dng
        95007001.on1
        95007002.dng
        95007002.on1
        95007003.dng
        95007003.on1
        95007004.dng
        95001004.on1
    1997/

If the project was edited in ON1 Photo RAW then the edits are contained in sidecar files alongside the RAW image file. In this case I create a zip file in the year directory that contains these sidecar files, hosted inside the project directories inside the zip file. I would typically call this file year.on1.zip. Once this file has been created I can delete the zip files from the project directories to tidy things up. So in the above example the directory structure is:

//NAS/archive/
  Masters/
    1993/
    1995/
      1995.lrcat
      1995.on1.zip
      95001 Miscellaneous
        95001001.dng
        95001002.dng
      95007 Mera Peak
        95007001.dng
        95007002.dng
        95007003.dng
        95007004.dng
    1997/

whilst the zip file “1995.on1.zip” has the following content:

95001 Miscellaneous
  95001001.on1
  95001002.on1
95007 Mera Peak
  95007001.on1
  95007002.on1
  95007003.on1
  95001004.on1

In addition to the online NAS archive, I also archive these files to an external USB disk that is kept in my firesafe. The organisation of the files on this hard disk follows the same arrangement as for the NAS, except the disk top level directory is equivalent to the //NAS/Archive/ directory:

DISK/
  Masters/
    1993/
    1995/
      1995.lrcat
      1995.on1.zip
      95001 Miscellaneous
        95001001.dng
        95001002.dng
      95007 Mera Peak
        95007001.dng
        95007002.dng
        95007003.dng
        95007004.dng
    1997/

Generally the content of the external disk archive is created by just copying the contents of the NAS version of the archive.

Finally I will burn these Masters archives to optical media, either DVD+R or BD-R. The way I do this varies:

• After a project has been completed, I may burn a disk of Masters archives for that project.

• After all of the projects in a specific year have had their final edits, I might burn a set of disks for the Masters archives of all projects in that year.

  

If creating a project based archive disk, I use the content of the NAS archive for the projects but with some adjustments:

• I only include the project subdirectories I want to be included in the disk.
• If a project was edited in Lightroom, I would export a catalogue of just the project subdirectory itself. So each project on the disk has its own Lightroom catalogue.
• If a project was edited in ON1 Photo RAW, I adjust the ON1 sidecar ZIP files to only have the sidecar files for a single project. Each project would have its own zip file.

Thus in this example:

MASTER1995/
  95001.lrcat
  95001.on1.zip
  95001 Miscellaneous
    95001001.dng
    95001002.dng
  95007.lrcat
  95007.on1.zip       
  95007 Mera Peak
    95007001.dng
    95007002.dng
    95007003.dng
    95007004.dng

One example of a project based Masters archive was when I scanned the slides from a major trip I did to the Himalayas. In the folder that held the slides I also store a Blu-ray with Masters archives for this project.

JPEG Archives

Although my Masters archive provides the best quality for my photographs, it suffers from the disadvantage of requiring particular software to access the edited image. To mitigate this restriction I also create JPEG archives of the edited image based on an exported snapshot of the edited image. Being JPEG this should be readable by many types of software for years to come.

One of my fears about digital photography is that over time the images will “evaporate” as media and formats become obsolete, and software companies have gone to that big corporate business park in the sky. Similarly, once I’m long gone, will my pictures be gone with me? What is the equivalent of the old family album. To avoid this I feel an easily accessible folder full of JPEGs is the modern equivalent of that family album.

Thus from time to time I will export a snapshot of my edited images as a JPEG file, to be stored in archive areas. Typically I perform this export after I have finished work on a project, or all of the projects for a given year. This export of the edited images uses the following format:

• Format: JPEG.
• Image size: Original.
• Quality: Maximum
• Colour space: AdobeRGB.

Compared to the original RAW file there will be some loss in image quality; it’s 8-bit not 16-bit, the colour gamut is a bit more limited, some fine detail might be lost. But I still feel these settings offer best of the quality of a final image, but with the advantage of being universally readable.

I keep one copy of these archive files on my NAS. My NAS has a share called “archive”. In this share is a subdirectory called “JPEG”. This directory is used to store the JPEG archive files.

In this JPEG directory I have a subdirectory for each year. Whenever I export the files in a project from my Masters area, I store the JPEGs in a subdirectory under this year using the same directory name as the project uses in the Masters area. Thus the directory format of this JPEG archive area mimics that of the Masters directories, except the images are “.jpg” instead of “.dng”.

//NAS/archive/
  JPEG/
    1993/
    1995/
      95007 Mera Peak
        95007001.jpg
        95007002.jpg
        95007003.jpg
        95007004.jpg
    1997/

In addition to the online NAS archive, I also archive these files to an external USB disk that is kept in my firesafe. The organisation of the files on this hard disk follows the same arrangement as for the NAS, except the disk top level directory is equivalent to the //NAS/archive/ directory:

DISK/
  JPEG/
    1993/
    1995/
      95007 Mera Peak
        95007001.jpg
        95007002.jpg
        95007003.jpg
        95007004.jpg
    1997/

Finally I will burn these JPEG archives to optical media, either DVD+R or BD-R. The way I do this varies:

• After a project has been completed, I may burn a disk of JPEG archives for that project.

• After all of the projects in a specific year have had their final edits, I might burn a set of disks for the JPEG archives of all projects in that year.

  

One example of the former scenario was when I scanned the slides from a major trip I did to the Himalayas. In the folder that held the slides I also store a Blu-ray with JPEG archives for this project.

Future Considerations

The above sections discuss my current photographic media organisation, but I realise it has many weaknesses and shortcomings that I will have to address at some time. This section highlights some of those that I am aware of.

Videos

My organisational structure is focussed around still images. I still have not settled on approach to handling video files. Video files require special consideration:

• The file size of video is extremely large and consumes a lot of disk space, both on my workstation and in the backups.
• Video will be edited by a different set of applications to those I use for still media.

Currently, when I archive the RAW images from my camera I also include in this archive any video files I created in this camera. I am likely to continue this approach as it seems logical to me.

I have not been doing much video editing recently. So what has been happening is that the video files get imported into my photographs masters collection in the same project folders as any still images. I am not sure this is a good approach.

Previously, when I was doing specific video projects, I used to move the video masters to a video specific directory on another disk. Like the photos, each video project had its own folder with a date based serial code. In this folder I would have:

• the master video files,
• any intermediate working files,
• the video editing program’s project file,
• any renders from the finished video.

Generally this video project would only be held on my workstation’s disks whilst I was working on it. Once I considered I had completed the project I would:

• Move the project to the NAS and an offline hard disk, in the similar way I archive still photos above.
• Make a DVD for the video project, to archive the project.
• Copy a final render to my media server.

This approach is a little disjointed, compared to my still photo approach, but is significantly reduces the amount of disk space that is being actively managed and regularly backed up.

Non-Photographic Images

The organisational approach above is focussed on the photographic media I have taken. What I have not worked out an approach on is other types of media. For example, when writing this web site I have been taking screen grabs of the processes I follow. How should these be organised?

• Add to my photographic masters catalogue?
• Store in a project specific area? For example, for the web site store all master images in my private copy of this web site?

I have started to follow the first approach of storing these images in my photographic masters area, but will have to see if I feel comfortable with this approach.

Off-Premises Backup

The biggest weakness with my current approach is that almost everything is located in my home. A major incident there, such as a fire, theft, asteroid impact, and my masters, backups and archives could all be destroyed. My only mitigation is that I store a hard disk with my PC backups at a relative’s home every year. So, in theory, I can only lose at most 1 year’s worth of important information. That’s something, but its not great.

In the good ol’ days I used to periodically take an offline backup hard disk into the work and store it in my desk. However, now I no longer visit the office, so this is not an option. I also used to find you would forget to do this and the office backup could easily be half a year or more out of date.

The ideal solution would probably be to store encrypted backups in the cloud. For example, I might consider taking out Microsoft 365 subscription primarily for the cheap storage they provide. Unfortunately, with my 800 kbit/s Internet uplink speed, storing a backup in the cloud is not likely to complete before the heat death of the Universe.

Thus, for the moment, I will probably have to rely on my yearly offsite backups and offline backups in my firesafe. In then end, there is nothing really critical as this is all personal material. With the off-site yearly backup hosting many key personal records, I ought to be able to recover anything critical, and its just photos etc that I’ll lose.

Manual Backup

Every month I create a manual backup of my computers, made on an external USB drive that is stored offline in my fire safe. Well, that’s the theory. In reality I forget and it can be several months before I remember to make the backup. I need to improve my strategy for creating these offline backups.

Some things I am mulling over:

• Get organised. Schedule some recurring monthly event in my calendar to make these manual backups, similar to the way I use the New Year holidays as an event to do my yearly archiving of my stuff. I just fear I am not that disciplined to be this organised every month.

• 2nd offline NAS. Arrange to have a NAS that spends most of its time powered off but is scheduled to power up automatically once a month. Once it has powered up, it would take a copy of the most recent full monthly backup image for each PC held on my online backup NAS location. The offline NAS would automatically delete any old backup images it has stored when more space is required. Once it has a copy of the latest full monthly backups, it would automatically power down again.

• Firesafe NAS. Currently I use external USB drives for my monthly backups, and these are stored in my firesafe. It is a bit of a pain trying to fish out the correct drive for that month’s backup. This process could be simplified by storing a small NAS in the firesafe instead of all these disks. Like the “2nd offline NAS” option this could be configured to automatically take a copy of the latest monthly PC images when it powers up. So the manual backup routine becomes:

  • pull NAS from fire safe,
  • power it up (it would have a WLAN interface already),
  • let it run for a while to complete its work,
  • power down,
  • return it to the firesafe.

Bit-Rot

An insidious issue with long term storage of digital files is that they can be susceptible to bit-rot, that is the contents of a file can become corrupted over time, even if not accessed. There can be bit flips on a hard disk or SSD, and optical media can become damaged.

Currently I have no mechanism to detect and recover from this. However, I do have 2 mitigations that help a little:

1. If I spot a corruption quickly after it occurs, I may be able to recover a version of the file from a backup taken before the corruption happened.
2. Since I tend to keep multiple archives on DVD/Blu-ray, many written not long after the photograph was taken, I have the potential to recover from a corrupted file by trying to obtain the original from one of these old archives.

However, at some time I need to put a more robust recovery strategy in place. I probably need to sort out some kind of checksumming approach that can detect changed file contents from some date a snapshot was taken, or when the file was written to an archive. This probably needs to be complemented with running this check regularly, detecting the issue quickly whilst it may be recoverable from backups, or, for the archives, perhaps on a yearly validation of archive integrity.

Format Obsolescence

A danger with digital media is that information is written in a format that become no longer readable as technology evolves. There are two main potential issues here:

• It becomes impossible to get a device to read the media the file is stored on.
• There are no programs you can still run that read the image file format.

For the first point:

• My masters are stored on a SATA SSD inside my workstation. Disk storage formats do become obselete. However I expect to refresh my workstation every 5 years at least, so I would expect my masters files to migrate to the current technology when this upgrade occurs.

• My backups and archives are stored on a NAS. Like the workstation, I will expect my NAS will require periodic upgrades, probably on a longer timescale than the workstation. So I expect the storage media will convert to the latest standards when this upgrade occurs.

• My backups and archives are also stored on an external portable hard drive. This uses a USB3 connection and I expect this standard will be supported for a very long time. The disks are formatted using NTFS and this volume format may be less durable. I will need to consider how my external hard disks are formatted should I change Operating System. I do expect over time I’ll be upgrading my external drives once the current ones become too small for the information they need to store, something that will probably occur within 5 years.

• I also have archives on optical media. This format is most likely to be unsupported in the future. It is already becoming increasingly harder to source optical drives. I am likely to have an issue here, though I am unaware of any replacement at this time that fulfils the role of optical media (tape is very expensive and, based on my previous experience with DAT, I am not convinced it is that reliable for long term storage). I do have 3 optical drives, so at least I may have bought myself some time to cover for individual drive failures.

As to the problem of reading image formats:

• My masters generally are stored in DNG format. There is no guarantee this will be readable long term, but neither is proprietary RAW. At least DNG is documented so that barrier to implementing a software reader is removed. Working in telecommunications, I am a big believer in open standards for interfaces being a good thing.

• My main concern is accessing my photographic edits. Since I use a non-destructive RAW workflow, the edits on the original RAW image are stored in proprietary sidecar files, only really useable by the application that created them. Thus if I cannot use this application, then I have lost all my edits. I have already encountered this with my move from Lightroom to ON1 Photo RAW. Some edits survived the transition, but fundamentally you are starting again. We are really a hostage to to the software developers with this workflow.

  One mitigation is to take up a subscription, but to me that feels like I am paying extortion money to keep access to my own work irrespective of whether the software application is offering me any additional value or not. Another option is to keep an old PC, or a Virtual Machine, to run the application should it become unsupported on your latest PC. However, in the era of online activation, you might not be able to install the application on a new PC, as I found out when Adobe’s activation servers refused to allow me to move my copy of Lightroom 6 to my laptop.

  The only mitigation I currently have is to export a snapshot of the edited file to a disk in a format that is generally accessible and has the edits baked in. Currently my JPEG archives fulfils this role, but it has dropped some of the original image quality. To avoid this I need to consider if I should also export to 16-bit TIFF using ProPhoto colour space.

• Backups are currently stored using the proprietary format of Macrium Reflect. I am less concerned about this image format since backups normally only have a limited lifespan. My online backups tend to be retained only for 6 months. Offline backups may be retained for over a year. For these, the backup disk also has a bootable version of Macrium Reflect, so I feel confident I can get at the stored files by booting from the external disk, at least for the timespan these backups will remain relevant. Generally, when I make the offline backups I also refresh the boot image on the external hard disk, to ensure it is current to the backup image format.

• My other archives are stored in a high-quality JPEG format. I feel confident the JPEG format will be readable for many years to come. So I hope this file format makes a good format of last resort, accessible to people other than myself.

Optical Media Issues

I store a copy of my archives on DVDs and Blu-rays. Most people would see these as an antiquated media format and they are. Unfortunately I have not encountered anything more modern that:

• Is read only. Once data is stored it cannot be erased/overwritten.
• Is cheap to create many copies, and small enough to store multiple copies long term.

It is already becoming hard to get optical disk drives. The number of manufacturers has shrunk and I think the quality of the drives is decreasing. It is now very hard to get a computer case that will accommodate an internal drive, meaning one increasingly has to go down the external USB drive route. Getting a drive that supports scanning the quality of a burn is nigh on impossible now.

Getting quality media is also becoming a problem. I used to use Taiyo Unden for DVDs, but they exited the market. Then there are other brands that may support a good burn, until you find out only some of their media is good, others is only so-so, and it can be very hard to determine what you are buying. Finally, there is a lot of dodgy, or counterfeit, media being sold. It’s a nightmare.

Another issue that is becoming an increasing problem is the size of storage. I have already had to move to Dual-Layer (DL) Blu-ray, just to limit the number of disks I have to store. For example, my RAW archives may now take up to 5 disks for a single year. I may have to move onto a larger Blu-ray format but it is very expensive and I worry the more layers on the disk the less reliable it may be. Also, not all Blu-ray players can read this higher capacity format.

The final concern is longevity. Blu-rays can be advertised to hold data for over 100 years. Yeah, right. I recently came across some disks I burned 10 years ago and had difficulty reading them. Some disks had “rusted”. For others, I had put a paper label on them which is sayonara for DVDs. Fortunately I found other copies of the files amongst my archives. On the other hand, I have had hard disks fail too and been able to read some ancient DVDs. So my approach is to not trust on any one solution, diversity of options is safest.