Photo-editing-topics

From Ed's Mediawiki

Update: A lot of this is no longer valid, because I have made some updates, including different cameras, moving to an Adobe Creative Cloud photography subscription, adding a NAS box for short term photo sync, and writing code to automatically update archives. I will bring this up to date soon.

How I organize my photos

All of my photos are on one drive (V:) and they are backed up in multiple ways (discussed later.) The directory tree is organized like this:

  • Camera-buf top directory
    • cannon-elph jpeg files from Andrea's old point and shoot camera
      • Exact copies of flash memory
        • 2010 06 21 ...example directory: pictures copied from the camera on June 21, 2010
    • fuji-m7000 jpeg files from my old fuji camera
      • same structure as cannon-elph directory
    • nikon-d200 jpeg and raw files from my Nikon D200. I normally shoot both NEF raw files and jpg files
      • same structure as other camera directories above
    • Edit_work top level directory for editing
      • 2008c-antigua example edit area directory. One for each trip, event, season, etc, named to display in chronological order (that is the extra letter is after the year.) To start, I copy some pictures from the camera directories to this working directory. As I go through pictures, they get deleted from here and edited versions go to done, below, or rejects. For my Nikon D200, I start only with NEF raw files and save both adobe digital negative files (DNF) and JPG
        • done complete edits get saved here
        • rejects pictures I don't use go here and eventually get deleted. The original camera versions never get deleted
        • other_pictures (optional) any event pictures from a source other than one of our cameras
        • print_groups (optional) copies or shortcuts to record pictures that got printed
        • sitesource (optional) working directory for generating an event web site using my old PERL tool. Not using this any more
        • batchsource (optional) used as temporary area for doing batch editing. Example is setting white balance and saturation enhancement on a bunch of outdoor shots
        • video (optional) used for video files from the same event, including originals and edited versions
      • 2009a-winter-spring another example event
        • ...
    • nx2-settings-files (Obsolete) place to store settings for the Capture NX2 editing software. No longer used
    • Other Picures pictures from various sources, including web sites, friends and family, etc
      • Russia trip ship pics example. Pictures taken by crew members on our Volga cruise
    • rename work working directory for renaming camera picture files. Pictures get renamed with a timestamp in the form yyyymmdd-hhmmss_originalname. Example: img_2094.jpg got renamed 20071122-201712 because it was shot November 22, 2007 at 8:17:12 PM. This renaming happens as part of the flow from the camera to the camera snapshot directories and virtually eliminates duplicate file names across all of our pictures
    • jalbum top of the directory tree used by JAlbum, which I use for generating event web sites. This replaces my own Makesite.pl tool I used to use.
    • My Albums the real top of the JAlbum directory tree, generated by the program. Everything from here down is managed by JAlbum.
    • advantix-cds cd images from Andrea's old film camera
    • New Epson Scans scanned images from our Epson scanner
    • New HP Scans scanned images from our HP scanner. Not much here because that scanner was exchanged for the Epson."
    • Poloroid scans scans from our very old photo albums."
    • Program Sources sources for programs I have written for photography work"

Methods and Tools

Camera to computer

When I copy files to the computer, they end up, after renaming, in a directory corresponding to the camera and a subdirectory named after the date of the copy. If more than one is on the same day, I add a, b, c, etc to distinguish them. This never happens anymore because of the size of the camera memory cards.

Renaming with timestamp

When I copy files from a camera, go first into the rename work directory, where they get renamed with a prefix time stamp in the form yyyymmdd-hhmmss_originalname and set in the time zone where the picture was taken. Strictly speaking, it should probably be in UTC, but it is easier to keep track of things in the local time zone with any daylight time in force at the time. After renaming, they are moved to the appropriate camera snapshot file.

Rename tools

The main tool I use for renaming is a command line program called exiftool, available from this site. This tool works on all files of the target type in a given directory, so I use a work area called __rename work__ for this. A key point is that this tool works not only on jpeg files, but also on Nikon NEF raw files. Camera files to be processed are copied here and the appropriate batch file is run on them, then they are moved to the appropriate camera file area. exiftool has a bunch of features, but the ones I use are:

  • To add N hours to the files' EXIF time stamps
    • exiftool -overwrite_original_in_place -AllDates+=N .
  • To subtract N hours from the files' EXIF time stamps
    • exiftool -overwrite_original_in_place -AllDates-=N .
  • To rename the files based on the EXIF time stamps in the specified format and make the file's create date match the timestamp
    • exiftool "-filemodifydate<createdate" "-filename<${CreateDate}_$filename" -d %%Y%%m%%d-%%H%%M%%S .

Because I post these files to my LINUX-based web site, I want all of the files to have lower case names, so I use a simple windows shell command to rename them:

  • For /F "tokens=*" %%F In ('Dir /B /L') Do Move "%%F" "%%F"

My rename work directory is populated with batch files that use combinations of the above commands to correct timestamps and rename files. I could get more fancy and use command line arguments, but instead, I have a batch file for each operation, which allows me to just move the files from the camera to the work area, double click on the appropriate script, and then move the renamed files to the appropriate camera snapshot file.

One script just lower cases the names. I often copy this to an event directory and run it there.

Basic Editing Flow

As shown above, each event, trip, etc, has its own directory tree, and generally includes a done subdirectory and a reject subdirectory. To start editing for an event, I add a new directory named appropriately. At this writing, my working directory is 2013k-uk-trip. For each travel day, I copy the renamed NEF files from the camera directories to the working directory, and edit with photoshop in chronological order (see details below.) Each time I complete a file, I save it to ...\done directory both in DNG (digital negative) form and in JPG form. Then I delete the original from the working directory. The camera original is never deleted. For some pictures, there may be multiple crops, and the names are appended with -crop1, -crop2, etc. Any picture I decide I do not want to edit or publish gets moved to ...\rejects just to keep track of them. Later, I empty out the rejects directory.

Editing

I use Photoshop CS5.1 for most of my editing. For raw files(Nikon NEF) this happens in two steps, because CS5.1 opens the files in Camera Raw, a front end editor for raw files. Camera Raw's native file format is Adobe Digital Negative (DNG,) which is a generic replacement for brand specific raw files like the NEF files that come out of my Nikon camera. Photoshop cannot edit raw files, so instead, it spawns Camera Raw.

Raw file editing

My raw file flow looks like this:

  • Drop the NEF file onto Photoshop or double click the file
    • Photoshop opens Camera Raw and displays the image. If there are more than one images, it opens all of them and provides thumbnails for selection.
  • Do the front end raw file edits. Camera Raw has a wide variety of image functions that duplicate some of the functionality of Photoshop, but these adjustments are being made on the raw camera data, which is sometimes beneficial because of its highher dynamic range. Some of the more important ones I use are...
    • White balance adjustment, which can be done by picking a preset like Daylight, Shade, Flash, etc, or by using the white balance eyedropper and clicking on something that is supposed to be a shade of gray.
    • Adjust the exposure.
    • Recover over exposed bright areas.
    • Adjust clarity, a special purpose contrast adjustment to sharpen details.
    • Correct lens distortion. It knows which lens was used and what focal length, so there is a lot of automation there.
  • When the front end edits are done, save the image to DNG format. At this point the NEF file is abandoned. A "sidecar" file (*.xmp) is also saved that remembers all of the adjustments made by at this stage, so if the DNG file is opened again, the adjustments can be tweaked from that point. If more than one file is open in Camera Raw, a XMP file is saved for each of them. In my flow, the DNG file goes into the "done" subdirectory and the XMP files go into the top level event work area. They will be manually moved to "done" later.
  • After saving the adjusted image, open it in photoshop. The is a button for doing this in Camera Raw, which will now close, revealing the image in photoshop ready to edit.

Photoshop editing

If the camera file was a JPG file, the above raw stuff is skipped. At this point, any raw pre-edits are done and photoshop editing starts. For many pictures, the raw edit was adequate and I just save the file to JPG, but there are usually a few things I want to do here. Most common:

  • Adjust response curves. Camera raw is limited in this functionality, and often I want to adjust some parts of the image differently using selection functionality.
  • Healing, clone stamping, removing wires and other objects, brightening faces, etc. Stuff that photoshop is famously good at.
  • Selective sharpening or blurring.
  • Burn, dodge, desaturate, etc, in specific areas.
  • Advanced stuff that I am still learning, and will be forever.
  • Cropping, straightening, stretching, etc.
  • Merging pictures to make panoramic images.
  • Merging pictures to make high dynamic range images.

Once edits are done, I save the image as JPG into the "done" subdirectory. For images that came from raw files, I manually move the XMP file to the "done" subdirectly, and delete the NEF file. Note that the original NEF file is still in the camera snapshot area, which will never get deleted.

Backup

I'm pretty paranoid about backup of my photos and the work I have done on them, so I have multiple backup methods.

Off Site Backup

I use CrashPlan backup for my entire computer, which includes 600+ GB of photo files, the size of the Camera-buf directory tree. Mozy worked well, but the prices went out of sight at the beginning of 2011.

On Site Backup

CrashPlan provides a way to make a copy of their backup on a local disk. I have a dedicated external drive (X) for this.

Photos Only Short Term Backup

In addition to the overall system backup strategy, I keep a different copy (M:\Camera-buf) of Camera-buf that gets populated by running a batch script every 15 minutes using the Windows task scheduling facility. I also occasionally force this script to run during long editing sessions. The script uses robocopy, an advanced Windows command available at no cost from microsoft as part of the Resource Kit . I did not have to do this download for Windows 7 Home Premium, but I did for Windows XP Pro. I have two versions of the script, one that just adds new files (automatic scheduled script uses this one) and leave old ones alone, and one that makes an exact copy (run manually periodically to clean things up):

  • Keep old files
    • robocopy V:\Camera-buf M:\Camera-buf /E
  • Exact copy
    • robocopy V:\Camera-buf M:\Camera-buf /E /PURGE

The key thing that makes robocopy attractive is that it only copies files it needs to, so it can go through the 600GB pretty fast when only a few files have been added or changed. An interesting side note is that robocopy can deal with longer path names than most other windows tools, so if you have a very long path because of long file or folder names, it can copy them or delete them. See this maintenance topic for an interesting related story.

Task manager details

I just restored this after a hard drive crash, and I remember it was complicated last time, and I might not have gotten it right this time. Here is how I set it up this time:

  • run the windows task scheduler
  • create a new task. I called mine Photo backup
  • set a trigger. I set it to "On Idle" so it will run when I stop working for a while. I might want to make this happen on a schedule instead.
  • set the action as "Start a program." I use a batch file that contains the robocopy command above.
  • set conditions. I set it to wait until the computer is idle for 5 minutes and to wait for idle for 5 minutes. This idle state stuff is confusing, so I looked it up. and found this useful information:
In Windows 7, the Task Scheduler verifies that the computer is in an idle state every 
15 minutes. Task Scheduler checks for an idle state using two   criteria: user absence, 
and a lack of resource consumption. The user is considered absent if there is no keyboard 
or mouse input during this period of time. The computer is considered idle if all the 
processors and all the disks were idle for more than 90% of the last detection interval. 
(An exception would be for any presentation type application that sets the 
ES_DISPLAY_REQUIRED flag. This flag forces Task Schedule to not consider the system as 
being idle, regardless of user activity or resource consumption.)

I seem to remember that a previous time I set this up, there was some process that prevented the idle state from ever happening, but I thought this got fixed when I switched to windows 7,