Category: General

Interviewed on TWiT Photo

I was honored to be the featured guest on TWiT Photo, which normally features only the very best professional photographers. The on-demand video (and audio-only version) are now live. While the usual hosts +Leo Laporte and +Catherine Hall were in Norway, I had a terrific time with stand-in co-hosts +Trey Ratcliff and +Sarah Lane. And thanks to +Tony Wang for putting the show together. Hope you enjoy.

[I was particularly impressed with the skills of the TWiT team. I've always appreciated Sarah's on-screen talents, but it's not until you sit down on a set with her that you realize how good she is at listening and keeping the show moving. And Tony not only runs the show, but has the uncanny ability to switch in web pages and images almost before they're mentioned, even when he doesn't know what's coming. True professionals.]

HDR Workflows

Update 1/19/12: Based on feedback from many including John Omvik at Unified Color, I’ve improved my #1 workflow and substantially edited all the workflow descriptions below.

This is my first-ever high-dynamic-range (HDR) image, shot nearly three years ago. Since then my HDR workflow has changed quite a bit — almost weekly, it seems. As I’ve recently been running tests on some aspects of HDR processes and tools and particularly methods for transferring images between applications, I thought this would be a good time to pin down and document the workflows I’ve been using and explain why I’m still using them or not.

My primary applications are Adobe’s Lightroom (LR) and Photoshop (PS). I use LR to organize my images and for basic processing. I turn to PS for images that require adjustments beyond LR’s capabilities. The HDR tools I’ve used at one time or another include Photomatix ProLR/EnfuseHDR Efex ProHDR Expose 2 and its cousin, 32 Float. Note that all of these applications are available as free-trial versions. I encourage you to download them and experiment with these workflows.

Because you may not have some of these applications, I’ll describe seven different multi-image workflows as well as some for single-image HDR. My goal is to cover not only the tools and methods I’m currently using, but also those that I tried, tested and in some cases abandoned before settling on my current choices.

Workflow #1: 32 Float (with Photoshop)

I’ve recently started using Unified Color’s 32 Float for most of my high-quality HDR images. 32 Float is a PS-plugin version of HDR Expose. Given that I usually end up in PS anyway, this gives me a simpler workflow than I’d get with HDR Expose. I’m only giving up batch operations and a few functions that are better performed in LR or PS anyway. This is my current workflow for my highest-quality HDR images.

  1. Unless you’re shooting action/sports, which rules out multi-exposure HDR anyway, always shoot in RAW. Otherwise none of this applies to you.
  2. In LR, make only two adjustments to your images: lens correction and camera calibration, which can only be done at this stage. Defer everything else until your RAW files have been merged into a single 32-bit image. (I have camera profiles for each of my body/lens combinations made using a ColorChecker Passport.) Use Copy/Sync to apply the same adjustments to all the bracketed originals. If you’re not an LR user, you can instead perform the equivalent of this and the following step using Adobe Bridge and Adobe Camera Raw (ACR).
  3. Select the bracketed RAW files in LR then (from the Photo menu or ctrl/right-click) Edit in…Merge to HDR Pro in Photoshop. Importantly, and unlike some other Edit in… and Export options, Merge to HDR Pro sends full-range 32-bit files to PS, even if you’ve made adjustments in LR. This launches PS’s own HDR Pro module. [PS's tonemapping is weak, which is why all these third-party tools exist. But the merge-to-HDR function seems to work as well as any other.] In Merge to HDR Pro don’t be alarmed if you don’t see your highlight or shadow details. You’re only looking at a low-dynamic-range (LDR) preview of the HDR image, which can’t be properly displayed on your monitor.
  4. In HDR Pro set the Mode to 32 Bit and click OK. This merges the originals to create a 32-bit HDR image, then opens it as the Background layer in PS.
  5. Launch 32 Float from PS by selecting Filter…Unified Color…32 Float.
  6. In 32 Float tonemap the HDR image to low-dynamic-range (LDR). Many operations are best done here, while you’re in 32-bit mode and your luminance data are separate from your color data. Go ahead with global changes such as sharpening, noise reduction, local contrast and color and tone adjustments, but I’d stay away from dodging and burning in particular. At least in the version I have, there’s no undo feature, so an erroneous burn can wipe out what you’ve done, just like in a wet darkroom!
  7. You’ve now done pretty much all you can do in 32-bit mode, so set Upon Apply Convert To to 16 bpc. This will return a 16-bit image back to PS and change PS to 16-bit RGB mode. It’s also better to change to 16-bit mode while you’re still in 32 Float since there can be some rather quirky artifacts when the 32-to-16 bit conversion is done within PS.
  8. Back in PS, decide whether you want to merge in any of the original exposures. I will often do this in cases where there are ghosts such as people in different positions or if there are artifacts, details, tones or colors that are much better in one of the original images than in the merged one. If so, go back to LR, select the original RAW images and click on PhotoEdit In…Open as Layers in Photoshop… Back in PS, select the merged image then use Layer…Duplicate Layer to copy it as the top layer in the RAW-image stack. Use layer masking and other tools to manually combine your  originals and the merged image.
  9. Further tweak the combined image as necessary. If you need a filter that’s only available in 8-bit RGB mode such as Pixel Bender, Distort or the Topaz Labs suite, change the Mode to 8-bit RGB, but do so as late as possible. Otherwise keep the image in 16-bit mode.
  10. Click Save, which returns the image to LR.
  11. In LR do your cropping, final sharpening, noise reduction and vignetting.

Workflow #2: HDR Expose (without Photoshop)

This variation is designed for those who either don’t have PS or just prefer to do all their retouching in LR.

  1. Don’t make any adjustments in LR’s Develop module. The one exception might be if you need to correct for severe chromatic aberration, which is quite difficult to do later in the workflow.
  2. Select the bracketed RAW files in LR then select File…Export…Merge and Edit in HDR Expose. This uses HDR Expose’s alignment and merge engines instead of PS’s. Important: Make sure you select Merge and Edit Original Image(s) in the Export dialog box. This will send your RAW files directly to HDR Expose. If you select Merge and Edit Images with Lightroom Adjustment(s) you will be sending LDR 16-bit TIFF files and thereby throwing away a lot of important data. The only time to use this option is if you corrected chromatic aberration in step #1. It’s a tradeoff. [There's a workaround for this tradeoff: You can make adjustments in LR's Develop module. (I'd suggest limiting them to lens correction and camera calibration.) Then Export your set of bracketed RAW images as DNG files. This retains their full 32-bit range. You can then start HDR Expose as a standalone application and merge the saved DNG files.]
  3. Tonemap and adjust the image in HDR Expose.
  4. Click OK and save the image as a 16-bit TIFF. It will be returned to LR.
  5. You have the option at this stage to perform additional processing on the merged LDR image in PS. Use Edit in…Photoshop CS5 to export to PS. When you’re done, Save will return the results to LR.
  6. Crop, sharpen, reduce noise and vignette in LR.
  7. Continue with step 8 in Workflow #1.
Workflow #3: LR/Enfuse

Before a friend told me about HDR Expose and 32 Float, this was my first choice for HDR merging and tonemapping. Not only is it free (donationware, actually) it’s also fast and simple and produces a fairly linear tonemapped image ready for additional adjustment. LR/Enfuse is the open-source Enfuse command-line utility packaged as a LR plugin.

  1. Select your bracketed RAW images in LR.
  2. In the LR menu, select File…Plug-in Extras…Blend exposures using LR/Enfuse…
  3. In the Output tab select 16-bit ProPhoto TIFF and Reimport image into Lightroom.
  4. LR/Enfuse will perform the HDR merge and tonemapping. Although you can make some adjustments before it runs, there’s no interactive preview.

The tonemapping in 32 Float, HDR Expose and LR/Enfuse generally give me the most realistic results. But occasionally I want a less-realistic look, in which case I typically turn to Photomatix Pro. There are three different ways to use this utility, depending on the balance you want to strike between simplicity/speed and quality/control.

Workflow #4: Photomatix Tone Mapping Plugin for Photoshop

This plugin is sold separately by HDRsoft. It’s the best way to use Photomatix Pro with PS.

  1. Follow steps 1-4 in Workflow #1.
  2. Launch the Photomatix plugin by selecting Filter…Photomatix…Tone Mapping…
  3. In Photomatix tonemap the HDR image to LDR and make other adjustments. I tend to use default settings here, waiting until I return to PS to make further changes.
  4. Click OK, which returns a 32-bit HDR image to PS.
  5. Don’t worry if the LDR preview of the 32-bit HDR image looks all wrong in PS. Click on Image…Mode…16-Bits/Channel.
  6. If you want the double-tonemapping look, you can make further tweaks in this second HDR Toning step.
  7. Continue with step 8 in Workflow #1.
Workflow #5: Photomatix Pro (with Photoshop but without the Plugin)

If you want to work with PS but don’t want to buy the Photomatix Tone Mapping Plugin for Photoshop, this variation works fine. It’s just a little more complex.

  1. Follow steps 1-4 in Workflow #1.
  2. Save the HDR image as a Radiance (.hdr) file in a temporary location. (You don’t need to keep this for archival purposes since it is easily reproduced from the RAW images at any time.)
  3. Start Photomatix Pro and open the .hdr file.
  4. Use tonemapping and other adjustments to create an LDR image from the HDR image.
  5. Save the LDR file as a 16-bit TIFF and Import (Copy) it into LR. Use a filename that will place it near your originals in the LR grid view.
  6. If you need to make PS adjustments, use Edit In…Photoshop CS5, and return the results to LR as another LDR 16-bit TIFF.
  7. Continue with step 8 in Workflow #1.

Workflow #6: Photomatix Pro (without Photoshop)

This workflow bypasses PS altogether, moving images from LR directly to/from Photomatix Pro. This process is fine for some HDR images, but it has certain weaknesses. First, it depends on Photomatix Pro to perform the HDR merge and image alignment — functions performed better by PS.

  1. Select the bracketed RAW files in LR then select File…Export…Photomatix Pro. But make sure you change the File Settings in the Export One File dialog box to DNG. If you don’t do this, you’ll be sending LDR images to Photomatix Pro and therefore losing highlight and shadow detail. (For an in-depth explanation, see Are You Wasting Dynamic Range?)
  2. Photomatix Pro will merge the DNG (RAW) files into an HDR image.
  3. In Photomatix Pro use tonemapping and other adjustments to create an LDR image from the HDR image.
  4. Click on Save and Re-Import and Photomatix Pro will return a 16-bit TIFF to LR.
Workflow #7: HDR Efex Pro

Nik Software’s HDR Efex Pro is the new kid on the block. While I like Nik’s control-point UI and have settled on their Silver Efex Pro as the #1 plugin for monochrome, HDR Efex Pro has become a tool I don’t use much any more. But many people use it, so I’ve included it in my workflows.

  1. Follow steps 1-4 in Workflow #1.
  2. In PS select Filter…Nik Software…HDR Efex Pro. 
  3. After tonemapping and possibly making other adjustments in HDR Efex Pro click OK, which returns a 16-bit LDR image to PS.
  4. Continue with step 8 in Workflow #1.
Single-Image HDR

When your source is just a single RAW file, there’s no need for the Merge to HDR Pro step. In fact, you can’t run Merge to HDR Pro with just one image. In this case you need to get your RAW file to your tonemapping utility as directly as possible. Starting in LR, use one of these tools, ranked in order of my personal preference:

  • LR/Enfuse: Because it’s simple, this is a workflow I sometimes use for single-image HDR. Just follow the same steps described for Workflow #3.
  • Photomatix Pro: You can’t send a single RAW image through PS to the Photomatix Tone Mapping Plugin, but you can export a single RAW image directly to Photomatix Pro, which automatically returns the resulting image to LR. Use File…Export…Photomatix Pro, but make sure you change the File Settings in the Export One File dialog box to DNG as described in Workflow #6 for multiple source images using Photomatix Pro.
  • HDR Expose: This workflow is more cumbersome than using LR/Enfuse or Photomatix Pro, but it’s the process I use for single-image HDR if I’m not satisfied with the results from the simpler tools. HDR Expose is the standalone version of 32 Float, and it can open one or more RAW files directly. Unfortunately it can’t deal with a single image exported from LR, so you need to start HDR Expose and open the file from there. This means you’ll have to save the output of HDR Expose as a file (typically a 16-bit TIFF) then import that back into LR.
  • 32 Float: Because PS’s Merge to HDR Pro won’t accept a single RAW image, the only way to get the full dynamic range of a single RAW image into PS is via the extended-EV TIFF method, described below.
  • HDR Efex Pro: This application does not include a RAW file processor. The only way I know to preserve the full dynamic range of a single RAW file for HDR Efex Pro is to use the extended-EV TIFF method.
The Extended-EV TIFF Method

This is a method for recovering data from RAW files when the tools you’re using cannot read those RAW files directly. It is not required or recommended for any of the multi-image workflows above. The only time I use it is when I want to process a single RAW file using 32 Float or HDR Efex Pro, which is now pretty much never. You’re probably better off just using another tool such as LR/Enfuse, Photomatix Pro or  HDR Expose, but if you don’t have one of those applications or plugins, the following is your best choice.

I first learned this technique from Klaus Herrmann in the section in his online HDR Cookbook entitled Creating HDR Images the Right Way. (Look for the Five TIFFs method.) A RAW file can contain image data from the darkest shadow detail to the brightest highlights, spanning a range of 10EV-12EV. But an LDR file such as a 16-bit TIFF can only represent 6EV-8EV. The idea of the extended-EV TIFF method is to replicate the wide dynamic range of data found in a RAW/HDR image using a bracketed set of LDR TIFFs from that image. Each TIFF file will contain the data from a different (but overlapping) portion of the RAW image’s brightness range.

  1. Open the RAW image in a RAW file processor application such as Lightroom, Adobe Camera Raw (Photoshop), Phase One’s Capture One Pro, Nikon’Capture NX2, etc.
  2. Using the app’s Exposure slider or equivalent, reduce the luminosity by 4EV.
  3. Save the image as a 16-bit TIFF, preferably in the ProPhoto RGB colorspace, with a filename that both identifies the original image as well as the adjusted EV. Something like “IMG1234_-4ev”.
  4. Repeat the previous two steps so you end up with five TIFFs, one each with exposure adjustments of -4EV, -2EV, 0EV, +2EV and +4EV.
  5. Check the +2EV and +4EV images and decide if they should be included in the set. While the -4EV and -2EV are very likely to include extra highlight data that appears blown out in the 0EV image, the same is often not true for shadow recovery. If the +2EV and +4EV images don’t contain true shadow detail that doesn’t appear in the next-lower-EV image, don’t use them — they won’t add detail to the shadows, but they will increase the noise.
  6. Treat these three, four or five images as bracketed originals and submit them as input to your HDR merge application such as HDR Pro (Photoshop), HDR Expose, Photomatix Pro and HDR Efex Pro. You can see the results of my tests of this method at HDR Tools Comparison.

If you’re working with multiple bracketed RAW originals and you’re still getting blown-out highlights when tonemapping, you may be able to recover them using a variation of the extended-EV TIFF method:

  1. Create a 16-bit ProPhoto RGB TIFF with no exposure adjustment from each RAW original.
  2. From the darkest (lowest-EV) RAW image, create two additional TIFF files: one darkened an extra -2EV and another at -4EV.
  3. If you started with three RAW images, for example, you’ll now have five TIFFs.
  4. Use these TIFFs instead of the RAW files as the source images for your HDR merge process.
  5. As with the single-RAW image variation, you can also try +2EV and +4EV TIFFs made from the brightest RAW image, but again to avoid noise don’t use them unless they provide details in the shadows that don’t already appear in a lower-EV image.

[Please post all comments questions on this Google+ page since there are far more photographers there than are reading this blog.]

Aligning Images for HDR

This is the fifth in a series of articles abut passing images between Lightroom, Photoshop and various plugins:

One step in the process of creating an HDR image is the alignment of the bracketed originals. If you’re working with a single RAW image (possibly extracting extended highlight or shadow detail using Klaus Hermann’s Five TIFFs method) this isn’t a problem as you’ll only have one image or the pixels in your extra images will be perfectly registered. If you’re using multiple exposures and a tripod, you won’t have perfectly aligned pixels, but they’ll be close. However, if you’re shooting bracketed exposures handheld, aligning your source images presents an interesting software challenge.

For this article is used two bracketed images (to keep things simple) that were shot handheld. The images below show how well four different applications were able to align the images.

  • Photomatrix Pro (upper left) doesn’t appear to have a particularly good alignment algorithm, although we might be seeing some other artifact of that apps’ processing. It’s hard to tell
  • LR/Enfuse (upper right) does such a bad job of aligning the two images (using default settings) the offset is almost the entire width of these 200% scale images.
  • NIK’s HDR Efex Pro (lower right) looks a little better than Photomatix Pro but still not as good as Photoshop (next).
  • From Lightroom, Merge to HDR Pro in Photoshop CS5 (lower left) appears to do the best job of aligning the images.

   

   

But while Photoshop may be the best tool for aligning images, we also know that it’s one of the weakest HDR applications for other reasons. How then can we take advantage of Photoshop’s alignment feature while using one of the other superior HDR apps?

For my more critical HDR images, my workflow now includes the following steps:

  • Select all the images (including any extended-EV TIFFs) in Lightroom.
  • In the Lightroom menu: Photo > Edit in > Open as Layers in Photoshop…
  • In Photoshop, select all layers.
  • From the Photoshop menu: Edit > Auto-Align Layers > Auto.
  • Crop the image (all layers) to eliminate areas that aren’t present in all layers.
  • Save each layer as a 16-bit ProPhoto RGB TIFF.
  • Use these derivative TIFFs as the input to my HDR application of choice.

To see the results of this portion of the workflow, consider the following images, each produced in this way.

  • Photomatix Pro (upper left) has been improved somewhat.
  • LR/Enfuse (upper right) has gone from worst to best.
  • HDR Efex Pro (lower left) has also improved a bit.

This portion of my workflow is fairly time consuming and I don’t use it all the time. But when I have an HDR image that’s critical, particularly when it was shot handheld or if I’m trying to achieve a particularly realistic (non-HDR-ish) look, this is what I do. And increasingly I’m using LR/Enfuse as my HDR tool of choice.

   

[Please use Google+ for comments: http://goo.gl/Retbw]

HDR Tools Comparison

[Update 1/15/12: Added tests for Unified Color's HDR Expose 2]

This is the fourth in a series of articles abut passing images between Lightroom, Photoshop and various plugins. The other posts include:

For this installment I ran a very simple test of five different tools for merging bracketed images. I started with a single RAW image — see the previous article – then created three 16-bit ProPhoto RGB TIFF files (0EV, -2EV, 2EV). I fed these TIFFs into the applications then generated the JPEGs below using the apps’ default settings.

   

   

 

It’s important to remember that this test was performed with just one class of image. As you use images with different parameters (or take advantage of the various adjustments available within each app) your mileage may vary greatly. For example, LR/Enfuse offers relatively few parametric options and it’s not an interactive program, whereas the other three are WYSIWYG apps.

However there are a few observations that I see consistently. First, Nik’s HDR Efex Pro and Adobe Photoshop’s HDR Pro are generally inferior to the other two applications. Second, LR/Enfuse and Photomatix Pro are remarkably similar other than the latter’s obvious color shift, which is easily correctible. HDR Expose looks equally as good although the default settings appear to over-saturate some portions of the image. Because it’s quite fast and simple, I find I use LR/Enfuse more and more, particularly when my goal is to create a realistic (rather than stylized, grungy, etc.) final image. LR/Enfuse is inexpensive donationware based on the open-source Enfuse application. [I learned about HDR Expose after this article was originally posted. I have not evaluated it other than to perform these tests, but my initial impression is that it's quite good at least as for as these tests are concerned.]

[Please use Google+ for comments: http://goo.gl/d45YK]

RAW File Processing: Photomatix Pro vs. ACR/Lightroom

[Update 1/15/12: Added tests for Unified Color's HDR Expose 2]

This is the third in a series of articles abut passing images between Lightroom, Photoshop and various plugins. The other posts include:

In this post I’ll examine the best way to get RAW files into Photomatix Pro. We’ll look at three methods: (1) using the default Lightroom method of exporting an image to Photomatix Pro; (2) opening RAW files directly in Photomatix Pro (and using Photomatix Pro’s RAW file converter); and (3) exporting bracketed TIFF files from Lightroom and then opening these files in Photomatix Pro (ie, using Lightroom’s RAW file converter, essentially that of Adobe Camera Raw or ACR).

We’re only trying to evaluate these workflows and the quality of the two different RAW converters, so to keep things simple we’ll start with a single RAW file. The first/left image below is a low-dynamic range (LDR) JPEG exported from that RAW image in Lightroom with no corrections or adjustments. It’s generally overexposed, but there are substantial areas that appear entirely blown out. However, because the original is a RAW image, we can possibly recover some detail from those highlights that don’t appear in a 0EV LDR JPEG.

      

The second image was made by selecting Plug-in Extras…Export to Photomatix Pro… in Lightroom. This creates a single TIFF image which is then opened by Photomatix Pro. I used Photomatix Pro’s default settings and returned a 16-bit sRGB TIFF to Lightroom, which I then exported as the JPEG above. There is virtually no recovery of any burned-out highlights in the resulting image.

The third image is the result of opening the original RAW file directly in Photomatix Pro, using its default processing, saving the results as a 16-bit sRGB TIFF from which the above JPEG was made. You can see that some of the details in the blown-out areas have been recovered.

I then used the technique which I recommend in order to save/recover as much dynamic range as possible when working with Lightroom and Photomatix Pro to create the image below.

The steps to create the above image are a bit more complex that the other workflows. The first step was to export three different 16-bit ProPhoto TIFF files from Lightroom. All three were from the same original RAW file as was used for the first two images, but the Exposure slider in Lightroom’s Develop module was set to 0.00, -2.00 and -4.00 respectively. I then opened all three TIFFs in Photomatix Pro as a set of bracketed originals. As you can see, this process recovered far more detail in what at first appeared to be the blown-out areas. (Don’t worry about what appears to be a shift in color. This is a separate issue and is easily resolved.)

These tests support the general understanding of many experienced HDR photographers that the Lightroom/ACR RAW file processor is dramatically superior to the one built into Photomatix Pro and that Lightroom’s Export to Photomatix Pro should be avoided in all cases. To take advantage of this workflow in realistic situations (eg, when you have bracketed originals instead of just one) I recommend following Klaus Hermann’s Five TIFFs method.

Update: Based on a friend’s suggestion, I decided to compare Unified Color’s HDR Expose to Photomatix Pro. Below are two images from HDR Expose. The first was made using the application’s own RAW processor. The second was made from the same three bracketed TIFFs as used for the last Photomatix Pro test.

   

If you ignore the differences in saturation, which are fairly easily corrected, these two images are quite similar to one another and to the three-TIFF Photomatix Pro output. When you consider that the left image was made without the hassle of first exporting three different TIFF variations, it appears that HDR Expose would be a better choice for this test case. This is even more so when you further consider that the left image above requires less color hue and saturation correction.

[Please use my Google+ page for comments. There are a lot more photographers there!]

The Lightroom “Edit in…” Problem

This is the second in a series of articles abut passing images between Lightroom, Photoshop and various plugins. The other posts include:

[See Update at the end of the post. 4:20pm PST on 1/10/12.]

Yesterday I posted the first in this series of articles (Are You Wasting Dynamic Range?) about problems passing images between Lightroom and Photoshop, plugins, etc. Today I want to look at just one such situation, the use of the Edit in… menu option in Lightroom 3. An hour ago I saw the announcement of Lightroom 4 beta. I wonder if this is something Adobe would consider fixing for version 4.

The Challenge

We start with the image below of the Golden Gate Bridge at night. Not only is the image generally overexposed, the highlights are completely blown out. Can we rescue this image? It’s easy to darken it overall, but what about those highlights?

Luckily it was shot in RAW so there’s a lot of data in the highlights and (to a lesser extent) the shadows that we can’t see in this uncorrected version. Just to be clear, what you see below is not a RAW image. There’s no way to display a RAW image in a browser or on your screen. The dynamic range of the image is just too great. Instead, what you see here is a JPEG derived from the RAW file using the default Lightroom settings. [Note: You can click on any image to see a larger version.]

Uncorrected

Correcting in Lightroom’s Develop Module

Below is an example of what we can do in Lightroom (or pretty much any other RAW file processing application) to recover the highlights and reduce the overall exposure. (For the curious, the settings are Exposure=-2.55, Recovery=65.) Additional corrections could certainly be made, but this illustrates what we’re trying to show. We’ve recovered a lot of detail in the highlights in both the moon’s reflection on the water and the city lights in the distance.

Corrected in Lightroom

Passing the Image to Photoshop

Now suppose you use Lightroom to organize your images, but you want to make your corrections in Photoshop rather than in Lightroom’s Develop module. Select Edit in…Photoshop CS5 from the Photo menu or right/ctrl-click on the image in the grid. Photoshop starts up and shows you pretty much the same thing as the first image at the top of this post.

But if you now try to recover the highlights, the image below is pretty much the best you can get. (Exposure adjustment layer with Exposure=0.38, Gamma Correction=0.37.)

Edit in...Photoshop CS5 and Corrected There

This looks a little better than the original, but not as good as the image corrected in Lightroom. Why? It’s because of how Lightroom passed the image to Photoshop. Rather than passing the original RAW file with all it’s wide-ranging data, Lightroom created a TIFF file, which is inherently low-dynamic range) and passed that to Photoshop instead.

From the Lightroom…Preferences…External Editing menu you can select the format and colorspace of the intermediate images passed to Photoshop, but a 16-bit TIFF using the ProPhoto RGB colorspace is the best option available. You cannot pass a DNG or other RAW-file image with high dynamic range.

What to Do?

Does this really make a difference to you? It all depends on whether you need to recover highlight or shadow details in your images. If not, then go ahead and use the Edit in…Photoshop CS5 feature. But if you want to recover any such data, there are two choices. First, you can do your recovery using the Lightroom Develop module and then Edit in…Photoshop CS5, in which case your corrections will be baked into the intermediate image. Alternatively, you can open the original RAW file in Adobe Camera RAW (ACR) or in Adobe Bridge, which will start ACR for you. This will allow you to make the same corrections as you can in Lightroom’s Develop module since it uses the same RAW file processor engine as ACR.

And if you shoot in JPEG instead of RAW, you don’t have to worry about any of this. Those highlights are burned out and gone for good. There is nothing to recover from a JPEG, which is already an LDR image. You can go from high-dynamic range (HDR) to LDR, but there’s no poing in going the other direction. Once you’re in the LDR world, the extended data is forever lost.

Edit in…(something else)

What about all those other options under the Edit in… menu? If you have some plugins installed, you might see them listed there. Passing an LDR intermediate image to these plugins is the only option. Therefore, if you want to recover highlights or shadows, you must do that in the Lightroom Develop module before invoking the plugin. My suggestion is that you also perform certain other Develop-module tasks before invoking the plugins such as Lens Correction — the plugin may delete the lens’ EXIF data — and preliminary sharpening and noise reduction.

[If you have any comments or questions related to this post, please make them on my Google+ page. There are a lot more photographers reading g+ than reading my blog!]

Update!

I’ve discussed this issue and my idea that Edit in… might support DNG intermediates in LR4 with Eric Chan, one of Adobe’s ACR gurus on the Adobe Labs Forum. Eric made a good case for why this might not be an appropriate feature. You can read the discussion here.

But over on Google+, Marko Haatanen provided a solution. In LR you can Photo…Edit in…Open as SmartObject in Photoshop… It won’t appear as though you’ve successfully moved the RAW file into PhotoShop, but if you double-click on the SmartObject in the Layers panel Photoshop will open the image in ACR. And if you’ve previously made adjustments in Lightroom’s Develop module, you’ll see them there, slider-for-slider. (Remember LR’s Develop module is virtually the same as ACR.) Very cool.

If you select two or more images in LR and go for Photo…Edit in…Open as SmartObject in Photoshop…, you’ll get the same number of images in Photoshop. But if your goals are either HDR or you just want to mask-in selections from multiple selections, what you really want is a single image with a SmartObject layer for each original. Again on Google+, Tarun Bhushan showed me how to do this. “In PS, click on a Smart Object layer in one open document and use Duplicate Layer. In the dialog that comes up, choose the destination document as the one where you want the Smart Object to be as a layer. Now you will have the two Smart Objects as layers in one document that you can then manipulate independently.”

I’ll have more to say about this as I continue to explore some of the best options for HDR in particular.

Are You Wasting Dynamic Range?

[Update 1/15/12: Added tests for Unified Color's HDR Expose 2]

This is the first in a series of articles abut passing images between Lightroom, Photoshop and various plugins. The other posts include:

The other night at the meeting of our local photo club’s HDR Special Interest Group, we began a discussion about the preservation of the full dynamic range of RAW images when you use plugins, exports and scripts in Lightroom and Photoshop. I made the statement that, for example, when you Export from Lightroom to Photomatix Pro, the default is to pass the source images as TIFF files, which inherently reduces the dynamic range and looses data. A few people challenged that assertion, so I’ve set out to research it in some detail. This post represents the first round of my test results.

Note that this discussion does not apply only to HDR. The principles apply equally to exporting any RAW images to Photoshop or any plugin. [Spoiler: I'm going to demonstrate why you should use Adobe's Digital Negative (DNG) file format when exporting images to Photoshop or Photomatix Pro.]

Buzzword Backgrounder

Let’s make sure we understand the classes of image-file formats. Only the RAW file formats (.NEF, .DNG, .CR2, etc.) can preserve the full dynamic range of data captured from your camera’s sensor. Once you convert to any other format (.TIFF, .JPEG, etc.) you will lose dynamic range. It doesn’t matter what colorspace you use (sRGB, Adobe RGB, ProPhoto). It doesn’t matter whether you use 8- or 16-bit encoding. And it doesn’t matter whether you select compressed or uncompressed options. All file formats other than RAW (or true HDR, which is rarely used) are designed for viewing or printing and are therefore inherently low dynamic range (LDR) to match the LDR-only capabilities of our displays and printers. If you add bit depth (switching from 8-bit to 16-bit) you’re just increasing the number of colors that can be represented and therefore minimizing banding. You are not significantly increasing the dynamic range of what can be represented.

RAW, HDR and LDR

It’s also important to understand that we don’t have the tools (hardware or software) to view the full dynamic range of a RAW or HDR image. Even if you shoot in RAW format, the image you see on your camera’s display is an LDR derivative. If you open your RAW images in Lightroom or Adobe Camera RAW (ACR), you’re again looking at an LDR derivative. And when you merge images in Nik’s HDR Efex Pro, HDRSoft’s Photomatix Pro and Unified Color’s HDR Expose the output is an LDR image. (We often look at an image and say, “It’s an HDR” or “It looks like HDR.” In fact, these are LDR images created from one or more originals or HDR intermediates. The data are only truly HDR while you’re within the HDR apps.)

The RAW File Converters

There’s a class of applications called “RAW file converters” which includes Lightroom, ACR, Phase One’s Capture One Pro, Nikon’s Capture NX2, etc. These apps have one goal: to create an LDR image from a RAW file. In doing so, the dynamic rage of the image will necessarily be reduced and data will be lost. The adjustments (sliders, curves, etc.) within these apps allow you to decide which data are removed and which are preserved, but “preserve all” is not an option. You have to lose something in order to create an image that can be viewed or printed. (Note that Photoshop cannot directly process a RAW image. If you try, Photoshop will first launch ACR and require you to create an LDR image that is then passed into Photoshop.)

The images below help to explain this point. I started with a RAW file that’s just one of a bracketed set. This is the -4EV (ie, most underexposed) of the set of five. I loaded this RAW file into Lightroom and then created the two images shown below. [Click on any image in this post to see a higher-resolution version.]

   

Yes, both of the above JPEG (ie, LDR) images were created from the same RAW file original. The left one used the default settings in Lightroom’s Develop module. For the right, I used Exposure=+4.00, Fill=70, Recovery=100 and Brightness=0. I could have used Adobe Camera Raw (ACR) instead of Lightroom and achieved the same results because the RAW processing engine of both apps are identical. Note that in order to bring out the detail in the shadows, I had to compromise and let the highlights burn out.

Using LDR file formats it takes multiple images to represent the full dynamic range of even one RAW image. You can’t squeeze all this information into a single TIFF or JPEG even by just reducing the contrast. If you try, you’ll lose too much tonal distinction. That is, levels of brightness will clump together and you’ll end up with tonal banding.

As you can see, there’s a lot more information in the shadows of the original RAW image than you might think if you only saw the first image. The goal I want to explore is how to ensure that all of that information is available within Photoshop or the various plugins and HDR applications.

Some popular HDR tools such as HDRSoft’s Photomatix Pro also can accept and fully exploit RAW images, but if you pass those RAW images to these applications from Photoshop, Lightroom, etc., you may unknowingly be first converting your RAW files into an LDR format and throwing away substantial detail that you cannot ever recover. Lightroom’s export to Unified Color’s HDR Expose is unique in that the default is to pass the full RAW image to the plugin. Because NIK’s HDR Efex Pro apparently cannot process RAW files directly, this is exactly what will happen if you use that plugin.

Exporting from Lightroom

What happens instead if you export the RAW image from Lightroom to Photoshop using the default settings (16-bit Adobe RGB TIFF). This is what you’ll see. It looks pretty much like the default JPEG from Lightroom.

But suppose you then want to use Photoshop to recover that shadow detail? The image below shows what happens when you add an Exposure adjustment layer.

It’s clear that some detail in the dark areas can be recovered, but the image is very contrasty and saturated and the highlights are now even more blown out. Certainly a lot of information has been lost by using a 16-bit TIFF as an intermediate format.

Could the problem be with the choice of the colorspace in the intermediate TIFF image? The images below were created in the same manner as the above image except that I used the sRGB (left) and ProPhoto (right) colorspaces.

   

None of these images comes close to the JPEG I was able to create directly from Lightroom. Specifically, none of the images made using an intermediate TIFF and Photoshop were as good in recovering shadow details. (Check the area under the statues at the very center/bottom of the image.)

Using DNG as an Intermediate Format

If you want to export images from Lightroom to Photoshop, how can you avoid this loss of data? The simplest solution I know of is to use Adobe’s DNG format for intermediate files. The image below shows that result. When Photoshop opens the DNG, it first launches Adobe Camera RAW (ACR). This gives you the opportunity to extract the extended-range data before creating the LDR image used in Photoshop.

As you can see, this is quite similar to the JPEG created directly in Lightroom. Again, that’s because Lightroom’s Develop module is based on ACR. Like Lightroom, ACR is also a RAW-file converter, which means it generates an LDR image from a RAW file. It’s that LDR image that is passed to Photoshop when you Open a RAW file from ACR. And as with Lightroom, that means there’s the potential for losing even more data.

Exporting for HDR Processing

If DNG is the best intermediate format between Lightroom and Photoshop, what about getting images into our HDR tools such as Photomatix Pro and HDR Efex Pro?

Starting with the same single unmodified RAW file in Lightroom, I ran an Export to Photomatix Pro. Here’s the result using Photomatix Pro’s default settings and three different intermediate options: (1) 16-bit Adobe RGB TIFF (Lightroom’s default); (2) 16-bit ProPhoto RGB TIFF; and (3) DNG.

      

The DNG version is again quite superior to the TIFFs.

What about exports from Lightroom to HDR Efex Pro Pro? The images below were exported from Lightroom to HDR Efex Pro using: (1) 16-bit Adobe RGB TIFF; and (2) 16-bit ProPhoto RGB TIFF. I had to use some rather extreme settings in HDR Efex Pro to make the images look even this good: Exposure=+1.7EV, Contrast=+25%, Saturation=-45%, Blacks=+85%. As far as I can tell, there is no way to pass an image from Lightroom to HDR Efex Pro as a DNG or other type of RAW file.

   

And what about the newcomer, Unified Color’s HDR Expose 2? The default export to this plugin apparently passes the RAW file, and the RAW converter is quite good. The image below is the result of the default export from a single RAW image to HDR Expose 2.

Conclusions

I’m going to run a few more tests. In particular, I want to demonstrate how it is possible to use extended-EV TIFF files created from RAW files as intermediates. I first learned this technique from Klaus Herrmann in the section in his excellent online HDR Cookbook entitled Creating HDR Images the Right Way. (Look for the Five TIFFs method.) I also want to get some feedback from other photographers who have studied this. I expect I’ll have to make a few corrections to this post even as far as I’ve gotten so far.

It seems DNG is the best format for a simple export from Lightroom to Photoshop or Photomatix. Unfortunately, HDR Efex Pro doesn’t support this. For that application, you should use Klaus’ Five TIFF method, which is a lot more time consuming. If you’re using HDR Expose, you don’t have to worry — the default work well. More to come.

[See also the next post in this series: The "Edit in..." Problem in Lightroom.]

Autofocus Calibration

I only brought one lens to the #MarinPhotowalk on Sunday: a Nikkor 135mm f/2 AF DC, which I bought used a few weeks ago. I shot everything wide open at f/2. But when I uploaded the images, it seemed as though every one of them was a little soft. In fact, the in-focus point was just a bit farther than where I intended, usually someone’s closest eye.

So today I dragged out my MTP LensAlign and checked that lens on my D700, which I also bought used about a year ago. Sure enough the tests were back-focused about 1/2″ at f/2 and a distance of six feet. That may not sound like much, but with that configuration the depth-of-field is only +/-0.36″.

I assumed the problem was just for my 135mm f/2 lens, but no. It’s off the same amount for all my lenses. The good news is (a) it’s adjustable in my D700, and (b) instead of changing it for each lens, there’s a Default setting that applies to all lenses. The range of correction is +/-20. (I don’t know what the units represent.) It took -13 to get it right.

I mentioned this to Scott Loftesness, who wasn’t aware that you can also do this on a Canon 5DMkII. I know it’s also adjustable on my cropped-sensor Nikon D7000.

The Lens Align MkII is way overpriced at $79.95, but it does work quite well. I haven’t seen discount prices anywhere. I recommend borrowing one if you can. BorrowLenses.com rents the Pro version for only $11 for three days. The difference among the models is relatively insignificant.

I previously bought the Spyder LensCal which is a few dollars cheaper, but I sent it back to Amazon because the construction quality was so poor.

Chronology of a Lost Month

This is one of those too-much-information (TMI) posts. I spent most of October 2011 in the hospital due to a liver abscess and subsequent complications, but due to my condition and medications I remember very few details of what happened during that time. I’ve reconstructed the following chronology based on notes taken by, and discussions with my wife, son, sisters and doctors. I wrote this primarily for myself, so I’d have a better understanding of the events. Warning: I don’t expect it will be of much general interest. And it’s long!

October 1-2

The last week of September 2011 was filled with photography: two photowalks with Google+ friends in Santa Cruz and Half Moon Bay; a third in Golden Gate Park with a group from the Marin Photo Club; and a long afternoon/evening private shoot on Alcatraz with Joe McNally. I was also scheduled to co-lead one of the Worldwide Photowalks with Catherine Hall in San Francisco on Sunday, October 2.

But on Friday it all caught up with me. I was wiped out and felt like I had really done too much. As the day progressed, I felt worse and worse. I knew something was seriously wrong. I had only a slight fever, but the shivers and shakes were severe. I was producing very little urine, and what I did produce was extremely dark brown. Uh-oh, I thought: There’s a problem with my kidneys.

At 3am on Saturday October 1, I called the advice nurse at Kaiser Permanente, our HMO. She asked the appropriate questions and said she’d discuss my symptoms with an M.D. on duty. The advice nurse called back and told me to head to the lab for some blood tests as soon as they opened that morning. My wife had her own health issues (retinal bleeding) and couldn’t drive, so I drove myself to the lab, had the tests and drove home. Within two hours I got a call from the doctor who told me I should come in to see her ASAP and to plan on heading directly to the emergency room from there.

I drove myself to the ER, which probably wasn’t too smart. My wife rode with me, but I think it was a bit of a wild ride. In the ER I was quickly put on IV saline and given lots of tests: a chest x-ray showed some pneumonia; there were kidney problems and bowel problems. (The first doctor was concerned about a possible bowel obstruction.) Finally a CT scan showed a mass “the size of a large grapefruit” in my liver. They didn’t know what it was, but the possibilities included a malignant or benign tumor or an abscess. The only way to tell would be a biopsy, and since it was now Saturday night and the procedure couldn’t be done until Monday morning, I was admitted to the hospital and waited 36 hours. My fever was up to 103.4.

October 3

Monday morning they wheeled me down to a department I’d never heard of: Interventional Radiation, where a team of doctors and assistants perform surgical procedures with the aid of live x-rays, fluoroscopy, etc. It’s pretty wild. There’s a lot of hardware and computer-enhanced imaging and the team members are all wearing lead coverings. They inserted a large needle into my liver. I was quite sedated, but I do remember the point at which one of the doctors said, “We’re getting pus,” or something like that. I realized that was good news. It was an abscess, not a tumor. They drained some of the fluid and inserted a drain tube, which I kept for the next four weeks. They never did perform a biopsy of the liver itself.

It was getting harder and harder to find veins for IVs, so somewhere along the line — I think it was this day — a nurse inserted a peripherally inserted central catheter (PICC) line into my left arm. This is a catheter in the vein that runs up the arm, across the chest and into the vena cava. It’s not as creepy as it sounds. Now I had two ports that could be used for IV medications and even blood draws without having to poke a new vein each time.

I spent the rest of the day in a recovery room, but I was getting sicker and sicker by the hour, so they moved me to the intensive care unit (ICU). The pain in my abdomen was now excruciating when I breathed. I had developed a pleural effusion in my right lung. My breathing was faltering and my arterial blood gases weren’t good. I became acutely septic. In other words, I was one very sick puppy.

They started respiration therapy and told my wife if I couldn’t breathe more deeply they would have to put me on a ventilator. I was holding my breath to avoid the pain, but apparently no one understood that. I wasn’t able to communicate because I was just too out of it. Finally a nurse figured it out, got me started on continuous morphine — it apparently took quite a bit — and she and my wife  talked me into exhaling instead of holding my breath. Ventilator avoided.

The good news at this point was I didn’t have liver cancer. The bad news was I was going downhill fast from a bacterial infection in my bloodstream and messed up blood gases due to my poor breathing.

In the meantime, the labs identified the primary/original culprit: a bacterium known as Streptococcus milleri. This bacterium is one of those that we all have in our mouths, stomachs and intestines, but if it gets into our bloodstreams it’s extremely toxic. Normally the liver can deal with small amounts of milleri, but in my case there was way too much and the liver was overwhelmed. Rather than filter it, as the liver normally does, it created a separate space (the abscess) where it kept the infection segregated. Because of the size of the abscess, the infectious-disease specialists thought this process had been going on for four to six months. They started an IV antibiotic that was specifically targeted to Streptococcus milleri.

October 4

Tuesday the doctors were worried I was developing Acute Respiration Distress Syndrome (ARDS), which along with the acute sepsis was a potentially fatal condition. I had pneumonia, but they weren’t too worried about that since the antibiotics should be taking care of it. I also had serious edema. I had gained 25 pounds of fluids since this whole thing started.

October 5

The next morning my team decided to perform a thoracentesis. Using local anesthetic, they inserted a needle below my right shoulder blade into the pleural cavity around my lung to remove some of the fluid that had accumulated there. They removed 200-250cc of fluid, but another x-ray revealed 600-700cc — two-thirds of a litre! — remained. At this point I was getting a chest x-ray at least once every day, which continued until my last day in the hospital. The fluid they removed was more similar to the fluid from the liver than the doctors liked. They told me it was not uncommon for the infection and fluid from a live abscess to reach the lung, sometimes via a “tunnel”. The liver is just under the right side of the diaphragm, which is why it had become so painful to breathe. The doctors increased my pain medication again.

October 6

By Thursday the size of the liver abscess was dramatically smaller. My fever subsided and my blood gases and blood pressure, which had been very low, returned to normal. In the afternoon, after three days in the ICU, I was transferred to a regular floor. But the doctors continued to drain more fluid from my pleural cavity. The subject of thoracic surgery was raised, but only if the abscess “shelled” out from the liver or the fluid in my lungs developed into too many separate “pockets” such that it couldn’t be easily drained.

Unfortunately, I developed swelling in my left arm and an ultrasound revealed I had developed a blood clot — an acute deep vein thrombosis (DVT) — from the PICC line. I was immediately started on Heparin to thin my blood and reduce further clotting.

October 7-9

Over the next three days my condition improved steadily. I still had a lot of tubes including an IV (saline, two antibiotics, pain meds, diuretic), an arterial catheter (to track blood gasses), the liver drain and the IV PICC line, but with the help of the physical therapists I was finally able to get out of bed (to use the bathroom instead of a bedpan!) and to walk. At first it was just a few steps but by Sunday I was able to climb twenty steps, a requirement for going home.

But the hospital is a quiet place on weekends. A number of departments have minimal staff on Saturday and Sunday and in general it seems procedures and decisions tend to be delayed until the start of the week when the full team returns.

October 10

On Monday things got busy once again. It started with another CT scan, which unfortunately showed more fluid was accumulating in the pleural effusion, which had also become loculated — split into a number of those separate pockets. This meant they couldn’t be reasonably reached with further thoracentesis, so the only solution was thoracic surgery, and for that I’d have to be transported to the larger hospital in San Francisco. Furthermore, the original liver abscess wasn’t resolving as quickly as the doctors had hoped. It hadn’t changed in size since the previous CT scan. To make matters worse, I now had several small blood clots, which could be new or could have broken off from the original one. These would need to be watched.

The medical-transport crew arrived like a SWAT team of paramedics. With a drill-sargent nurse as their crew chief, they whisked me off by ambulance to San Francisco. Once there, the PICC line in my left arm was removed and replaced with a new one in my right arm.

October 11

Tuesday was thoracic surgery day. I learned they would make multiple relatively small incisions and use a camera and tools to remove the material of the pleural effusion. There was a 20% chance it might be worse and they’d have to perform a more complex procedure with a lot more cutting. (That didn’t happen, thank goodness!) Post-op there was some chance of scarring of the lungs and that I could lose 10-20% of my lung capacity.

I woke up after the surgery in the cardiovascular (CV) ICU in a lot of pain and with some new tubes: a catheter in my bladder and drain tubes in my chest. I still had the liver drain as well. To make matters worse, there was a mixup on my pain medication and I was without it for many hours. Of the times I can remember, this was the worst for me. Finally they gave me IV Dilaudid (synthetic morphine) and things calmed down.

October 12

Recovering from the surgery, was tough. Again, this may be because I was also becoming more coherent and aware than I had been for the first eleven days of this ordeal. I was still in the CV ICU. My edema had become quite serious, but the diuretic (Lasix) was finally kicking in. My blood pressure reached 215/145, and it could only be measured on my legs because I had a clot in one arm and the PICC line in the other.

October 13

Thursday was more of the same, but I was stable so they moved me out of the CV ICU to a CV post-surgery floor. This is also pretty much the first day I can remember. Except for the day just prior to surgery, the first twelve days are still pretty much a blur to me at best. I think it’s a combination of medication and the body’s reaction to the infection, procedures, etc.

Along with full consciousness came real discomfort. I still had serious edema. My feet looked like bloated potatoes. My body temperature swung back and forth between cold chills and hot sweats, which was a bigger problem than it might sound. The pain was weird. It was never acute. In fact, it took me a while to realize it was actually pain because it was so non-specific. All I knew was that I was extremely uncomfortable and the only solution was narcotics. After surgery I was given an intravenous Patient Controlled Analgesia (PCA) device. You get a button you can press to get an immediate small dose of Dilaudid, then it’s locked out for some period of time like 10 or 20 minutes. I also had oral narcotics (Norco/Vicodin and Percocet) but these took 45 minutes or more to work. (Actually, the Percocet just made me stupid. It didn’t seem to do anything for the pain.) The PCA was great when the pain broke through the other drugs and I couldn’t wait until the next oral dose took effect. Unfortunately, I worked hard to use it as little as possible, so they took it away! That will teach me. I guess if I’d pressed the button a few times an hour they might have thought I needed it longer.

My other error had to do with food. The regular menu was decent, but they had to test my blood sugar before every meal and possibly give me a somewhat painful insulin injection depending on the results. I’m not diabetic, but I guess the blood sugar/insulin relationship frequently gets weird after surgery like mine. So I made a deal with one of my doctors. I said I’d be willing to eat the diabetic menu if I could stop getting the tests and injections. Big mistake! Whatever you’ve heard about hospital food, there’s nothing as bad as the food they serve to diabetics. I don’t really how to describe it other than to say it has no flavor whatsoever. And of course there’s nothing interesting on the plate to begin with.

October 14

The main event Friday was a transesophageal echocardiogram. From the time of the original diagnosis, the infections-disease doctors were concerned about my heart. Streptococcus milleri often causes endocarditis, an infection of the heart valves. The test sequence for this is rather strange. First they perform a non-invasive (ie, from outside the chest) ultrasound transthoracic echocardiogram. If the results are negative — ie, they don’t detect any endocarditis, as was my case — then they perform the more invasive transesophageal echocardiogram. This is one of those procedures where you have to swallow an ultrasound transducer to get it into your esophagus, which positions it close to the heart valves so the radiologists can get a very clear picture. As with some of these other procedures, it wasn’t as bad as it sounds. My results were thankfully negative for so-called vegetation on the valves.

But I was still quite uncomfortable. I wasn’t allowed to get out of bed on my own until I had the approval of the physical therapist. I had six different IV drip bags feeding the two ports in my PICC line: saline, two antibiotics, blood-pressure medication, pain medication and the diuretic. But for the first time since the earliest days in the emergency room, they removed the supplemental oxygen I’d been breathing. And with the help of the pain medication, I finally got a good night’s sleep.

October 15-16

Finally, on Saturday, the doctors began talking about my going home. Each department had their own requirements for my discharge. The surgeons wanted my chest drainage to stop. They discontinued the active (vacuum) suction and let gravity take over. Other MDs were still concerned about the known DVT (blood clot) in my left arm and possibly one in the right arm, so I was wheeled down to radiology for another set of ultrasounds. The left arm clot was smaller and there was no sign of one in the right. Regardless, they told me I’d be on steady Heparin until my release form the hospital, then three months of Warfarin (coumadin) at home. I was also still receiving intravenous antibiotics (Ceftriaxone) for the original liver infection, which would also be continued after my discharge. As before the lung surgery, physical therapy wanted to make sure I could get in and out of bed on my own and walk up and down stairs. The bed was still a challenge, but I was able to walk about 1,000 feet and climb 24 steps. My edema was still a problem, so the IV Lasix continued.

I was feeling better and getting stronger every day, but still had a lot of pain from the surgery. The pain had shifted from the incisions to my ribs. The surgeon explained that they had to spread the ribs, tweak some muscles and stretch the cartilage to do what they needed to do, hence the pain I was feeling.

Like I said before, not much happens on weekends, so Sunday was just another day of waiting, trying to control the chills, sweats and pain, and wishing I hadn’t opted for that diabetic menu. On occasion I was able to talk a technician or food server into giving me something I wasn’t supposed to have. I was desperate for anything with flavor.

October 17

Monday morning the first string returned to work and I was able to get the procedures and tests I needed to wrap things up. The most significant was another visit to the Interventional Radiation department to check and reposition the drain in my liver, since that one was going to stay in even after I went home.

October 18

After 17 nights in two hospitals, I was finally discharged. At the very last minute, a physician’s assistant from the surgery department came and removed my chest tubes. Just like he said, it hurt slightly for about three seconds and then it was done.

Because of the long wait through the weekend, I was fairly strong and stable on my feet. Physical therapy had given me a cane, but I no longer needed it. I could walk up and down a full flight of stairs, and while it was a bit awkward and painful, I could get into and out of bed on my own.

I went home with a fair amount of paraphernalia and medications including:

  1. the PICC line in my right arm so I could self-administer intravenous drugs and get blood tests without another needle in my vein each time;
  2. a “JP” drain in my liver with an external suction pouch, which I safety-pinned to my clothing;
  3. lots of bandages over about eight incisions and other wounds from various procedures;
  4. intravenous Ceftriaxone (Rocephin), an antibiotic to kill off any remaining Streptococcus milleri;
  5. Metronidazole (Flagyl), another antibiotic to fight an amoebic infection they thought I could have. I was still taking it because a few weeks were required to get the results of the lab tests I had early on;
  6. Warfain (coumadin) for the potential and known blood clots; and
  7. Lisinopril for my blood pressure which had become higher than normal during my hospitalization.

Recovery

Once home, my recovery progressed quite rapidly. My wife, a retired R.N., changed my dressings. Kaiser has an amazing Home Infusion service, which delivered and monitored my IV antibiotics. I went frequently to the local outpatient infusion center for blood tests and PICC-line dressing changes. After about three weeks, I ended the antibiotics and the PICC was line removed.

On November 1 I returned to the Interventional Radiation department in San Francisco. It was supposed to be another “drain check” procedure, but they removed the liver drain altogether without me even knowing it. Never felt a thing.

Because of the DVTs (blood clots) I’m still taking Warfarin daily and getting blood tests once or twice a week to monitor the clotting times. This should end in early January 2012.

I’ve also been getting chest x-rays nearly once a week. I still have a pleural effusion: something (liquid or some kind of gunk) between my right lung and the pleural lining. This reduces my lung capacity by 10%-20%. My doctor says it will eventually dissipate, but it sure is taking a long time. I do notice that I have non-infectious pneumonia-like symptoms. There’s occasional slight pain, and I sometimes get a little short of breath and get tired a bit more easily than I’d like. But I don’t generally notice theses symptoms. Compared to how I felt six weeks ago, I’ll take what I’ve got.

How Did This Happen?

Okay, so how did all this happen? What was the cause of the Streptococcus milleri liver infection in the first place?

As I mentioned, this is a bacterium that we all have in our mouths, stomachs and intestines, but it’s toxic in the bloodstream. My team of infectious-disease doctors were hardcore medical detectives. They were extremely inquisitive about my travel, activities, diet and dental history. One doctor in particular kept asking me about recent dental work. Yes, I’d had a cleaning by a dental hygienist, and I got a new crown during the summer, but that didn’t seem to be it. I had to think back. What happened in the April-May timeframe of when this might have started.

And then it occurred to me. Back in the spring, my hygienist convinced my to start using a Waterpik in addition to flossing, and I did so every day. But there was one area of my gums that always bled. If I’d read the instructions for the Waterpik, I’d probably have found they said something like, “If your gums bleed, stop.” But I just figured I needed to toughen up those flabby gums, so I kept using the Waterpik on them night after night. Every night they bled. And every night that opened a pathway for more Streptococcus milleri to enter my bloodstream.

We’ll never know for sure. The evidence doesn’t give us a provable cause-and-effect relationship, but the circumstantial evidence is so strong, that I and my doctors are satisfied that my constant traumatizing of my gums was the ultimate cause of my live abscess. Not surprisingly, my Waterpik was swiftly and unceremoniously disposed of once I got home.

Cessna’s Vision Problems

My wife, Cessna, has been having some vision problems, and since so many friends and family members have asked for details through email and all the social networks we’re on, I thought it might be easiest to publish the info here and just link to it. This is my version, not hers, so I hope you and she will forgive my inaccuracies.

For a number of years Cessna thought she had more floaters in her eye than normal. Three years ago her optometrist noticed some minor irregularities on her retinas, but no one thought it was anything serious. Cessna teaches Aikido so she rolls and falls a lot, and at one point she noticed a few flashes in her eyes and other irregularities. She had some tests by our HMO’s opthalmology department, which merely confirmed that yes, there was something on her retinas, but no big deal.

Then in June of this year she started having more serious things in her left eye: blobs of stuff, larger than the usual floaters, that more severely obscured her vision. She returned to opthalmology who diagnosed her with retinal vasculitis, inflammation of the blood vessels in the eye. They suspected the inflammation was causing a reduction of oxygen in the blood and the eye to therefore generate additional (undesirable) blood vesels as well. The blood vessels (veins, arteries or both) were leaking some blood into her eye, and that’s what was obscuring her vision.

The doctors started running all sorts of tests: x-rays, MRIs, blood tests, etc. They also performed eye angiograms using injected fluorescein dye, which enhances the image of the blood vessels and apparently can actually show the blood leaking. They confirmed their diagnosis, but Cessna had a pretty nasty allergic reaction to the fluorescein. She had a second angiogram and the reaction was so severe, even with a dose of Benadryl, the assistant was ready to administer an EpiPen.

Although the doctors were moderately confident in the diagnosis of her condition, they still didn’t know what was causing the inflammation so they didn’t know how to treat it. They told Cessna that if it didn’t go away (and it didn’t appear to be) she might need to start using immune-suppressant drugs like Humira to reduce the inflammation. These are nasty drugs for anyone, but Cessna doesn’t have a particularly strong immune system to begin with. The doctors also said they might want to inject Avastin directly into her eyes to halt the growth of new blood vessels. (Yeah, my thought, too.)

The optometrist suggested Cessna get the advice of a nutritionist, which she did. If there was anything that might avoid the immune-suppressant drugs it was worth a try, so Cessna had another slew of blood tests looking for food sensitivities. The result was that she started an incredibly strict diet in mid July. We’re not just talking gluten-free; we’re talking everything-free. It’s a diet based upon her specific sensitivities.

That’s the course she was following until a week ago, when her right eye (which was the good one) suddenly became completely occluded. She couldn’t see anything through the floating mass. We went to the opthalmic experts the same day and they told her it was blood — a lot of it. But the recommended treatment was the same: maybe the Humira, but just wait and see. Well, that’s a bad joke since she now couldn’t see well enough to drive or do many other things. And there wasn’t any indication it was getting better. In fact, it was getting worse.

We decided to go outside of our HMO and get second opinions from other local opthalmic gurus, and that’s what we’ve been doing for the past three days. Here’s what we’ve been told so far:

  • The diagnose is still retinal vasculitis.
  • They don’t know the cause, but there are still a few more tests that will be done.
  • If they can’t find a treatable cause, they may want to give her prednisone. But that’s a steroid with all sorts of bad side effects — Cessna had an aunt who died from taking it long term — so it can’t be used for more than a few months.
  • If that doesn’t work, then they’re talking about Humira, etc.
  • They used ultrasound today to determine that she doesn’t have any retinal detachment. (I got to see this in real time. It was an amazingly clear picture, taken through the eyelid and all.)
  • The blood in the right eye might dissipate by itself, but it’s going to take “months”. If it doesn’t, they’ll have to remove the blood surgically.
  • The doctors really want to see what’s going on in the right eye, but they can’t because the blood is in the way. Once it’s gone (on its own or via surgery) they want to do another fluorescein angiogram, but due to Cessna’s reaction she’ll need prednisone and Benadryl beforehand.
  • Once her eyes clear, the doctors also want to go in there with lasers and zap the extra blood vessels. Apparently she’s already permanently lost vision in those spots anyway, but it doesn’t sound like they’re too critical.

At this point Cessna’s hanging in there. It looks like she’s got another week of tests and doctor visits, and the extreme diet continues. The best news came today from the first doctor to tell her that he didn’t expect any of this to be permanent. He wasn’t sure, of course, but he told her she should expect to recover her vision.

Thanks to everyone for your good wishes. I’ll pass them on to Cessna.

Labs and Papers for Black & White

This post is a review of black-and-white printing on eight different papers from four U.S. photo labs.

I’ve been uploading my recent photos to Google Plus, where I’ve been getting good feedback and meeting great photographers. When I published this b&w image of Bubba’s Diner in San Anselmo, California, the comments were particularly enthusiastic. And then, totally out of the blue, two people said they wanted to buy prints. How cool is that? I didn’t get (back) into photography to sell my images, but why not? If someone can get pleasure from hanging one of my photos on their wall, that would be pretty cool.

Bubba's Diner, San Anselmo, California

Bubba's Diner, San Anselmo, California

How to sell prints to my first two customers? I quickly cleaned up my SmugMug portfolio at DougKaye.com — it still needs a lot of work — and upgraded to a Pro account so I could order the prints through there and even sell them directly. But before I accepted money for my work, I wanted to know what the prints would look like, so I decided to order prints of most of my portfolio images for myself. SmugMug uses two labs, and I opted for BayPhoto, which appears to be their more high-end lab. (The other, ezprints, is somewhat less expensive.) I first ordered a print on Kodak Endura paper, which SmugMug/Bay Photo refer to as their Lustre stock. When the print arrived, I was rather disappointed in the color and texture of the paper. So I turned to other photographers on Google+ and asked them what labs and papers they used for b&w. I got a few recommendations and then ordered prints from four labs (including BayPhoto) on eight different papers. Here’s a summary of my opinions, listed by the coolness/warmth of the papers, starting with the coolest. It’s not an exhaustive test, as I’m sure there are far more papers and labs out there. But if you’re thinking about black-and-white printing, this may be a helpful starting place.

Bay Photo’s Lustre is Kodak’s Supra Endura VC, a resin-based photographic paper finished with a “fine grain pebble texture,” which is too much artificial texture for me. SmugMug recommends it as a compromise between full matte and glossy and as a way to minimize fingerprints. I expect my prints to be matted and mounted behind glass, so fingerprints aren’t really an issue. This is the coolest of all six papers. It actually has a noticeable blue cast to it. I’d say it’s my least favorite of the batch. ($3.23 via SmugMug for an 8×10 color-corrected print. Direct from BayPhoto: $3.50, or $1.79 without color correction.)

Bay Photo’s Metallic (Kodak Endura Metallic VC) is actually a touch warmer than the Endura, which shows how cool/blue the regular Endura really is. The metallic is obviously very glossy and has a bit of a greenish cast to it. The whites and highlights are very reflective/silvery, hence the metallic moniker. I don’t think I’d be likely to use this paper. ($4.12 via SmugMug for an 8×10 color-corrected print. Direct from BayPhoto: $4.03, or $2.06 without color correction.)

Bay Photo’s Glossy (also a Kodak Supra Endura VC) is the third coolest paper, and still not particularly warm. The blacks are deep and there’s pretty good detail in the shadows. I’d probably use this for images where I wanted to emphasize the drama of a contrasty, particularly crisp picture. ($3.23 via SmugMug for an 8×10 color-corrected print. Direct from BayPhoto: $3.50, or $1.79 without color correction.)

MPIX offers a paper they call True B&W, Ilford’s True B&W. This is a silver photographic process, so there are no color dyes or inks at all. It’s yet another cool paper, almost as cool as the Bay Photo papers. Like the Endura Metallic, it has a slight greenish cast. Of all the printer/papers combinations, it’s the lowest contrast. There are no deep blacks and it has the least detail in the shadows. The opposite of Bay Photo’s Glossy paper, I might use MPIX’s True B&W when I particularly wanted a softer, gentler low-contrast look. ($2.49 for an 8×10 print)

I wasn’t really thrilled with any of these combinations, so I asked Matt Russell, a friend who shoots and sells a lot of b&w landscapes, about the high-end labs he uses. He suggested I look into West Coast Imaging and Digital Silver Imaging. WCI has a $250 minimum order, but they were willing to work with me on these tests. Obviously, you don’t want to order one or two 8×10′s at a time from WCI.

DSI uses Ilfospeed Resin-Coated paper with an Ilford Pearl (lustre) surface for their Custom RC prints. This is another lower-contrast combination, but not as low contrast as the MPIX True B&W. The blacks are also deeper and richer than the MPIX, but still not as deep as others. Furthermore the blacks are rather warm. It’s a very nice combination: a neutral paper with slightly warm blacks. One of the best. ($18 for the first 8×10; $9 for prints 2-10.) DSI also offers a less-expensive Direct to Print option (ie, not their Custom service) that delivers Ilford RC Pearl prints for much less ($4.59 for 1-9 8×10 prints; $4.19 for 10 or more).

DSI’s Custom Fiber Base prints are on Ilfobrom Galerie Fiber paper. This is fairly warm paper, but the blacks are actually cooler, similar to the MPIX True B&W. It’s a heavy double-weight semi-gloss fiber paper, about the same weight as the Ilford Gold. It’s in the lower-contrast category like the MPIX True B&W and the DSI Custom RC, but not as low-contrast as the others. The paper is quite warm, but the blacks are cool (again like the True B&W). The depth and richness of the blacks are excellent as are the shadow details. ($38 for the first 8×10; $25 for prints 2-10)

WCI offers Ilford Gold (Ilford Galerie Gold Fibre Silk), warmer still than DSI’s Custom Fiber Base. It’s a very heavy paper made with real rag and has a marvelous rich look to it without sacrificing good, deep blacks. ($12.22 for the first 8×10 of a single image; $10 for prints 2-9; $8.33 for 11-.)

WCI also offers Silver Rag (Crane’s Museo Silver Rag), a 100% cotton paper. It has slightly more rag texture than even the Ilford Gold. This is the warmest of all the papers I tested, with a slightly yellow cast. I would use this paper if I wanted a particularly warm look. Otherwise, I’d stick with the Ilford Gold. (Same price as WCI’s Ilford Gold.)

With the exception of the Ilford True B&W paper used by MPIX, all of the above have deep, rich blacks. It’s possible that a different print on the True B&W might not have such a low-contrast look. But while all the others have solid blacks, all but the two WCI combinations do so by increasing contrast and therefore losing some detail in the shadows.

There’s no question that the more costly prints from DSI and WSI are superior to the others. DSI’s Custom Fiber Base prints are downright expensive.

I’m sure your experiences vary and you probably have used labs and papers not listed here. Leave your reactions in the comments for all to see. DSI’s pricey Custom Fiber Base prints are perhaps the best of all for most of my work, but damn expensive. It’s the one option that starts to become more than a substantial part of the total (including matting and framing) costs. A 12×18 costs $88 plus tax and shipping. For most high-quality work, I’d probably chose Ilford Gold from WSI if I had enough work to justify their $250 minimum order. Otherwise, I’d probably go with DSI’s Direct to Print Ilford RC. For by far the fastest service and the lowest cost (and so long as I wanted a very crisp look), I’d use Bay Photo’s Endura Glossy. WCI’s Silver Rag is an option I’d reserve for those times when I needed very warm (almost toned) whites.

Fair Use?

This is a fascinating case, particularly for me both as a photographer and a fair-use advocate. You should probably read the story for yourself, but I’ll summarize it here. Andy Baio is well-known and respected in the tech world. He produced an album (Kind of Bloop) based on the songs from Miles Davis’ classic album, Kind of Blue. He got all the permissions and rights he needed to the music, but when it came to the album art, he created a somewhat pixelated version of the original image without getting any permission. It turns out the orignal album-art photo was taken by and belongs to a great photographer, Jay Maisel. Jay sued Andy and they settled out-of-court for $32,500. Andy still feels he was right based on the concept of “fair use.” Here are the two versions: Jay’s original and Andy’s interpretation.

kind_of_bloop_comparison-20100701-172352

What do you think? Should Andy have been able to sell his album using the cover on the right without first getting permission from Jay? Would you say that Andy’s version qualifies as “fair use” of the original? It’s a tough call for me.

First, you should know that I’m a supporter of and contributor to the Electronic Frontier Foundation (EFF), who played a role in this case, so I’m a strong believer in the fair-use concept. I believe our copyright laws are severely inhibiting creativity and are increasingly just serving a copyright consortium rather than serving the public good, as originally intended. I have some experience in copyright, trademark and other intellectual-property law, but I am not an attorney. I’m a layperson who has taken an interest in this area for decades. Most notably, I am not up-do-date on the latest details of the fair-use doctrine. In other words, I’m not qualified to give a legal opinion about who is right or wrong in this case — only an emotional one.

Given that disclaimer, I do have an opinion, event though it’s not based in law. To me, I think Andy’s image is a derivative work that goes beyond what I consider to be fair use. From a purely practical point, I can’t figure out why Andy didn’t try to get permission to use Jay’s image in the same way as he did for the music? Did he think it was somehow more incidental? If you’re a photographer, your images are as important to you as a song might be to its composer. This is an iconic album cover, which on one hand suggests that it’s fair game for fair use, but it’s also a work of art and deserves the same protections as any other.

Ultimately, Andy asks an important question at the end of his blog post (scroll to the bottom of the page) where he writes, “Extra credit: Where would you draw the line?” Is there some point in abstracting the image at which the original image is obscured to the point at which the derivative work is no longer infringing of Jay’s copyright? Is this even a legitimate way to evaluate the issue? A fascinating debate in any case. What do you think?

Update: I should have mentioned that I first heard about this from Thomas Hawk, for whom I also have great respect. In this case, however, I disagree with him. But check out Thomas’ blog post and the comments.

Salvaging the Shoot

Once again, I’m determined to get the shot. In this case, it’s the full moon rising behind downtown San Francisco. Last night was my first attempt, but given the horrible results, it won’t be my last. I was about to delete all the images from the session, but first I decided to play with them to see how much I could extract before giving up.

Like all serious shoots, it began with research.

  • The experts told me the best time to shoot is when the moonrise is 30 minutes before sunset. That’s often the night before full moon on the calendar. In this case (June 14, 2011) moonrise was at 7:48pm and sunset was8:33pm. Not a bad spread.
  • To find the best position I used The Photographer’s Ephemeris, an awesome iOS app that shows you the exact position of the sun and moon on any date at any time.
The Photographer's Ephemeris

The Photographer's Ephemeris

Everything was ready, save for the one big fear: the fog, which everyone knows can come barreling in through the Golden Gate during the summer. But fog didn’t turn out to be the problem. Due to a moderate high-pressure system just offshore, there was no marine layer and no wind. And that meant haze and smog: a fairly heavy layer up to about 1,000 feet. Yuck.

But having gone this far, I schlepped all the gear (including a second body+tripod for a timelapse) to the location where I found three other photographers, all with Nikon gear. Two of them had pinpointed the location using The Photographer’s Ephemeris as well. It was so hazy, we couldn’t even see the moon until it was well above the skyline, so the photo below is one of the first of the evening. And one of the best. This was shot about 25 minutes before sunset.

Original from the Camera

Original from the Camera

As you can see, it’s horribly flat and dull. After some tweaking in Lightroom, I was able to recover some of the contrast and clarity:

700_9034

With Global Lightroom Tweaks and Crop

Yes, I could have further lightened the unnaturally dark and saturated water and made a number of other improvements, but I just didn’t want to waste a lot of time on this one.

I posted the tweaked image on Facebook, where photo pal Scott Loftesness suggested I see how it looked as a black-and-white. I popped it into Silver Efex Pro 2, where I spent some time making a number of global and local adjustments and ended up with this:

700_9034-Edit-Edit

Further Tweaked in Nik Silver Efex Pro 2

What do you think? It’s still not at all the shot I’m looking for, but compared to the original, I think it’s at least a serviceable image. If nothing else, it shows that if you keep working at it and consider all the options (b&w in this case) you can sometimes salvage a shot that would otherwise end up in the trash.

Update: I went back and tweaked the moon. First I changed the mapping from RGB into b&w, then I adjusted the contrast. Finally, I used a layer mask in Photoshop to merge the enhanced moon into the original image. It gives the picture an entirely different look, doesn’t it?

700_9034-Edit-Moon-720w

Happy Birthday, The Conversations Network

Yesterday was the 8th anniversary of IT Conversations, the longest running podcast in existence and the flagship channel of The Conversations Network. Since its founding, The Conversations Network has published 2,918 audio programs for an average of one every day for these past eight years.

Thanks to our members,major supporters and TeamITC, the wonderful folks you never hear about that bring you those new programs every day.

I’m a TWiP Again

Once again I had the privilege of being a guest host on the This Week in Photo podcast (#202), sharing the show with Frederick Van Johnson, Syl Arena and Ron Brinkmann, three of my personal photo heros.

On this episode of TWiP, in case of a water landing – take pictures, Getty Images acquires PicScout, Adobe gets touchy feely, and an interview with SnapKnot.com co-founder Reid Warner.

My first appearance was on episode #153, nearly a year ago.

Photography Workshops

Like any other photographer, I’m always looking for ways to improve my skills. There are a lot of options out there: books, magazines, community college classes, online videos (free and $$$) and local photography clubs. And then there are the photo workshops — they’re everywhere. I’ve attended two workshops in the past few months, and while that certainly doesn’t make me an expert, I do now feel like I know what to look for in the next one. (I’m not including the San Francisco stop of the FlashBus 2011 Tour, which was fun, but more of an event than a workshop.)

Artist's Road, Santa Fe, at Sunrise

In March I attended a workshop led by Derrick Story. A good friend, Scott Loftesness, had been to one of Derrick’s earlier workshops and enjoyed it. Since I was able to talk Scott into trying another one with me, and because Derrick’s classroom and studio are in Santa Rosa, California (just an hour from home), it was a low-risk investment. The two-day workshop included eight students and cost $495. Derrick provides lunch both days, but you’ve got to get yourself to Santa Rosa and pay for a hotel room unless you’re local.

Santa Fe Cathedral at Sunset

Two weeks ago I went to a very different kind of workshop: the Mentor Series Photo Trek in Santa Fe, New Mexico. This three-day program had 37 students, two instructors, a bus and driver for the first two days and cost $1,000, which included no food, housing or transportation to/from the event. Mentor Series is owned by Popular Photography and runs about a half-dozen  workshops each year all over the world.

So how did they compare? In the case of the Mentor Series Trek, it’s “trek” that’s the operative word. It’s more about the location and somewhat less about photography. Yes, the attendees are all photographers (some with some very fancy gear) but you spend virtually all your time on the go. The first two days we were on the bus getting from one scenic location to the next a few hours each day, and once we arrived, there were often miles of walking to do. Beautiful scenery to be sure, but more hiking than shooting. And certainly not a lot of time to stop and “work” a subject for an extended period. The best shooting was actually the day they dumped the bus and we walked the city of Santa Fe on foot: once at sunrise and once at sunset. [Santa Fe is one of the best cities I've ever shot in. You could easily spend two or three days just walking its streets with a camera. Great art and architecture, terrific light and shadows, and a community that is very accepting of (and used to) photographers wandering around.]

By comparison, Derrick Story’s workshops often include a location such as a local safari park or (as next month) an early morning balloon launch, but there’s usually just one outside event per weekend. The rest of the time is spent in his studio — he usually includes at least one model session — and in the classroom. And it’s the classroom (and the class size) that really sets the two experiences apart. Derrick spends some of his time actually teaching from a podium and he gives the students actual assignments. For example, he might send you into the studio to shoot a model using only a single strobe. That’s something you can do when there are only eight students and they break into groups of four. With 37 students — forget it; everyone is on their own.

This brings up the question of why take one of these workshops at all. Professional photographers on assignment are obviously going to shoot a lot. But we serious amateurs have an interesting challenge. When my wife and I recently went to Egypt, I would have loved to have been able to stop and spend an hour or two studying the light and playing with the composition at each location. I would have given up half or more of the less-visually interesting sites in order to have more time at a few of the good ones. But that’s just me. My wife doesn’t particularly enjoy standing around while I study and experiment, and certainly the 22 other non-photographers in our tour group wouldn’t stand for it.

In one sense this is the role that weekend or weeklong workshops play. They allow the serious amateur to immerse him/herself in photography, surrounded by other photographers in a context where their peculiar habits of stopping, studying and shooting are socially acceptable. I imagine this is why Trekkies go to conventions. Wearing Mr. Spock ears to the grocery store is going to earn you some very strange looks. At a workshop you can truly geek out. Even when you’re on a bus, it’s all photography. All the time.

And what about the other students? Looking back, it’s not too surprising that a group of 37 would include a wider range than one of only eight. But I was surprised that the Mentor Series Trek included same true novices, some with the most expensive DSLRs. There were times when the instructors had to explain the relationship of aperture to shutter speed and ISO, and that surprised me. The instructors were even cornered by students with questions like, “What is ISO and how do I set it on my camera?” or “How do I focus this camera?” (Perhaps not surprisingly, some of these technically naive students sometimes produced some of the compositionally most exciting pictures.) In the smaller group of Derrick Story’s workshop, the range of skills was somewhat narrower although it still varied more than you might expect. Derrick does a good job of giving assignments that are applicable to each student’s skills.

In Santa Fe, I had relatively little access to the instructors given the 1:18.5 ratio as opposed to 1:8 at Derrick Story’s workshop. But even in Santa Fe, they were there if you had an important question. Towards the end of the Mentor Series weekend each student had the chance to show each of the instructors five images for critique (ten images total), and those sessions were quite valuable. We each got four or five minutes of constructive criticism that was appropriate for our skills.

Another benefit of any workshop or joining a photo club is the chance to see how other photographers interpret the same objects and locations. This happens in both the small and large workshops. No matter your level of experience, there are always those moments of, “Wow, I missed that!” that are truly educational.

So which of these two (or any other) do I recommend? It depends on what you want, of course. If pure learning is your goal, then I’d recommend a workshop with the smallest number of students, even a day of one-on-one. And I wouldn’t worry about finding the absolutely best photographer. So long as it’s someone whose work you respect and has been shooting it for a lot longer than you, you’re going to learn. Of course, reviews and opinions of previous students will help a lot.

On the other hand, if it’s a destination you particularly want to shoot or if you particularly want to travel, a larger more-distant workshop might be better for you. Mentor Series, for example, runs treks to places like Switzerland, London, Hawaii, Sedona and Wyoming. If you’re drawn to one of those locations and you want to experience the places in the context of photography, these might be better choices for you.

As for me? My prejudices probably show through in this blog post. I’m signed up for Derrick Story’s Hot Air Balloon Photo Workshop in a few weeks. None of the Mentor Series treks are on my calendar. I’m going to continue looking for small-group workshops that I can get to without hopping on an airplane. I’m also going to spend as much time as possible taking photo walks with friends. For example, tomorrow Scott and I will be shooting at the San Mateo Maker Faire as we did together last year. It’s tremendously visual and there’s enough to keep you engaged for a full day or more.

The Amazon Web Services (AWS) Outage

Like many other sites hosted on AWS, all of The Conversations Network’s websites went down at 1:41am PDT on April 22, 2011. It would be 64.5 hours until our sites and other servers would be fully restored. A lot has been written about this outage, and I’m sure there’s more to come. Don MacAskill, another early adopter of AWS, has posted a good explanation of SmugMug’s experiences during the outage.  Phil Windley and I are hoping to interview our friend Jeff Barr from AWS for Phil’s Technometria podcast once the dust has settled at Amazon.

Many pundits have suggested this event highlights a fundamental flaw in the concept of cloud computing. Others have forecast doom and gloom for AWS in particular. I disagree with both arguments. While it certainly was the most significant failure of cloud computing to date, I predict this event will become not much more than a course correction and a “teachable moment” for Amazon, their competitors, all cloud architects and of course us here at The Conversations Network. For the geeks in the audience, I’m going to describe our architecture, the AWS services we utilize, and give a bit of an explanation about what happened and what we learned.

The Conversations Network utilizes three basic AWS services, plus a few more that aren’t really pertinent to this episode. Our servers are actually instances of AWS Elastic Compute Cloud (EC2) servers. The root filesystem for each server is stored in a small (15GB) AWS Elastic Block Storage (EBS) volume. Not only are these volumes faster than local storage, they’re also persistent. So if/when an EC2 instance stops, the root filesystem for that instance remains intact and will continue to be usable if the instance is re-started. [EC2 instances are booted from Amazon Machine Images (AMIs). In our case, these are based on Fedora 8 (Linux) customized to our standards. The AMIs are identical for all our servers, but the EBS root filesystems, which change dynamically once a server is booted, are unique to each server.]

We also use EBS volumes for non-relational storage. For example, we have one large EBS volume for IT Conversations and other podcast filesystems. This holds all the audio files and images used on the website. We have another for SpokenWord.org, and so on. These EBS volumes are each mounted to one EC2 instance, which in turn shares them with the other servers via NFS. Finally, we use the Relational Database Service (RDS) for our MySQL databases. Like EBS, this is a true service as opposed to a “box” or physical server.

One very important feature of EBS is that you can take snapshots at any time. For example, we make a snapshot each night of each EBS volume. We keep all snapshots of all volumes (other than the EC2 root filesystems) for the past seven days, plus the weekly snapshots for the past four weeks and the monthly snapshots for the past year. The cost of keeping a snapshot is based only upon the incremental differences since the previous snapshot, so it’s quite a reasonable backup strategy even for large volumes so long as they don’t have changes that are both major and frequent.

Designing any server architecture, cloud-based or otherwise, requires that you consider the failure modes. What can fail? What will you lose when that happens? How will you recover? Automatically or manually? How long will recovery take for each failure mode? It’s not about eliminating failures — you can’t really do that. Rather, it’s about planning to deal with them. And like traditional architectures, the cost of the configuration increases geometrically as you increase the reliability (ie, decrease the amount of time it will take to recover from a failure).

We’ve been using AWS for more than four years. During the period when IT Conversations was part of GigaVox Media, we were the basis of one of the first case studies published by Amazon. [Here's a diagram of one of our AWS-based configurations.] Because The Conversations Network (a non-profit) runs on a shoestring budget and can’t afford the level of redundancy deployed by some commercial enterprises (eg, SmugMug), we’re not looking for a particularly high-reliability architecture. Until last week, we’ve have EC2 instances that haven’t stopped in well over a year. We can’t tolerate any significant loss of data so we need the redundant storage of EBS, but a 99.9% uptime is good enough for us, and that’s what we’ve had from AWS until now. Because of our experience with the high-reliability of AWS, we have never gotten around to automating the re-launching of EC2 instances in case of failure. We do use two separate monitoring services, and there are two of us (me and Senior Sysadmin Tim) who are capable of restarting servers, etc., if something does go wrong.

AWS operates in five regions around the world. We happened to pick US East in Virginia instead of US West (northern California) for no particular reason. Within each region there are multiple physical locations called availability zones. These are probably separate data centers within a metropolitan area. The availability zones within a region are connected by very high-speed fiber. This means you can have some degree of geographic redundancy by deploying servers in multiple availability zones, or achieve even greater protection by also deploying duplicate systems in multiple regions. The latter is far more complex, since the connectivity between regions is not as good as between availability zones. Our needs are humble, so all of The Conversations Network EC2 instances, EBS volumes and RDS databases are located in the us-east-1a availability zone. And of course, that’s where last week’s failures occurred.

Amazon hasn’t yet said what the original failure was. All of our EC2 instances were running and they could communicate with the RDS databases. I think the problem might have been the association between the EC2 instances and the EBS volumes. The volumes used as root filesystems were reachable, but not the others that contained our site-specific files.

After a few hours of downtime, I decided to re-boot our EC2 instances and that’s when things went from bad to worse. All of our EC2 instances entered the Twilight Zone. They were stuck in the “stopping” state. The operating system halted (no SSH access) but the servers didn’t release their EBS volumes. I could have launched all-new EC2 instances, but I wouldn’t be able to connect them to the volumes and hence, no websites.

Because of our backup strategy, however, we did have one more option: We had snapshots of our EBS volumes. I could have created all-new EBS volumes from the daily snapshots, and I could have done so in a different availability zone to get away from the problems. But there was one gotcha. We make the backup snapshots at 2am Pacific time each night. The failure occurred 19 minutes before that, which means our snapshots lacked the most-recent 24 hours of activity: new programs, audio and image files, logs, etc. As with the few previous problems we’ve had with AWS (mostly of our own causing) we thought this outage would be fixed quickly. It was a tradeoff: It seemed better to wait an hour or two rather than to re-launch with day-old data.

Of course “an hour or two” dragged on. Soon the outage was 24 hours old; then 48. It always seemed that the fix was imminent, so we delayed the restart process. Eventually, we decided to go ahead, and that’s when we discovered our one real mistake. Remember that we make snapshots of our EBS volumes every night? Well it turned out that we weren’t making those snapshots of all of our volumes. There was one volume that we somehow missed. The only snapshot we had of that volume was from the date it was created, more than a year ago. That means we would have had to launch our sites with some very old data. In this case, when we finally got access to the most-recent data (on the in-limbo EBS volumes) it would be difficult to reconcile it all. In the end, we decided just to wait it out. Finally, after 64.5 hours, the one EC2 instance that was holding hostage our last EBS volume stopped. We were then able to re-attach that volume to a newly-launched instance. We brought up all-new EC2 instances, attached all the then-current volumes and we were up and running, still in availability zone us-east-1a.

So what did we learn from all this? We re-learned that you have to think through these architectures carefully and understand the failure modes. But most importantly, we learned that once you have a good plan, you have to follow through with it. If we had been making nightly snapshots of that one remaining EBS volume all along, we would have been able to re-start the websites with day-old data at any time, regardless of the problems AWS was having disconnecting EBS volumes from running EC2 instances.

I also have a new strategy for deciding when to stop waiting for AWS to recover and instead switch to the snapshots: Once the length of the outage exceeds the age of the backups, it makes more sense to switch to the backups. If the backups are six hours old, then after six hours of downtime, it makes sense to restart from backups. In this case, we should have done that after the first 24 hours.

But we still know we don’t have ultimate redundancy: We still have to re-start things manually. So long as we accept the downtime, we can survive the total failure of the us-east-1a availability zone and even the entire US East region. That’s because all EBS volumes are first replicated to multiple availability zones within the region, and our nightly snapshots are stores in Amazon’s Simple Storage Service (S3), which is replicated across multiple regions. So our current data can survive a failure within a region and our day-old data can survive a failure of our entire region.

We still have a few things to cleanup and repair from this experience, but all-in-all we remain fairly happy with how things turned out. We didn’t, after all, lose any data. And while we aren’t proud that our sites were down for nearly three days, the world as we knew it did not come to an end. Maybe our team is even glad to have a few days off. (Too bad we couldn’t have told them in advance.) We still have one EC2 instance that refuses to stop, but it’s one of those that used NFS to reach EBS volumes attached to another server. Amazon says “We’re working on it.” Other than that, we’re now better prepared for the next failure, so long as its just like this one. Actually, I think we’re in pretty good shape for most events I can foresee. AWS. It continues to be a great platform for us.

Confessions of a Facebook Slut

There’s one problem with being a Facebook Slut (accepting nearly every friend request) and having 700+ so-called friends. I’m now entirely dependent on the filtering in FB’s ‘Top News’ stream. It’s smart enough to know whom I really care about and filters out the rest. The other ‘Most Recent’ stream is unfiltered, so I have to scroll through pages of stuff people who aren’t really friends and family in order to read stuff from those who are. Okay…that works on the website, but as far as I can tell, the FB apps for iPhone, iPad and (my favorite) Flipboard don’t have access to the filtered ‘Top News’ feed. Apparently they can only deliver the unfiltered ‘Most Recent’ feed, which renders them pretty much useless for a slut like me. Unless one of my 700+ friends has a fix.

PocketWizards for Nikon

I’ve been using Nikon’s light-based CLS system for triggering my SB-600 and SB-900 strobes, but as others have experienced, I’ve been running into the line-of-sight limitations of that system. Last week I bought a set of Nikon-specific PocketWizard radio triggers. Learning how they work took a little longer than I expected, but the preliminary results are good. The supplied instructions are rather terse, so perhaps the following will save you some time if you go this route. In addition, you’ll want to refer to the wiki-based online documentation. (The Nikon-specific information is in an appendix.) There are all sorts of peculiarities such as how the PW system interacts with Nikons VR lenses.

The Nikon-specific PocketWizards are primarily designed to work with Nikon’s excellent TTL-based exposure system, iTTL, although they will also trigger older PW receivers. The basic setup is to pop a PocketWizard MiniTT1 transmitter on the camera’s hot shoe and a FlexTT5 transceiver under each Nikon strobe. You then set the strobes to TTL mode, make sure all PW devices are on the same configuration (C1/C2) and you’re set. All strobes will fire in sync and the Nikon CLS will do its thing to compute the exposure. I found:

  • Flash exposure compensation works as usual.
  • High-speed sync (FP) works well to 1/8000 sec, and you don’t have to do anything special to enable it. It just works all the time.
  • Even the modeling light works when you press the camera’s depth-of-field preview button.
  • Don’t put your strobe into Remote mode. Just set them up as though they were connected to your camera’s hot shoe.
  • In this basic configuration, the selection of groups (A/B/C) on the FlexTT5 makes no difference.
  • Automatic strobe zooming does not work, which makes sense whenever the strobes are not in the camera’s hot shoe. You must zoom your strobes manually.

Nikon’s Commander Mode, the ability to adjust the power of remote strobes individually (Nikon menu: Flash Control for Built In Flash) doesn’t work with PocketWizards. Instead, you need to buy a third device: the AC3 Zone Controller. This gadget sits on top of the MiniTT1, which is already atop your camera. The AC3 lets you dial-in power adjustments in 0ne-third stop increments for strobes in three groups (A/B/C). Note that these have nothing to do with Nikon’s A/B/C groups. It took me a while to comprehend this. The remote strobes think they’re each connected directly to the camera’s hot shoe. When used with PWs, the strobes know nothing about Commander Mode. They’re not “remotes” in that sense.

The AC3 really is a must-have unless you’re only shooting manually.  In addition to adjusting the power for each group relative to what Nikon’s CLS/iTTL would otherwise direct, you can switch a group into Manual mode to override the CLS control. The AC3′s +/- control wheel for each channel is mapped into controlling the flash output from 1/64 to full power. Note that so long as you want to use the AC3 for exposure control, leave your strobes set to TTL mode, even if you select Manual (M) on the AC3.

I occasionally use a Sekonic L-358 flash meter, so I decided to buy the optional Sekonic RT-32N module that fits inside the meter and allows one to trigger the strobes from the meter via PW radio signals. It took me quite a while to figure out how to configure everything for this mode of operation. It required changing the internally stored configurations of the PocketWizard devices, which in turn requires that you connect them to a computer via a USB cable, then use the PocketWizard Utility, which you can download from the company’s website. It runs on OS X or Windows and is very simple to use. You can save configurations in files, which makes updating a set of devices a simple matter.

I ended up using the two configuration settings (C1/C2) for TTL and “metered” mode, respectively. Here are the configuration values I’ve used successfully:

Config 1: AC3 for TTL or Manual Exposure Control

  • Strobe: TTL/FP Mode
  • FlexTT5: Normal Trigger Mode, Channel 7
  • Mini TT1: Normal Trigger Mode, Channel 7

Config 2: Sekonic-Meter Triggering and Manual Exposure Control

  • Strobe: Manual (M) Mode
  • FlexTT5: Basic Trigger Mode, Channel 27
  • MiniTT1: C2: Basic Trigger Mode, Channel 27
  • Sekonic: Channel 27, Group A (or other groups as needed)

With the above configuration, you can simply switch all the devices from C1 (for TTL) to C2 (for manual metering). In the manual-metering mode, you no longer have the ability to control strobe output using the AC3. Instead, you have to go to each device and set its power output manually. This is because the trigger signal is being transmitted directly from the Sekonic meter to the FlexTT5 transceivers. The camera, MiniTT1 and AC3 aren’t involved. Of course, you can still press the camera’s shutter release to trigger the strobes, which is why you need to set both the MiniTT1 (on the camera) and the Sekonic meter to the same channel as the FlexTT5 transceivers.

It all makes sense once you work your way through it. Or you can just copy my configurations as a shortcut. You might want to use channels other than 7 and 27 if you’re going to be shooting near me!

After just a few days, I’ve grown to like the PW system, as have most others who’ve tried it. On one hand there are more gadgets, batteries and things to go wrong. On the other hand, they don’t seem nearly as finicky as using Nikon’s optically based system. I can just set and aim my strobes where I wan’t. I don’t need to worry about whether they can read the signals from the camera. Now to see if I can get past the gadgets and make some good pictures with them.

Tracking the Cost of Disk Storage (Feeling Old?)

Cory at BoingBoing blogged David Isenberg’s tracking of the historical cost of rotating magnetic disk storage.

YEAR — Price of a Gigabyte
1981 — $300,000
1987 — $50,000
1990 — $10,000
1994 — $1000
1997 — $100
2000 — $10
2004 — $1
2010 — $0.10

I can remember buying an IBM 5022 disk subsystem in the mid 1970s composed of two 2.5MB platters (one fixed, one 5440 removable cartridge). According to the 1971 IBM press release (http://www-03.ibm.com/ibm/history/exhibits/system7/system7_press.html) the purchase price was $16,225, which comes to $3,245,000 per gigabyte in 1971 dollars or $16,999,437.09/GB in 2009.