[ Previous Tip ] [ Mac Tips ] [ Next Tip ]
This page exists primarily for my own benefit so that I can remember my own photo archiving process. I figured that others that are doing the same thing might find this information helpful too.
iPhoto is a Macintosh application that is designed to allow a user to archive, sort, edit, display, save and print still photographs that have been saved in digital form. It will download photos directly from many digital cameras, but I choose not to do it that way. iPhoto comes with every new Macintosh. The newest computers ship with the latest version, older computers may benefit from an upgrade to the iLife '05 suite ($79 and worth it).
The tools that I use are listed below. If you don't use these exact tools, you may have to adapt your methods to those tools.
In order to really understand the method for importing photos to iPhoto, you really need at least a basic understanding of the iPhoto Library. The iPhoto Library is the place where iPhoto keeps everything related to a batch of photos. The number of photos in the library is more or less unlimited. You will need enough disk and RAM to support a large library but for most computers, several thousand to several ten's of thousands of photos may be a practical upper bound. iPhoto can work with multiple libraries, although the ability to deal with more than one library at one time is limited.
A large library can be taken apart from within iPhoto and all or parts of it burned to a CD or DVD. Individual libraries can be remerged later. However, to make this effective, the individual photos must be originally imported with enough associated data to make search and sorting the library practical.
iPhoto takes your imported photo and installs it into its library as is. It never renames or redates the actual file that contains the photo itself. All modifications that you make from within iPhoto (such as the name, date, ratings or comments) to a photo are stored in a metadata form. Metadata is data associated with an object that is not actually part of the object. This means that the changes are associated with the photo but not actually stored within the photo. You can manually go into the iPhoto library and copy photos back out, but the metadata stays behind hidden in iPhoto's own data structures. This means that you should have the photos properly named and the dated BEFORE you import the photos into iPhoto.
Dates are particularly important to iPhoto. The date determines where the photo is stored. It is also the easiest and most useful way to search and sort on photos. Personal photos with similar dates usually have similar subjects and should sort together. Upon import, iPhoto grabs the date from the date stamp on the file that the original photo is stored in. This may or may not be the date that the photo was taken. If the camera date wasn't set right, the dates are going to be wrong. If the photos were subsequently modified in a graphics program, the date may be changed, especially if the file format of photo is changed. Scanned photos will retain the date when the photo was scanned, not when the picture was taken. These have to be corrected by hand. Fortunately, there is a fairly easy way to deal with dates and filenames in batch form, without involving iPhoto at all, once they are organized into folders together.
The key to making iPhoto the most useful is to properly pre-process the individual photo files so that everything iPhoto needs is there when the photo is imported. iPhoto 4 has limited photo editing capability but if you have any heavy duty editing to do it is better done in another application. It turns out that GraphicConverter ($35 shareware) is an ideal pre-processing tool for iPhoto. iPhoto '05 has better photo editing capability, but any editing that you do may be lost if the iPhoto Library ever gets damaged. This happens so be prepared.
I store all of my photos in a hierarchal file structure after passing them through GraphicConverter. This makes mass importing thousands of photos to iPhoto easy. Even though the computer may take a while reimporting thousands of photos, it doesn't take much effort on my part. Preprocessing photos is still time consuming because some attention must be paid to each individual photo, but GraphicConverter's batch renaming capability makes getting all of the info that iPhoto will need into the filename relatively easy. GraphicConverter can also change the photo's date and time stamp in batches. If the date is also encoded into the photo's file name, there will be no confusion later as to when the photo was taken even if some subsequent editing changes the file's modification date. This, of course, assumes that you actually KNOW the date when a particular photo was shot.
The following table summarizes how iPhoto makes it's metadata from information that pre-exists in the photos themselves.
|Resulting Metadata Type
|Import Method||Original Filesystem Data|
|Date and Time||Individual file import||Modified Date/Time Stamp on the file|
|Folder import||Modified Date/Time Stamp on the individual files|
|Camera import||Date/Time Stamp applied by the camera|
|Title||File import||Filename without extension|
|Folder import||Filename without extension of each individual file|
|Camera import||Camera's default filename (i.e. DSCxxxxx)|
|Film Roll||File import||Date of the import of the file and a film roll numbered in sequence|
|Folder import||Date of import and the name of the immediate folder that contained the individual files|
|Camera import||Date that the photo was imported from the camera and a film roll numbered in sequence|
|Comments||File import||No equivalent|
|Rating||Individual file import||No equivalent|
|Album||Individual file import||Implied by File Date, Default Roll Number as shown in the Library.|
|Folder import||Date of first picture in the folder, Folder Name|
|Camera import||Picture Date, Default Roll Number|
|Smart Album||Individual file import||No equivalent|
|Format||Individual file import||Picture Extension|
|Size||Individual file import||Picture Attribute (if encoded)|
As can be seen from the table, if you download your photos to the filesystem and then individually rename them and place them into folders to organize them, iPhoto will be able to use that information to create new metadata for those photos from both the filesystem name, the filesystem date and the folder name that contains the photos. If your Library becomes corrupted (see below) then you can rebuild an entirely new one by importing the top level folder of your hierarchal file structure. iPhoto will then rebuild its own metadata from the information associated with each file.
If you allow iPhoto to import photos directly from the camera and then organize, edit, redate or rename the photos within iPhoto itself and then your Library gets corrupted, all your metadata can be lost. Your photos are still in the Library and you can copy them out of the dated folders by hand, but they all have the default names originally assigned by the camera. They are sorted by date in the iPhoto Library, but any information on subject matter is history.
Ratings, Albums and Smart Albums are harder to deal with. There is no filesystem equivalent for a Rating. In the event of damage to the iPhoto Library these may be lost. Albums can be recreated by hand from the film roll information easily IF the photos were originally grouped by album. Smart Albums can be recreated automatically from a previously prepared template (see below).
During the process of importing about 16,000 photos from a variety of sources, I've settled on a method that seems to work well and is robust against iPhoto Library issues (see below).
My sources for photos were scans of prints and digital photos taken with five different digital cameras, one dating back to 1996. I gathered the digital photos from a pile of Zip disks, stuff stored on an old laptop (and not properly backed up) and on archival CDs. They had a variety of naming and dating issues and methods. During the process of converting them for use in iPhoto, I managed to rename and redate them so that all that is necessary to know about the photo is actually stored in the filename of each photo and all of the filesystem creation and modification dates represent the actual date of the photo (as close as I could know it anyway).
Many of my 35mm photos were taken with cameras with date backs so that the date is actually printed on each photo. Other dates were gleaned from the photo processor's date printed on the backs of some of them. For others, I simply had to guess and I considered my self lucky if I could place them in the correct year.
All of the photos had been stored in a hierarchal filesystem structures that I allow to grow to the size of a CD to allow easy backup. When a structure exceeds about 650 MB, I lock the old one and start a new one. I could use DVDs for backup, but I consider CD's more stable over time provided that the media was good to begin with and they are properly stored. It also takes a long time to collect enough photos to fill a DVD so that the backups wouldn't happen on short enough intervals. I also keep a dual set of backups with the second set stored at another location. They also get backed up on two different firewire disks, also kept at different locations.
The overall process goes as follows:
The process may look involved, but it's pretty easy. It is a little time consuming though. What makes it easy is GraphicConverter. This excellent shareware ($35) allows you to convert from/to just about every graphic format ever created. Further, it has advanced browse and editing capability. What I use most is the batch filename and file date editing features.
The first step is to get your photos into a place where GraphicConverter can work on them. Mount your camera or memory card in the normal fashion and drag and drop the camera contents into a working folder. You can alternately use Image Capture to automatically download the camera or card, just don't let it import into iPhoto. Then select and drag all of the photos from the camera or card to a temporary working folder that you create anywhere on your disk. When the copy is done, you can unmount the camera or card. If you want, you can delete the contents of the card first, but remember to empty the trash or you won't recover the space on the camera or card.
The second step is to drag the whole folder to the GraphicConverter icon on your Dock. GraphicConverter will open a browser window in which you can preview every photo. You can also trash the ones that you don't want, rotate the ones that need it, mess with the color, brightness, contrast or anything else that you want to do to make the photo ready for further work.
One trick here is that you can rename the photos right in the browser window. To edit a photo name, select the photo and press return. This will highlight the entire photo name and you can overtype the default name directly. However you may not want to do that right now. Having the photos already numbered by your camera is handy for preventing duplicate names later. Just click at the front of the name or use the left arrow key and then type any information that is specific to this individual photo. Press return again to accept the changes. Don't worry about adding information that pertains to several photos, you'll be changing that for all of them as a group shortly. I usually just type keywords for that photo such as the names of the individuals in the photo.
After you've gone through the whole list, select all of them that are related and then right click (or control-click) on them. This will bring up a context sensitive menu. One of the items near the top will be an option to rename the selected files. Pick it and a dialog box will come up. My photos come raw from the camera named DSCxxxxx.JPG or something like that. I want to retain only the last three digits so I use the delete text box to remove the DSC00 part. I then use the add text box to type all of the keywords that I want to use to find these photos later, such as "barbados vacation 25Jul99." I put the date in so that it is hard coded into the filename and if there is any doubt later, the date is there.
Keep doing this by batch until every batch is named for its overall subject.
The third step is to fix the dates if necessary. As you are going though the photos doing the batch rename, keep an eye on the dates. If they are wrong, you can change the selected batch as well. Pick the Date option in the contextual menu and enter the creation and modification dates as desired. Usually these should be the date that the picture was actually taken. It is actually easier to redate a group of photos before you batch rename them. This is because after redating a batch, GraphicConverter leaves the batch selected so that you can immediately rename the batch too. If you rename first, you have to reselect the batch to redate them.
The fourth step is to build a temporary folder structure for the subjects of the photos. Don't do this in your master archive yet, you want to keep the new photos segregated until you are sure that they are all good. You can arrange it any way that you want, just see to it that the lowest level folders (the ones that actually contain the photos) has the general subject of the photos in its name. Let yourself go and get verbose. You've got 256 characters to work with. The folder names will eventually become the film roll names in iPhoto.
The fifth step is to move the photos to iPhoto. This is the easiest step as the hard work is already done. Open iPhoto with whatever library that you want to use (preferably an empty one) and select Library from the source list. Then drag the top level folder that you organized your photos into and drop it on the Library icon. iPhoto will then import the photos, build its thumbnail views and build its metadata structures. All you have to do is sit back and wait for it to finish.
Then check your dates. After importing thousands of photos, you are sure to find ones with bogus dates. You can selectively sort them out with an appropriately constructed Smart Album. My wife's camera also tends to date photos 31Dec97 if the internal clock gets reset. I look for this date too. You might find special dates related to a particular camera.
Since you may have just imported the photos into a temporary library for testing, you then need to import them into your real library. Then move the folders that contain your photos into your master photo archive.
iPhoto seems to be a little less than completely robust when it comes to corruption of its Library. Errors can creep in that iPhoto cannot correct. When too many errors accumulate, or an individual error is too severe, iPhoto will usually choke and quit.
If iPhoto is not totally stable, then why use it? The main answer is that when it works, it works very well. iPhoto does have some advantages over a straight hierarchal file system based database. First, all of your photos are gather together in one place so that searching and sorting them is pretty easy. This can be done as well with other photo viewing tools, but iPhoto does it particularly well. Second, iPhoto can make some really nice slideshows that can be exported and burned on CD or DVD. Third, iPhoto has a built in mechanism for ordering hardcopy photo albums of your own design. I've not used this third feature, but some people like it. I've found through experience that iPhoto IS INDEED stable IF the imported photos are clean to start with. This is why I do all the testing of photos before I import them.
If the Library does get corrupted, it is sometimes possible to rebuild an iPhoto Library by pressing option-shift while iPhoto is starting. You must hold the key combination until iPhoto asks if you really want to rebuild the Library. You will be prompted for the desired name and location for a new Library. iPhoto will then proceed to copy out as much information as it can from the old Library and build a new Library. If you have a Console window open at the time (a good idea because iPhoto sends messages to the Console that it doesn't display by itself) you will see if it complains about any particular photo.
The Library can become so corrupted that the Library rebuild will fail. This is indicated by iPhoto apparently getting to the end of the process, which could take a long time, and then crashing. It may or may not have created a good Library. If the console says something like:
2004-08-11 09:04:17.620 iPhoto *** Assertion failure in -[KeyMgr keyAtIndex:], ScopeApp/Key/KeyMgr.m:141
2004-08-11 09:04:17.715 iPhoto Exception raised during posting of notification. Ignored. exception: Index past bounds
you are hosed. This is a sure indication that the corruption of the Library was so severe that iPhoto could not fix it. At this point you should stop and backup your Library because to fix it, you are going to have to take it apart and put it back together. A workable manual process is described in the next section.
iPhoto is sensitive to bad photos during import. A bad photo might be rejected or it might be accepted. If it is accepted, it might cause problems later. It is a good idea to test the import in an empty Library. This way, if iPhoto chokes during import, it will not corrupt an already good library. Once you have successfully imported all your new photos into a test library and fixed or deleted all the ones that iPhoto complains about, then you can import them into your master library.
iPhoto will complain about bad photos at the end of an import, but it may not get that far. Be sure that you have a Console window open while iPhoto is running so that the messages that iPhoto generates in real time get recorded. Besides, iPhoto's complaint list can't be copied or printed and you can't do anything else with iPhoto until you dismiss the window so that you need another record anyway. It is very important to find and fix all of the photos that iPhoto has trouble with or these WILL give you trouble later.
If iPhoto's Library becomes corrupted but the damage isn't so bad that iPhoto just crashes on startup, then there is a way to retain nearly all of your photos AND their metadata. If the damage is so bad that your photos appear to be gone, you can still recover your photos, but your metadata is probably unrecoverable.
As with ALL of your data, a backup is a good thing. On occasions, and especially just after you have loaded a number of new photos, it is a good idea to make a backup of your whole library. Depending on its size, you can burn it to CD-R or DVD-R from within the Finder. You can also burn a collection of albums from within iPhoto.
Total Disaster, your Library is toast
If the worst possible thing has happened and iPhoto either won't open your library or there are photos missing, they are probably still in the library, stored by year, month and day. Navigate through the various folders and copy them back out. When you copy a photo out of the Library, use option-drag to make a new copy so that the original is left behind. All of the metadata is history at this point, but you at least have your photos. If the photos are actually not there, then there has been some serious disk issue that you should address immediately.
Partial Disaster, iPhoto runs but the Library won't rebuild
In this case, you want to retain as much of your metadata as possible. You don't want to just keep running this way because the problem will get worse to the point of Total Disaster some time in the future.
Set up a series of Smart Albums encompassing whole years for the entire span of your Library. Then burn as many as will fit onto a series of CD-R's or DVD-R's. iPhoto will make new and smaller Libraries on the CD-R's or DVD-R's. Also, some of the bad data that is messing up your Library won't be copied out. Then move your damaged Library to another folder and restart iPhoto. Create a new Library. Then insert each CD in turn. iPhoto will display the CD in the Source window. Drag the CD icon onto the Library icon and iPhoto will copy out whatever it can. Keep a Console window open. If iPhoto finds something it cannot copy, it will post a message on the console. You can then use these messages to selectively locate and re-import the originals of those photos provided that you have another copy somewhere. When you are done importing the last CD, then quit iPhoto. Restart iPhoto while holding down the option and shift keys and rebuild the NEW Library just to make sure that it is clean.
This method retains all of the metadata for entries that are not actually corrupted.
Lightweight Damage, iPhoto still runs but some stuff seems to be missing or not viewable
You may get away with a simple Library rebuild. Quit iPhoto and restart it while holding down the option and shift keys and rebuild the Library.
In the light of the sections above, you might infer that iPhoto still isn't 100% stable and you would be correct. iPhoto '05 is out and it appears to be no better in this regard.
If you use iPhoto as a source for the iPod photo then all you photos have to be in one library. If you've got a lot of photos, your library can easily exceed the size of a DVD so that it will be more difficult to back it up.
What to do?
This is my plan, we'll see how it works.
This works PROVIDED that the suggestions presented in the previous sections are followed. There should be NO editing done in iPhoto itself. Make sure that the photo dates are correct and that file names contain all of the relevant information about each individual photo. It helps if the folder name containing the photos in the master library are relevant to the global topic of the photos contained in the folder. This will make the film roll metadata in iPhoto relevant as well.
I've been doing it this way for over a year, but I haven't had to use this recovery method yet. iPhoto has been stable. This is probably because of the trouble that I've gone to to make sure that the photos that get into iPhoto in the first place are good. I suspect that the majority of problems that I've had in the past were due to undigestible photos. When these are eliminated before the fact, iPhoto seems to work fine.
Digital cameras have been around in a usable form only since the late 1990's. Many of us have still photo libraries that go way before that. All of my stills are in the form of color prints, others may have extensive slide libraries. In my case, these photos represent most of my family's history and while I have double prints of most of them (with the 2nd copy stored in a different location) the life of prints is variable. Some of my prints are fading badly, eventually all of them will. This may or may not happen in my lifetime, but it will happen.
I elected to undertake a task to scan a selected portion of my stills. Had I known how much work it would be when I started, I may not have done it, but after a year, it's done. There are perhaps 10,000 prints that I elected to scan and at least as many that I didn't. I needed to develop a method to triage the photos so that I could determine which to scan and then to organize them before the fact to make the job easier. The following paragraphs describe the method that I eventually settled on after considerable trial and error.
To create a digital image of a print or slide, some form of photo scanner is needed. I used a flatbed scanner (HP PSC 1210xi) that I got at Costco for about $80. This particular scanner has software issues (see HP Scanner Tips) but it did the job. Other scanners would work as well. For slides, a special slide scanner is needed. No matter what hardware is selected, you must play with it for awhile to get the scan parameters right so that you will get good images. I settled on scanning at 600x600 dpi into JPEG format. This produced images of about 1 Mbyte each with good quality when viewed or printed. Lower resolution resulted in some scan artifacts, higher resolution didn't improve the images much but made the files huge. This particular scanner seemed to do a good job in setting the brightness during the scan, I never had to mess with that after the fact. Indeed, some of the scanned images had better brightness and contrast than the original prints, go figure. Make sure that you get the scan parameters right before you start because you won't want to do it over if you eventually don't like the result.
My photos were scattered all over. I found them like this:
There are also many photos that I know that I have somewhere but I never found. There were also many double prints, some kept together, some in different albums or drawers, some in different locations.
This chaos made the initial organization of these photos difficult. I didn't realize until later how important it would be to get them all together and sorted by date until I got deep into the project. iPhoto is very anal about dates. The dates on all photos in iPhoto really need to be right or the sorting and selecting mechanism in iPhoto gets much less effective.
These are the steps that I recommend for organizing this non-trivial project.
This process is the most efficient if ALL of the photos associated with a particular day or event are gathered together. It can be done in pieces, but it takes more work. I recommend disassembling any photo albums but if you weren't the one that put the albums together in the first place, this may require some delicate negotiations. If pre-made albums must be kept intact, then presort the loose photos into stacks associated with the albums. The ones in the albums are probably the best of the batch anyway and once you have them together you can more easily determine which you are going to scan.
Get rid of the dud and duplicates early so that you don't have to deal with them later. Duds are obvious, out of focus, smeared, double exposure, under/over exposure, and those pics of the tops of your shoes. Duplicates come in several flavors. There are absolute duplicates (double prints) and near duplicates. The near duplicates are exposure bracketing, insurance shots or just pictures that have pretty much the same content. There is usually one in the batch that represents the group, you won't need them all.
Once you have downsorted to the ones that you will scan, then sort them by date. This can be hard if the dates are not recorded on the photo somehow. To figure out the dates of some undated photos, I resorted to several methods. One was to match the photo subjects to my video library. Since I was shooting video on vacations while my wife was shooting stills, often I could date the photos this way because my video was annotated in the audio track with dates. Other methods are to find a nearby photo WITH a date and estimating ones that came before and after. As a last resort, one can just guess based on the apparent ages of people in the photo in question or nearby photos.
Date sorting is important to lessen the work load while post processing the photos. All the photos of a certain date can be dated as a batch. This significantly reduces the piece work involved.
Select all of the photos on a given date and scan them. The scanning software that comes with the HP scanners allows a scanned image to be opened in another application, I use GraphicConverter. Your software may be different so you'll have to adapt this step to your tools. After a scanning session, I end up with a bunch of GraphicConverter windows open. I then rotate or modify each image as needed and Save As... with a name that represents some pertinent information FOR THAT INDIVIDUAL PHOTO. I would typically just list the names of the people in the photo, the overall subject of the group of photos is added later in batch form. Once each photo is saved to a working folder, I drag that folder to GraphicConverter to do the batch redate and rename as described above.
Once the renaming and redating is done, the batch is dragged from the working folder into a subfolder of a "pending import" folder. This subfolder is named for the subject matter and perhaps date of the group of photos. The working folder is then cleaned out of any working copies so that it is ready to accept a new set of scans.
At the end of a scanning session, I test the photos with a trial import to iPhoto as described above and then do the final iPhoto import. The contents of the "pending import" folder is then dragged into a hierarchically structured archive folder. When this folder grows to CD size, I burn it, lock it and start another archive folder.
[ Previous Tip ] [ Mac Tips ] [ Next Tip ]
This page has been accessed times since Aug 14, 2004
© 2004-2005 George Schreyer
Created 14 Aug 2004
Last Updated August 20, 2005