Dr. Jerry Pournelle

Email Me

Why not subscribe now?

Chaos Manor Subscribe Now

Useful Link(s)...


Hosting by

Powered by Apache

Computing At Chaos Manor:
April 30, 2008

The User's Column, April, 2008
Column 333, Part 3
Jerry Pournelle jerryp@jerrypournelle.com
Copyright 2008 Jerry E. Pournelle, Ph.D.

This has been long delayed. I explained the situation — a diagnosed brain tumor, and radiation therapy — in the first column this month (link). Since then the radiation therapy has ended and I've been recuperating. That has gone a lot more slowly than I thought (but not more slowly than predicted by my physicians). The symptoms have been days when I could do little more than go to bed. The result has been extended delays in adding to the weekly column segments. Since Computing at Chaos Manor has always been a monthly column, and I got something up this month — actually about as many words as we used to have in the old monthly BYTE column when it was a printed magazine — I can and will claim not to have missed a column deadline since I began in 1980; but even so my apologies for the delays here.

Outlook Follies

At the moment I live in Outlook. I may change that in future, but the combination of Outlook 2003 and Microsoft Desktop Search have been highly effective. The search engine does a great job of indexing files, including Outlook .pst files, and the result is that I can find emails up to several years old with a few minutes of search (and recent ones much faster). All I need are some reasonable clues as to what to search for — it's easy if I know who sent the email, or what the subject was, but I don't even have to remember that much. Search usually returns far more items than I care for, and I have to scan the results, but what I want is always in there. In short, it works.

This is just as well, since lately my short term memory mostly consists of what I have written in my log book (and remember to read).

I use Outlook for nearly everything. Not only does it manage my email, sorting the mail into categories (folders) according to fairly complicated rules so that I can read and answer the most important mail first, but it also keeps my subscription list with a complicated series of categories of subscriber (by year, category of subscription, etc.), my contacts including mailing lists (it's easy to build mailing lists in Outlook; caution, if there are more than about a thousand names in the list, Outlook doesn't like that; better to chop that into lists of under 1,000 or use a proper mailing list program), and other such items. All this is properly indexed by Microsoft Desktop Search, and it's easy to find any mail item using Search.

When I do my mail columns (both here and at www.jerrypournelle.com) it's fairly easy to do that since the mail is nicely sorted, and if I've lost an item, Search will find it. In a word, Outlook 2003 and Microsoft Desktop Search have done very well for me.

The problem is that every now and then Outlook 2003 goes wonky. I will be answering mail, or merely moving mail from one folder to another, and I will get a mysterious message: "Another application closed unexpectedly, and for your convenience you can no longer access any Outlook folders. We're not going to tell you why, or what really happened. Have a nice day and thanks for all the fish."

All right, it doesn't really say that, but that's the information it gives. Most importantly, it doesn't even hint as to what the conflicting application was. The point is that once I get this message, Outlook is useless.

Other times, Outlook simply refuses to send messages. They stack up in the outbox, and sit there forever; nothing I can do will get Outlook to send them.

One of these conditions happens at least once a week. Sometimes it's far more often. Calling it annoying is a vast understatement.

The remedy in both cases is to close Outlook, then use Task Manager to close any process named "Outlook", then open Outlook again. Four times out of five this will do the job. Outlook comes back up, all the messages in the outbox are sent, and everything works as I expect it to. Alas, sometimes that isn't good enough. Outlook remains useless. When that happens the only remedy is to close down and restart the system. And once in a blue moon even that isn't good enough: when that happens I copy all the .pst files to another computer (usually the IBM Lenovo ThinkPad t42) that also runs Outlook 2003. Then I start Outlook on that machine. It always runs just fine. I close Outlook, copy the .pst files back to Alexis my main communications machine. Then for good measure I use VOPT (link) to defrag Alexis. When I start Outlook on Alexis after that everything is fine. That always works.

Robert Bruce Thompson is travelling today, otherwise at this point he would joyfully point out that things like this don't happen with sane programs, and how it would drive Aunt Minnie nearly insane. I agree; but in fairness, Aunt Minnie doesn't get a thousand emails a day and have to deal with well over a hundred while keeping the ability easily to find them again; nor does she cut and paste dozens of emails to make up a commentary log.

In any event, it is needless to say this can get tedious. It doesn't happen often enough that I have been willing to change systems for mail handling; that would affect enough of my work to be a major change, and this is not the time to make major changes in what I do. Still, I want to do something about this, but I don't know what, so I have been experimenting.

One thing we know about Outlook is that it is far happier when the .pst files are well under a gigabyte. That's not easy when you get as much mail as I do. I have created a number of archive .pst files over the years, and they haven't been very logically organized; so one thing I have been doing is rationalizing those. A second thing I have done is export all the very old — 2005 and earlier — items into .pst files that will not be opened by Outlook at all.

Outlook has an archiving function, and that works; but keeping the primary .pst files small requires that I archive frequently, and the archive files must remain open because they contain fairly recent items. My new Deep Archive Files are a year and more old. Outlook won't open them. They'll just sit there as a deep archive, indexed by Search but hopefully only indexed once and never again. Active archive files may be indexed every day — and I have some suspicion that Search may be one of the applications that accesses Outlook and may be a cause of that mysterious message. More on this later.

Travel and Backup

One problem I always have is what to do about mail when I go on trips. My solution has been to use xcopy to copy all the outlook.pst files from Alexis to whatever laptop — usually Orlando, the IBM ThinkPad T42p, but sometimes LisaBetta the HP Compaq TabletPC — I will take with me. I also copy all the FrontPage files. That way I can keep my web site up (both View and Mail) and I have all the mail files and web pages I need. While I am on the road I get mail and deal with it as I would at home, and when I get back I use xcopy to copy all those files back to Alexis. Typically the file transfer takes about ten minutes each way.

This has the advantage of making backup copies of just about every critical file on Alexis, and thus serves as an addition to the usual backup schemes.

The problem with this system is that it takes a long time to copy all the pst files — and while the xcopy command I use is xcopy source destination /e/s/d/y, which causes it to copy all but only files with a later date, all the pst files that Outlook opens will have a later date, even if nothing was written to them. For some reason I didn't understand that when I devised the system. Of course the files were much smaller then.

I also didn't pay a lot of attention to the organization of the older .pst files, and some of them had gotten very large and had a ridiculously complex sub-folder structure.


The obvious remedy was to rationalize the archive files, and pay attention to which files Outlook should open. There was no point in keeping items from 2005 and earlier in files that Outlook would open. They needed to be put off in .pst files that would not usually be opened, but which would be indexed by Microsoft Desktop Search so that if I did need something in there I could find it.

It looked to be a formidable task, since I had a crazy arrangement of subfolders. The first thing was to get all the files over a year old into a single file for each category. Categories include "Sent Items"; "Posted"; and folders of conferences that I take an active part in. There are also "source" folders, that include such items as Richard Doherty's Envisioneering mailings. That took a couple of hours. Then I used Import and Export to send them off into .pst files that will not normally be opened at all. That turns out to be easy but tedious. The only tricky part is that once you give Import and Export a file name (Oldsentitems.pst) and click "Finish" you must also give that filename again: the one you have already given tells the OS about it, but the next one tells Outlook, and if you don't do it right you'll end up with a bunch of Outlook files called "Archive Folders" and no way to tell them apart.

The machine will trundle for a long time, but eventually it gets the job done. Now you have to delete the items you have exported, and compact the original file. (File / Data File Management / Settings / Compact Now ) At this point you have made the file that Outlook opens considerably smaller, and created a new file (Oldsentitems.pst) that contains the older archived items and which Outlook will ignore.

This turns out to make Outlook work much better: the smaller the files it has to deal with the happier it is.

Whether this lengthy rationalization cures the "another application unexpectedly closed" problem I can't say. I came up with another remedy for that: I now run Outlook in "Work Offline" mode, and periodically tell it to "Send and Receive". It no longer automatically goes out and gets new mail, but it doesn't seem to lock up as often as it used to, either.

The moral of this story is that Outlook doesn't work for years without some attention to how it is organized; but then I suspect there aren't many programs that do. Pay attention to your familiar old tools. You'll be glad of it.

Outlook 2007

I have not converted to Outlook 2007. I have some recommendations both pro and con, but none of that prevented me from moving to Outlook 2007. Microsoft did.

They did it in two ways. First, while for years I had both Office 2003 and Office 2007 on Roxanne, the Core 2 Duo Vista machine, early this year that ended: one of the "updates" from Microsoft made it impossible to have both brands of Office on the same machine. Neither would work properly. That wasn't confined to Vista systems, either. The same "update" worked on XP systems. Microsoft Technical Support pretended not to know anything about it; and if there is a workaround to allow you to keep both Office 2003 and Office 2007 one the same machine, I don't know it.

Eventually I deleted both Office 2003 and Office 2007 from Roxanne (In Vista Control Panel no longer has the item "Install and Delete Programs"; for inexplicable reasons Microsoft decided to rename that "Programs and Features", and assumes you will figure that out for yourself. Help, as usual, is no help at all.). I then reinstalled Office 2007 and tried to get that working.

In Windows XP, Outlook.pst is in C:\Documents and Settings \ Jerryp (or whatever user name you use) \ Local Settings \ Application Data \ Microsoft \ Outlook. This makes it tedious if you are using a command window to copy files, but once you have built a batch file you need only change the drive letters: the file will be in the same place on every Windows XP machine (provided that you have an account for that user on each machine.)

Back when you could keep Office 2003 and Office 2007 on the same machine, Office 2007 on a Vista Machine kept those files in the same place (Documents and Settings, etc.), and if you copied Outlook.pst from an XP machine to a Vista system, then when you open Outlook 2007 it would find the pst file and use that. I know it once did, because I had Outlook 2007 working on my Vista machine, and I was able to copy that pst file back to my communications system. It worked.

No longer. I am not sure when it happened or even what happened, but now Documents and Settings is empty. The outlook.pst files are in C:\Users\Jerryp\APPDATA\Local\Microsoft\outlook. Note that it's APPDATA; Application Data is empty.

I can guess when Vista decided to change my file names and structure without telling me; it was probably when I uninstalled Office 2003 and reinstalled Office 2007. The point is that the change was needless. In fact this is nonsense on stilts. If Vista can do that to me, what else might it do?

The result is that when I build Roxillanna, the Core 2 Quad 6600 machine, it will get Windows XP; I no longer trust Vista. Any operating system that will change my file structures and move my files so that I cannot find them — it took me half an hour to find the outlook.pst file — is not an operating system I am prepared to trust.

I often wonder if Microsoft has lost its mind. If it has as little competence at selling advertising and search placement as it seems to have in operating system structure and interface the company may not be with us much longer.

Comments and Considerations

Eric Pobirs comments:

You say that Vista changed file names and structures but I only see indication of structural change. As I've mentioned many times before, Outlook is not designed for your situation, which is atypical for a consumer and extremely atypical for the corporate seats that are the primary target. There are tools out there that would let you avoid all of this DOS-esque file manipulation but if you'd rather fight the gators than drain the swamp, that is your choice.

As for the Application Data vs. APPDATA, I'd chalk that up to regime change at the Product Manager level. Every generation of Office and Windows has seen changes like this, as have many non-Microsoft products over time and revisions. Whether it is whim or has a practical reason I cannot say for sure. Vista does introduce a number of changes intended to promote better practices by developers. For instance, the separation of application files into those that are permanent (barring patches) and those that are subject to change, such as settings files that are never intended to be directly touched by users. This makes it simpler to write apps that run without requiring excessive rights for users, which in turn makes life easier for IT folk.

Eric may well be right, but my annoyance at these changes without explanation and without reasonable Help files remains.

Even more interesting are Marty Winston's comments. Winston says that resistance is futile, and my real solution is simple to go to Office 2007 and have done with it:

FWIW, those Outlook leprosy problems seem to go away in the 2007 version. It's usually caused by a plug-in or add-on (especially when it isn't updated to play nice with a more recent Windows or Outlook update).

The size limit you're dealing with is a remnant of earlier versions; when you install 2007 over what you have, it will offer to upgrade-convert your data files - say yes. My current PST file is 3.2GB and gives me no problems.

He also has a new scheme for using Outlook both at home and on the road, and since he spends far more time on the road that I do, it's very likely superior to mine. I will try that and report in another column. I suspect I will just have to bite the bullet and go over to Office 2007, provided I don't find a better Apple solution to the mail problem.

Winston has one more Office/Outlook tip that's worth your consideration:

By the way, I groom my PST files early each July, moving messages from the second year previous into its own PST file. SO this July, I'll create a 2006.PST and move things from that year into it. As you noted, it all still gets indexed.

And I should add that I use a non-default location for my PST files, away from the Windows user directory "nest" that can get blown away if you ever have to reinstall Windows.

That latter seems like a very good idea indeed.

Converting To the Mac

My physical problems prevented me from doing much in pursuit of my efforts to convert to the Mac from Windows. The Outlook problems took up the time and energy I have. On the other hand, my experience with Vista changing my file names and structures without bothering to tell me, coupled with Microsoft's continued announcements that they will, too, kill off XP and how dare we mere customers insist they not do it, have made moving to the Mac OS a matter of more urgency.

I have a great deal of Mac equipment now, including Mac TV and a Mac Book Pro. I've been enough under the weather that I have done little with them. The Mac TV is going to require stringing a gigabit Ethernet line to the TV room, and that will take a few days to a couple of weeks.

I'll get to all of them. I find the MAC OS to have some vexing features but over all it works; and it doesn't spring a bunch of surprises on me like changing my file names.

Winding Down

The book of the month is Culture and Conflict in the Middle East, by Philip Carl Salzman. This is an important book for anyone who wants to understand Middle East culture and what we must do if we intend to stay in Iraq. Readers will recall that I was very much opposed to both US invasions of Iraq, and that I predicted that the second invasion would be far more expensive than was projected. I also said that we would have a lot of problems bringing peace because we did not understand Iraqi and Arab culture.

Alas I was correct.

One of our problems is that I doubt one person involved in setting up the original US administrative and command structures in the first days of Iraq had read any work comparable to Salzman's book, or indeed much about Arab culture at all. Certainly Bremer had no familiarity with the peoples he was sent to rule over as pro-consul.

Salzman describes the tribal structure of Arab society and why it is important to understand that. Since we now appear to have a long term commitment to Iraq — one cannot conceive of taking into the United States the flood of refugees that would result from US withdrawal — the sooner our Legions and their officers gain an understanding of the culture of the people we have chosen to liberate, the fewer casualties we will suffer. Salzman's book is not easy reading, but then important books seldom are.

The movie of the month is the romantic farce Miss Pettigrew Lives For The Day, with Oscar winning Frances McDormand in the title role, and Amy Adams as the flighty, flirty, Delysia Lafosse. The ending is predictable from fairly early in the picture, but then this is a romantic farce. Don't analyze this picture. Just sit back and enjoy it. There's not a serious moment in it, and that's precisely the movie I needed to see...

The computer books of the month are the Missing Manuals, and while I have recommended them before, they are indeed the vital books I needed for the month's work:

iPhoto '08 The Missing Manual, which will tell you how to use the iPhoto program that comes with your Mac; and iPhone The Missing Manual, which does the same for your iPhone.

If you are a Windows user considering converting to a Mac, you will need Switching To The Mac, Leopard Edition, The Missing Manual and Mac OS X Leopard, The Missing Manual.

And one final book: if you are converting to Office 2007 you will desperately need the Missing Manual for that. Trying to go from Office XP or Office 2003 to Office 2007 without that book will doom you to regions indistinguishable from some of the circles of the Inferno. ( Incidentally, Tor is reissuing Inferno by Larry Niven and Jerry Pournelle, and you can pre-order copies now).

All the Missing Manual books are from O'Reilly, and their quality is uniformly high.

I have a large stack of new books awaiting review, and I will have a special book review section next month.