My 06 has the exact same problem -- passenger seat does not heat or cool. When I press either the heat or cool buttons, the light stays lit for about 5 seconds, then goes out. I think its a controller issue, and not the crushed ductwork (see below), so I'll be taking it in.
In some cars, notably the 2003 and early 2004 models, the ductwork in the seat would get crushed and prevent proper airflow to the seat ventilation system. When in heat mode, this would result in an overtemp error (a safety feature) and the system would shut off after about 3-4 minutes. There is a TSB to fix this, and I've sent it to many people (some of whom thank me and some of whom do not -- which pisses me off, but that's another story).
The problem 2005 LS Ultimate and I have seems to be different. I'm no mechanic, but the fact that the lights go off so quickly suggest to me its an electrical issue, instead of a over-temperature issue. Indeed, under these circumstances, the system is not on long enough to generate any significant heating or cooling.
2005, if you take your car in, please report back what the diagnosis and fix were. I'll do the same.