Keeping up with the problems on ISS is difficult; the situation is complex.
Apparently, when the new solar panels brought up by Atlantis were installed, a power supply box failed. This provided power to the six computers on board used to maintain the station’s orientation, which in turn is critical to keeping the solar panels aligned with the Sun. The leading hypothesis is that electrical noise generated in the solar panel wiring is affecting the electronics. Several attempts to correct the problem have not worked, though of course the spokesfolks for NASA are saying this is not a serious problem yet. That may very well be correct, but this situation is very much cause for concern. The Russians may move up a launch to get a new power supply sent up in July.
An interesting line from an MSNBC news article caught my eye:
The [electronic intereference] spike apparently knocked out an electronics box that provides power to all six of the German-built control computers.
Which makes me ask: why would there be a critical single point of failure aboard the space station? This to me is the bigger story here. Once this problem is fixed — and it probably will be — I wonder if the media will follow up with this?
As usual, James Oberg has more info on the situation.
June 15th, 2007 at 9:38 am
That does seem a bit of a design flaw. Possibly it’s one of those ‘when the ISS is finished…’ things – the redundant systems may not be there yet (like the shuttle’s ‘jumper cables’ that would have been quite useful in this situation).
Another issue Oberg raises is that it seems the NASA network can’t carry the telemetry needed so they can only work on the problem when the ISS is over Russia! It seems that there’s a whole catalogue of things that have contributed to the problem, most of which sound avoidable (although hindsight is 20-20, of course).
June 15th, 2007 at 9:58 am
Gotta watch those power spikes, though I wonder why their(Russian) electronics are so dated that the computers aren’t hardened against such problems. The (now) standard switching power supply in my computer keeps it from injury against such spikes and it usually remains up even in the thunderstorms common in Georgia. I’m surprised the ISS comps. can’t do the same, unless the solar panel is producing some really hugh spikes???
Gary 7
June 15th, 2007 at 10:28 am
This was the first thought that went through my mind as well Phil. My guess as to why is the same as Rob’s: they simply haven’t installed the backup yet. This *has* to be the answer, because I can’t see them not having a backup for a critical system like that. Though, I find it strange that all six computers are wired into the same electronics box. Backup or no, that is odd. Unless all six computers need to be up and running for the system to work? Then it makes sense. In that case if one fails the whole system fails anyway, so there is no need for more than one supply (beyond the backup).
June 15th, 2007 at 10:28 am
“This to me is the bigger story here. Once this problem is fixed — and it probably will be — I wonder if the media will follow up with this?”
I’m not even sure if our media caught the problem you did, even though it’s in their own story. Seems like the media likes to say “thing broke, that sucked, inothernews here’s a dog that can skateboard.”
June 15th, 2007 at 10:46 am
I believe this is a case of the system was *already* in it’s backup mode – repairs simply have yet to be made to the primary system.
Gary, the reason that you have a power supply that isn’t capable of dealing with those spikes on it’s own is simple. They use computer technology several generations back for flight hardware. If I’m not mistaken they’ve only recently started using the original Pentium for flight systems in space. They do this so that the behaviour of the equipment is well and completely understood and also has had time to mature. They also do this becasue you have to harden these systems agaianst electromagnetic threats that do not exist on earth – hard raditaion and various fairly common high energy particles do not so fun things to computer electronics. That hardening process is a long and arduos one. Luckily we are protected from most of those issues down here on the surface – up there things a much iffier for complicated and sensitive electronic systems. They do take modern laptops with them but they can be notoriously flakey while in orbit and are not used for life critical systems. Switchmode PSU’s, while handy for smoothing power, can be very finicky devices… this is not desireable in space.
Of course, neither is having your attitude control system go down when you increase your power supply. Seems to me that it really isn’t that big of a deal though. Worst case : they unplug the new solar panels until they can get the needed hardware to regulate the power problems. It’s inconvenient – that extra power is needed after all – but doable.
June 15th, 2007 at 10:46 am
Hmm. That article say that when the shuttle undocks, it torques the station enough (I love verbing words btw) that the gyros can’t handle it, and the thrusters are then needed.
If they can’t get this problem fixed before the shuttle needs to depart, I wonder if the shuttle can use its own thrusters to spin the station in the opposite way, with almost the same strength as the torque they will produce when they undock. That way, they can wait until the station spins around to almost the correct orientation, and then kick off.
They will spin the station almost to a halt, and then the gyros can be brought back on line (I assume they’d have to take them offline in order to get the station spinning, else the gyros would fire up and try to stop it), and they can gently slow the station down the rest of the way, stopping it right where it should be.
June 15th, 2007 at 10:52 am
Even on an international flight there are a sufficient number of cosmic ray strikes on your laptop to make it likely that it will lock up every 4 hours or so. I don’t remember exactly, but it has something to do with the accumulation of random errors in the memory, or on the CPU, or something. Yeah. Me write good explanation:P.
June 15th, 2007 at 11:15 am
Huh. No backup? This sounds very risky and… well…stupid to me.
June 15th, 2007 at 11:39 am
I’m not even sure if our media caught the problem you did, even though it’s in their own story. Seems like the media likes to say “thing broke, that sucked, inothernews here’s a dog that can skateboard.â€
I have to completely disagree, kingnor. This is the media we’re talking about. They’d never do a story as interesting as a skateboarding dog!
More like, …back to Paris-Lohan-Speares coverage.
June 15th, 2007 at 12:43 pm
“If I’m not mistaken they’ve only recently started using the original Pentium for flight systems in space.”
Let’s just hope they won’t do any division with it…
June 15th, 2007 at 12:54 pm
Single point of failure? Sounds like the system was designed by Star Fleet. I wonder if the computers sprayed sparks in the air when the problem started, and the main lights went out and were replaced by ominous red ones?
June 15th, 2007 at 2:28 pm
Let’s just hope the secondary subprocesser doesn’t get hit by Kason weapons fire 0_0. Then the self destruct will go offline, and the station will be completely screwed.
June 15th, 2007 at 7:17 pm
A comment on the ISS from Bob Park, the man who doesn’t know what he’s talking about.
2. CONSTRUCTION: WHY NOT JUST DECLARE THE ISS “FINISHED?”
Today was supposed to be the day astronauts would stitch up a rip in a thermal blanket that tore on launch. Atlantis is docked at the ISS on a 13-day construction mission to install new segments of solar panels to enhance the energy supply in preparation for Europe’s Columbus module which is supposed to join the ISS later this year. Plans changed when three Russian computers crashed. The computers maintain orientation of the ISS and control oxygen levels. The Russians think electrical noise from the new solar panels is to blame. They did what you and I would do, they rebooted, but the computers re-crashed. We all have days like this with our computers. In space it leads to scary talk about abandoning the ISS
June 16th, 2007 at 12:46 am
Well the computers are now back online, everything is A-OK now.
As far as systems engineering goes, the computer systems onboard are notoriously “obsolete” by computer geek standards.
But what most people do not realize is that the computer hardware onboard spacecraft work in an environment that has a higher flux of ionizing radiation. Modern computer chips are not engineered for use in these environments and would require expensive design and testing in order to work reliably.
As well informed as Jim Oberg is, another science writer had more detailed information on the crisis as is was happening. Alan Boyles Cosmic Blog on MSNBC has more information on the computer hardware and software:
http://cosmiclog.msnbc.msn.com/archive/2007/06/14/226502.aspx
Matthew Ota
June 16th, 2007 at 6:00 am
In any system as complex as the ISS, regardless of how much redundancy has been built in, there will always be single points of failure. (Heck, even the human body has them; viz. the Vulcan neck pinch.) You can only try to catch them before they fail, and figure out workarounds. Going by some consequential NASA examples (oxygen-rich crew cabins, exploding stirrers, cold O-rings, imperial units, foam abrasion), this one is seems rather benign.
June 16th, 2007 at 6:36 am
The human body has many single points of failure. But it was designed by an idiot. Or random chance. Take your pick.
June 17th, 2007 at 7:23 pm
[...] to compare launching a rocket into orbit versus building a complex space station, but… the latest fiasco aboard the ISS does bring this all into stark [...]