SpaceX’s Merlin Engine Malfunctioned Due To A Tiny Vent Hole Being Blocked
Officials of the National Aeronautics and Space Administration (NASA) and Space Exploration Technologies Corp. (SpaceX) provided crucial updates about NASA's first operational crewed mission (Crew-1) to the International Space Station (ISS) set to take place on November 14th during a press conference today. NASA and SpaceX were forced to stand down from the Crew-1 launch that was initially set to take place on Halloween due to problems with SpaceX's Falcon 9 launch vehicle's Merlin 1D engine that the company initially discovered when it was seconds away from launching the United States Space Force's GPS III satellite.
At this front, SpaceX's vice president for mission assurance Mr. Hans Koenigsmann provided more details for the Merlin 1D's faulty component responsible for the Crew-1 delay. According to the executive, the problem surfaced due to the cleaning process that SpaceX undertakes during engine construction with some of the cleaning material causing a valve in the engine to malfunction.
SpaceX Successfully Troubleshoots and Replicates Falcon 9 Engine Problem Before NASA Astronaut Launch
The component of the Merlin 1D that caused SpaceX to stand down from the GPS and Crew-1 launch is referred to as a gas generator. It's responsible for the engine's initial startup and is used to build sufficient pressure to push the Falcon 9's propellant into the Merlin 1D's main combustion chamber where the propellant is ignited to generate thrust that is responsible for liftoff.
During its troubleshooting process, SpaceX discovered that a relief valve within the gas generator started to open earlier and attempt to start the engines earlier than they are intended to during the launch process.
The problem, its troubleshooting process and SpaceX's initial fix were described by Mr. Koenigsmann in detail today. According to him:
There’s lots of activity here both on the Hawthorne side and also on the Cape side getting the dragon and the Falcon 9 vehicles ready for the upcoming Crew-1 launch. We did have an issue earlier this month, we stood down from a Falcon 9 launch attempt with the GPS III-IV spacecraft. We had an auto abort during the engine ignition basically in the last, I think it was three seconds or something like that. And it was caused by an early start behavior on two of the engines, in this case it was Engine 1 and Engine 2. And it was a good abort, it did exactly what we programmed it to do.
When we looked t the data we saw that two of the engines attempted to start early and the auto-abort prevented that and by doing that it prevented a possible hard start that could have been damaging to the engine hardware. The team then went right into inspecting the engines on the pad. Didn’t see anything, everything was configured right. Tubing was correct, nothing really obvious. So we pulled the engines, or we removed them rather and sent them to the test facility in McGregor in Texas for additional testing. And this is one of the rare cases where we were actually able to replicate it in Texas and was great for troubleshooting. It did the same thing on the test stand. We performed additional inspection then on the engine and then we found a relief valve like a little line that goes to the relief valve blocked in the gas generator.
The gas generator is basically like a little rocket engine that powers the turn wheel that then powers the pump which feeds the propellant into the main chamber. And it’s obviously a very important part. That little red substance was blocking a relief valve that caused it to function a little bit earlier than it was supposed to do. We found that the substance was a masking lacquer left over basically. It appeared to be left over from the build process where we mask certain parts during surface treatment, and probably during the washing and cleaning process some of that masking lacquer went into this vent hole and blocked it. We could actually show that when we removed that masking lacquer from the vent hole that the engine performed perfectly normal and started up at the right timing as it’s supposed to do.
So it was a really great find in that sense and allowed us to fix something that is very subtle but can have obviously some negative impact on engine operation. We then went and looked at all the engine startup signatures across the fleet and we found similar curves or similar early tendencies on the Crew-1 booster from two engines. Like I said earlier we had two on GPS already, and then we found another one on Sentinel which is the upcoming NASA launch from Vandenberg.
Problem Caused By Outside Vendor Reveals SpaceX Executive
In response to a question by Eric Berger of Ars Technica, the executive also revealed that the problem was due to an outside vendor. When asked why SpaceX did not encounter the problem before, Mr. Koenigsmann, who stated earlier in the conference that the problem did not surface with older Merlin 1D engines, replied:
I would say that there is certainly a possibility that we have had cases cases of that earlier and they were basically so harmless that we completely missed them. It's possible. It's also possible that you know little things changed. This is a process that's done out of house. And in a special...you know vendor, and so it could be that person is now more generous with cleaning fluids or anything. I'm really guessing here it's a little bit hard to figure this out. We've been there actually, we've talked to the people, we've made them aware of that and we're pretty sure it will not happen anymore from that perspective.
But in hindsight it's probably, it's difficult to explain how this works for so many years and then suddenly you see this coming out later. The important part I think for us is that we caught it before anything happened. We also had the right abort implemented. That's really actually is the other thing that I think is really great. We expect the engine to go through certain profile to measure the pressure and when it doesn't do that it stops and then we look at what it is and in this case it stopped beautifully. It stopped and then gave the chance to look at the engine and figure out there is something wrong. So yeah interesting how this happened in the end but really important for us we fixed it and tell the vendor what happened there and make sure that it never happens again.
The vendor in question is responsible for anodizing the engine components and the protective lacquer ensures that components that are not intended to be anodized remain this way. Anodizing is done to improve the durability of the components and it coats them with additional protective layers through an electrochemical process. The vent hole described by Koenigsmann is one-sixteenth of an inch, and the company has multiple vendors for its anodizing process.
To detect the problem, SpaceX conducted a computerized tomography (CT) scan of the Merlin 1D and determined that a non-metallic material was at the heart of the problem. The company is confident that it has determined the root cause of the problem, following which missions for NASA and the Space Force will comfortably proceed after NASA's upcoming weather satellite launch. NASA's program manager for the Commercial Crew Program (CCP) also stated the agency would prefer to have the GPS mission fly first to evaluate a change in the engine of the first stage of the Crew-1 launch vehicle.