Home / News

News Archive

Texel crash, Pixel back-to-back 180, Module 105

September 3, 2007 notes:

September 4, 2007 notes:


Texel Crash


(note that this is slightly edited from the original posting to aRocket with a few more bits of information)

August 18 was a bad weekend for Armadillo.  We set out to put some flights on Texel, the backup Quad vehicle, and it didn't go so well. 

We started out with a normal 90 second elevated / tethered hover test, but we ran into a problem with the actuator power.  We initially thought it was a bad main power switch, but it turned out to be the lithium-polymer battery pack cutoff circuit incorrectly shutting down at 16 amps of load instead of 40.  This was a new battery pack ( www.batteryspace.com HPL-8059156-4S-WR), and it had passed all the individual actuator checks, but when the igniter started firing with both high amp NOS solenoids, the battery shut down (went to 0.3 volts indicated) after one second and stayed there until it was physically disconnected.  Russ made a fairly heroic field repair, cutting open the battery pack and wiring around the protection circuit while sitting on top of the rocket.  The total time spent on this after three attempts was 90 minutes, and enough lox had boiled off that the vehicle hit lox depletion at 60 seconds of flight.  We got a few good data points from this: the batteries need to be checked at full current load, with vents open we boil off about two pounds of lox a minute, and lox-depletion runs are benign, if a little flamey.

For the second flight we were going to do a ground liftoff (still tethered for runaway protection) to test the automatic ground contact engine shutoff code.  We have had several reasons to want to automate this: We get a fair bit of bounce on touchdown, because the engine is essentially keeping the vehicle weightless during the terminal descent.  A computer controlled shutdown would be at least a half second faster than my manual punching of the shutdown when I visually see ground contact.  The quads will just safely bounce around on the ground a bit if the engine just goes to idle and doesn't shut down, but the module, with the gimbal below the CG, will try to tip itself over when a landing leg becomes a pivot point, so there is extra incentive to get it shut off fast.  You can see that in our XPC '05 vehicle flight.  We also need to handle the case of the vehicle landing in a situation where I can't shut the engine off promptly, either because there was a telemetry problem, or when we are doing high altitude flights, it lands out of direct sight.  There is a separate shutdownTime parameter that will keep it from sitting there at idle for ten minutes, but a telemetry abort could still have it on the ground and cooking for the better part of 220 seconds.  We could still shut the flight safety fuel valve, which would result in just idle level lox pouring out of the engine, but that has its own problems.

I have been very hesitant to put in ground contact shutoff code, because shutting the engine down for some incorrect reason would be catastrophic, and I would feel awful if that ever happened.  We had some switch based ground contact sensors on the old VDR, but they never got tested.  We have concluded that the landing jolt, as seen by the IMU accelerometers, is a good enough ground contact signal.  There is always the worry that combustion instability, or a nozzle ejection event, might trigger the signal level, so there are additional guards about it only functioning when you are within three meters of the ground (we must leave some slop for uneven terrain or GPS innacuracy) and trying to descend.

We loaded up again, being very thankful that we now pack three six-packs of helium for each test trip after we were forced to cancel the second flight on a previous test session due to insufficient helium after troubleshooting a problem forced a repressurization on the first flight.  Liftoff and hover was fine, and at the 45 second mark (no sense pushing it on a ground liftoff), I had it come in for a landing.  It hit the ground, and I saw it bounce back up.  My first thought was "That didn't seem to help at all".  My second thought was "Uh, that looks like it is accelerating upwards, not bouncing."  My third thought was "How the heck did the ground contact code cause that?"  My fourth thought was "Crap, its going to fly into the crane, I need to kill it".

After I terminated thrust, the vehicle coasted to an apogee of about 20 feet, and fell to the concrete.  It made a fireball that would make any Hollywood movie proud, but the vehicle didn't launch itself back off the ground, make an earth-shattering kaboom, or throw any shrapnel.  The fire truck moved into range of the crash and hosed down the vehicle until the fire was extinguished.  Surprisingly, the flight computer was continuing to operate and transmit through all of this, but the sensor and wiring harnesses were shorted out in the fire, so we didn't have any sense of the state of the tanks.  With trepidation, someone approached the vehicle and found that the lox tanks were still full and pressurized at the same pressure as when the vehicle was shut down.  The Aspen Aerogel insulation on the tanks had prevented them from even warming up.  We vented the lox, and started assessing everything.

It didn't take long to find out exactly what had happened.

On touchdown, the ground contact logic failed to activate at all.  The IMU in Pixel is an older model Crossbow that was rated for +/-10 Gs, but reads to +/-14 Gs.  That particular model was discontinued, and the newer IMU in Texel was only rated for +/-4 Gs.  I had set the ground contact trigger value to 5 Gs, which I had some recollection that the IMU read to, but it turns out that it was maxed out at 4.5Gs.

What caused the upwards flight was a GPS issue.  On ground contact, the GPS PDOP value went from our normal 200 or so up to a value of 1200.  The very next frame, it went to 0.  We ignore GPS updates when the PDOP is 0, and also some other cases where we know the data is bad, but after flight starts the rule has been that any valid GPS update is taken as authoritative for velocity.  Between GPS updates, and if the GPS goes invalid, the IMU will use dead-reckoning to coast for a while, but the accelerometers in particular aren't accurate enough to do this for very long.  The at-impact 1200 PDOP update contained velocity values that were significantly off, including a 5 m/s down velocity, which caused the vehicle to throttle up to try to regain the desired 1.5 m/s terminal landing velocity.

I briefly wondered if the GPS antenna mast might have actually fallen off on landing, giving a correct velocity before losing sat view, but reviewing the video showed that it clearly stayed in place through the bounce.

We have known these GPS receivers are vibration sensitive for a while, and we take several measures to protect them from the rocket environment, but this was the first time we had a shock related failure.  I went back to the telemetry from the Oklahoma free flights to look for similar signs on touchdown, and found a corroborating point.  On the second free-flight, the GPS PDOP jumped from 200 to 359 at the time of touchdown, as seen by the accelerometers.  The first ground touchdown of the module also showed a jump in PDOP at touchdown. This effect has evidently always been there, and some combination of the still heavy propellant tanks from the shortened flight and just bad luck caused a jump all the way to 1200 PDOP and an unusable value.

The question of interpreting GPS PDOP values has always been an issue for us.  The exact meaning has to do with uncertainties in the calculated GPS position value due to the angles between the currently tracked sats.  Flights typically have a PDOP between 150 and 250.  We have a no-go set at PDOP 300, and an in-flight abort set at PDOP 400.  While the calculated GPS position can vary widely with higher PDOPs, the velocity value always seemed to stay fairly reasonable at high PDOP values, so I had intentionally elected to continue using velocity updates even if the PDOP was past the abort point.  The situation I was worried about was having the vehicle at 60 meters altitude, and having the PDOP change to 450, forcing both an abort, and, if I couldn't use that velocity data any more, a reversion to coasting on the IMU updates all the way to the ground, which I wasn't very confident of.  I'm not beating myself up about this original decision.

The change I have made as a result of this is to define an unacceptable PDOP value, initially set at 500, beyond which a GPS update will simply be ignored as if the PDOP were 0.  If we are in a situation of degrading PDOP somehow, that should allow it to abort at 400 and still hopefully make it to the ground with GPS based velocity updates.

In hindsight, there are a couple other, more reasonable-to-expect, things that would have saved the day:

If we had tested the ground contact logic by actually dropping Texel at the shop, we would have found that it didn't trigger, and I would have adjusted the value until it did work.  If that had been done, we might not have even noticed the GPS problem, because it would have shut down before any action based on the bad velocities was taken.

If I had planned on just shutting down the flight like I normally do, instead of just watching what the vehicle did with just the ground contact logic, it would have just looked like a higher-than-normal bounce, and we would have seen both the ground contact failure and the GPS issue when I looked at that part of the telemetry.  If the ground contact logic had functioned, my pressing the button shortly after would have had no effect.

There were a number of data points learned from the experience:

The vehicle fell straight down after the failure, as it was supposed to do.  In fact, three separate shutdown systems were activated.  I commanded the shutdown first, followed shortly thereafter by the computer thinking it had flown outside the shutdown box due to the abnormal velocity and triggering another shutdown code, followed a second later by Russ hitting the remote flight termination button and causing the independent flight cutoff valve to close.  The reaction times on all of this were quite good, especially considering the completely unexpected nature of the failure.  It is one thing to be watching a specific line and hit a button when something crosses the line, and another thing to analyze a completely unexpected situation and make a high-cost decision under pressure.  Everyone else reported that their first thought was "Why is John doing a touch-and-go?", and it took a little while to register that this was not intentional.

The vehicle had over 200 psi in the tanks when it went down.  There were two pressure vessel failures, both on the fuel side: one of the stubby "feet" on the bottom of one of the fuel tanks hit hard enough to tear at the weld, and the fuel pipe that supports the cutoff valve broke under the force of the impact.  All of the fuel was pushed out in very short order, and ignited by residual flames from the engine shutdown.  No lox was vented, although there were high heat signs of a small oxygen leak around our lox dump valve during the fire, probably due to the fire cooking the valve packing.  It would have been more dangerous if the lox tanks had also broken, but it is hard to say exactly how much.  It might have been just an extremely hot fire that melted the vehicle to slag, or it might have been a significant explosion.

All three remaining tanks were bent in somewhat at the bottom where the "feet" were, but they still hold pressure.  We will probably do some fatigue cycle tests on them now that the vehicle is scrap.  If this had been over dirt, or the tank bottoms had just been simple hemispheres with rubber bumpers without the threaded mounts protruding, the tankage would have most likely been completely undamaged after a 20' fall.  The top fuel pipe would still have broken, but that could be designed around if desired.  It is possible to make a pressure fed vehicle that can survive a 20' drop.

The fireball and blaze were very spectacular, but the parts of the wiring harness that were wrapped in leather actually came through the blaze ok.  With some more care, it would not be unreasonable to engineer a wiring harness that could actually live long enough inside a blazing wreck to allow you to do something useful, like vent pressure. One point that was brought up afterwards that I hadn’t considered was that alcohol fires are noticeably easier to fight than kerosene fires, since water mixes and dilutes the fuel, rather than just spattering it around. Several times a year, I debate with myself over the benefits of switching to kerosene, and this is one more point in favor of alcohol.

Everything inside the electronics box appears to be unscathed and still functional, although we know from a previous crash years ago that hidden faults may still lurk, so we aren't going to use them again.  We can give an extremely strong endorsement to McMaster-Carr " 85925K423 Fire-Retardant Silicone Foam Rubber Sheet Adhesive Back, 1/4" Thick".  We started covering our electronics boxes in foam years ago to give them a milder acoustic environment for the GPS units, but we managed to set the electronics box on fire a couple times when test stand engines misbehaved.  We eventually moved to this material, and it not only didn't burn inside this fireball, but it protected everything inside the box quite well.  The foam on the bottom of the box was hit by a high-pressure burning fuel jet from the plumbing rupture, but it didn't burn through.

We still have Pixel and Module 1 in flyable shape at the shop, so this doesn't have a critical impact on us, but it does change our testing plans for the next two months before the X-Prize Cup.  We are cancelling the untethered 180 second flights for Pixel at OKSP.  We will plan on doing two sets of back-to-back 180 flights under tether, but if we are going to risk a crash, it might as well be for the money at XPC now that we don't have a backup.  We are going to finish up Module 2 in the next couple weeks so we have a backup for level 1.  Modules 3 through 5 should also be at least frame constructed by XPC, but whether we get them wired and tested will depend on how our flight testing goes.  If we manage to destroy a module in the next two months, we can crunch hard and get an extra one put together if necessary.










Back-to-back 180s


Seven days after the crash, we did back-to-back 180 second flights with Pixel, fully demonstrating the performance and turnaround time for the level 2 lunar lander challenge.


The last 180 second flight we did showed some erosion on the fuel manifold below the nozzle exit, now that we aren’t hanging a crack-prone section of the graphite chamber down below it to protect it. For this flight, I made some phenolic rings and a stainless retaining plate to fill that area. We also increased the diameter of the four igniter holes to lower the igniter pressure and hopefully get less erosion.


Matt put parts of this in fast-forward, since watching Pixel sit there for six minutes is rather boring:



The phenolic spacers got burned through and ejected fairly early in the first flight, and the igniter still did erode a bit, but we went ahead and flew it again. After the second flight two holes had been burned in the fuel manifold, likely because it had already been pre-heated by the first run. This turns out to be non-catastrophic, because fuel spraying out cools the area, but the increased fuel consumption caused us to be running on fumes at 180 seconds on the second flight. There was no erosion after the first flight, so even though the phenolic rings got spit out fairly early, they did provide some protection. We are exploring other materials for the insulating ring, and also different graphite chamber designs that could avoid the need entirely, while still keeping the graphite in compression everywhere. The igniter holes didn’t seem to erode any more on the second run, so we can probably just increase the initial size to keep them from melting at all. The graphite chamber has no erosion after six minutes of flight.


Pixel had more rocket powered flight time that weekend than Space Ship One had in all of its flights combined. We have also spent more on operational consumables (helium, lox, alcohol, truck rental) than the vehicle itself cost, which is probably a first for any rocket vehicle.


Long Module Flight


Module development has stepped up in importance now since Texel’s drash. James has finished two more module tank sets:





The remaining modules will probably be done this month. We will probably fly a two-module dual-gimballed configuration after XPC as the first demo of “bolt together modules”, followed by a four-module differentially throttled configuration. The two-module configuration will need extra brace rods on the outside, but the four module configuration will be very strong with just the tank-to-tank connections.


For at least a year I had been telling myself to track down the shock stickers that they use on the Mythbusters TV show, but the Texel crash finally made me do it. It turns out that McMaster-Carr carries them…



For my birthday this year, my wife got me a crane truck. J



In hindsight, we should have gotten one last year. We have paid quite a bit in rental fees over the last few years, especially now that we are flight testing practically every weekend. The great thing about owning the truck is that we can customize it for our loads, which will save hours on each testing day. All the dewars will get bolted to the bed, the tool boxes will be hung below the bed, etc. We are making a roll-up weld curtain to protect the undercarriage, replacing the big boards we used to set up (hot rocks have a tendency to cut air lines under the truck):



We are intentionally dragging the performance of the module down to limit the hazard area at the X-Prize Cup. We are carrying 90 pounds of payload instead of the required 55, and we are using a smaller engine throat. In addition to having less injector elements, the injector igniter was changed from four angled holes to just two larger ones, hoping that it will allow more heat transfer away from the center of the igniter so it doesn’t erode. We machined a single graphite ring to replace the stack of phenolic rings as the insulator for the fuel manifold.


The first flight last week was probably the smoothest hover we have ever had, potentially due to using the IMU as a leveling inclinometer on the modules during gimbal calibration.




The chamber is perfect, and the igniter didn’t erode at all. The graphite insulating ring was not eroded at all, and still held firmly in place. Everything looked flawless.


For the second flight, we set up for a ground liftoff to check the code modifications made since the Texel crash. We were rather surprised when the top of the engine basically popped off. It looks like some propellant mixed and burned inside the helicoflex seal at the top of the engine. We had seen one case previously where the jacket had apparently unwrapped itself. It is possible that the faster turn-around times on the module contributed to a worse effect on the second flight here. The seals can be made with the open gap either inside or outside, but the usual practice for the manufacturer was to put the open side away from the media being sealed, so we put the solid side towards the higher pressure fuel jacket. This looks like it was a mistake, so we are going to get some made with it flipped the other way. We are also going to try normal viton o-rings again with a minor change to give it a little more heat protection.







© 2001-2011 Armadillo Aerospace, LLC. All rights reserved.