October 30, 2007 notes:
I could try and put the best positive spin on it, but the
bottom line is that we failed to win anything in the Lunar Lander Challenge at
the X-Prize Cup this year. We made two
perfect flights, and came within eight seconds of winning, but we also had
three engine-damaging hard starts.
Just like last year, this is a quick media-free update. Matt will be doing a separate update with all
the video when he gets back from another trip out of town.
There was a lot of stressing in the couple months before the
cup, but it was mostly over not having multiple backups for some of the things
we were taking, and we thought we were just about as prepared as we could be. We had already completed the full level 1 requirements
under experimental permit at the Oklahoma
spaceport earlier this year http://media.armadilloaerospace.com/2007_06_03/LLC1demo.mpg
, we had done six tethered flights of three minutes each, and we had literally
dozens of other successful tethered flights.
Crashing Texel two months
ago was a blow, but the modules were essentially flight ready, so we rushed the
permits ahead for them. It is worth
noting that we did not have the paperwork done at the time that would have
required AST to reply before the cup they would have been perfectly within
their rights to give us the permit a few weeks after the cup. We bitch and moan about AST as much as anyone
that deals with them, and I think we have plenty of legitimate gripes, but the
people we are working with very much want to see us succeed. We noticed that we were trading permit
related email with them on nights and weekends, which isnt the normal image of
a clock-punching federal employee.
One issue that we were
pitching a fit over was that somehow our rather expensive flight insurance policy
was deemed invalid for experimental permit work. After they had let us fly with it earlier
this year. Two weeks before the cup. Phil took on the work of getting it all
straightened out, which mostly involved separating out our non-flight insurance
and arguing over the validity of some industry standard exemptions. If you need rocket flight insurance, you
should be talking to Ralph Harp email@example.com,
he has gone above and beyond what we could have expected to help us out.
We were required by AST to do an untethered free flight of
the module at the Oklahoma Spaceport before flying it in front of the crowd at
the cup. We would have preferred not to
risk the vehicle, but I very much approve of actual demonstration over yet more
paperwork for convincing regulators, so I didnt argue too much. The OKSP people are always a pleasure to work
with, and the flight went perfectly: http://media.armadilloaerospace.com/2007_10_21/modFreeFlight.mpg
Right after we crashed Texel,
I had put in an order for another spare Crossbow FOG IMU, and was disappointed to
get a 14 week lead time for the replacement part (the FOGs now come from a
Russian company). I didnt worry about
it too much, because the IMU from Texels box
still seemed to be working, so I planned on keeping it as a backup to the main,
never-before-crashed electronics box.
When we were doing final backup checking in the few weeks before the
cup, I found that while that IMU was spitting out data, one of the gyro axis
was completely non-responsive. I really
didnt want to go to the cup with only one functional electronics box, so we
started scrambling to work a backup.
Plan A was to see if Crossbow had ANY other FOG based unit
in stock, even if it was a demo unit or one of the VG or AHRS models. Plan B was to see if Crossbow could repair
the unit in time. Plan C was to resurrect
the very first Crossbow IMU we used, which was somewhat lower grade, and required
slightly different software and a different power supply voltage. Plan D was to combine the Texel
crash IMU with one bad FOG and the IMU from our last peroxide vehicle crash
with two bad FOGS to get three functional FOG axis. Plan D was to resurrect the analog output KVH
FOGs that we used before we moved to Crossbow units. Crossbow checked for any units on hand, but
they didnt have any. I talked to their
repair tech, and he said that they wouldnt be able to fix anything for a
couple weeks because their bake-in ovens were completely booked with new
parts. Ah ha! All the new gyros had just arrived, so they
were able to skip a bit of final qualification and ship my original order out
the next morning. We also implemented
Plan C, so we had three complete electronics boxes to take to the cup.
One of the most distressing events was that one of the axis
drivers on our Haas trunnion table died a couple weeks before the cup,
preventing us from drilling injector holes.
It reported a cable fault, so we hoped that it would be an easy fix, but
after calling it in to our local Haas dealer, the days ticked by with no tech
arriving to look at it. Our daily (or
more often) calls had them telling us we were eighth in line for service, and
they will get there as soon as they can.
We asked them to see if one of the service people would take overtime
pay to come in on Saturday, and they agreed.
The tech checked the system and said there was nothing he could do, and
that we would have to send it back to the factory for repair. There were no demo or rental units
available. We would up taking it apart
ourselves to see if there was anything we could do, but the cable seemed to be
good all the way up to the deepest guts of the machine, so we finally boxed it
up and shipped it fed-ex off to Haas for repair. After telling them about the time-critical
nature of the problem, we got it back in short order. When we plugged it all back in, it gave
exactly the same error. We got the local
Haas dealer to send out a different tech, and he replaced a board in the mill,
which got us back running again.
We still had three sets of gimbal actuators, but due to some
issues that will be discussed below, we were concerned that we might break some
in the final weeks before the cup. The head
of http://www.ultramotion.com/ has
been very supportive of our efforts, and had two extra units rushed out to us.
Air Cryogenics http://www.tex-aircryogenics.com/
has been our lox supplier since we moved away from peroxide, and they have
really become personal friends as well. We
were still making changes in how we planned to measure and load lox for the
level two flights in the last couple weeks, and Reid was over at our shop long
after dark personally installing new dewar equipment. Since they were the official supplier for the
cup, we were a little worried that their might be some charge of favoritism
leveled, but the lack of any other competitors made it a moot point. After the lox truck and the helium truck had
left, Reid loaded up a final batch of parts for us into his pickup truck as he
was headed out for New Mexico.
Now we get to something that foreshadowed the outcome. We have gone through at least a dozen
graphite engine chambers in the last year as we have experimented with
different jacket and injector designs.
The team at Cesaroni Aerospace http://www.cesaronitech.com/
has given us excellent service all year, but as the cup approached, they would
machine an order from us the very next
day, and get it shipped out for next day delivery. Many of our chamber retirements were due to
an injector problem getting hot oxygen onto the graphite throat, resulting in an
uneven erosion (we had a few last-ditch chambers still on hand), but several
were due to cracks from startup transients, and a couple were more thoroughly
destroyed. We wanted to have four
pristine chambers of the final form on hand.
To reduce the worst-case failure distance that one of the
modules could fly, we had decided to shrink the nozzle throat from 3.25 to
2.75, giving a significant reduction in thrust. We had a few good flights with this
configuration, but we also had startup difficulties and eventually two hard
starts that completely destroyed engines.
Our theory was that the reduced throat caused more back pressure in the
chamber during startup, pushing hot gox farther back into the cooling jacket
before fuel filled it. We eventually got
AST to agree to let us use the same big-throat engines that we had been flying all
year, on the condition that we carry twice the required payload mass to ballast
it down a bit. We were happy with this,
and with the Haas mill working again and more graphite on the way from
Cesaroni, we had four engines we could use at the cup.
As a competitor, Holloman AFB turned out to be a great place
to work. We had better support and
facilities than we had at Las Cruces
last year, and we really didnt have any issues. We, and the event in general, had the full
support of everyone from the base commander on down. For instance, there was an awkward regulation
in place that prevented us from using our crane when the truck had both fuel
and oxidizer on board, so moving the vehicles around and standing them up was
resolved by having our base liaison just grab a half dozen convenient enlisted
men to heft things around by hand.
We had asked the chief judge for some specific rulings when
we arrived would we be allowed to fly the backup module as-is if necessary,
versus having to swap parts over to the original module, and would we be allowed
to fly Pixel for level one if we destroyed both modules. We were very happy to get a yes to both of
these on Thursday, but both decisions got reversed on Friday. The backup vehicle bit was specified in the contestant
entrance agreement (but not in the prize rules), so there wasnt too much we
could say about this, but we had been told on the ground by the judges last
year that we could fly the level two vehicle for level one if we were willing
to relinquish the chance for level two, so we were a bit upset about the
change. I didnt want to even try for
level two without having claimed level one.
The official ruling was that the tanks determined the airframe, and we
could swap everything else from the backup vehicle if necessary. While it turned out not to make any
difference, I dont agree with the thinking that doesnt allow backup vehicles,
since the point is to Get The Job Done, and backup vehicles versus backup
systems is a completely valid direction, and NASA had nothing against it in the
official prize rules.
The pad survey went fine, and we double checked everything
to make sure our new guidance algorithms were dealing with it correctly this
year. We had landed fine on the 8 meter octagonal
pads in Oklahoma,
so 10 meter circular pads shouldnt present a problem. The new pads were flush with the ground,
which prevents a tip over like we had on one of Pixels flights last year, but
it also made them harder to distinguish.
The lunar pad was more challenging than last year, with bigger rocks,
deeper craters, and no clear spot directly at the center.
We rolled across the starting line at 8:00 AM sharp, and
headed for the pads. Everything went
smoothly through the checklist, but the engine failed to start. Several tries kept resulting in an immediate
SHUTDOWN_IGNITER, because no significant pressure was being generated, and it
wasnt safe to open the main propellant valves.
Investigation showed that the fuel orifice on our igniter
was almost completely clogged, barely letting anything through. Russ had noticed that the igniter bleed step
in our checklist had taken much longer than expected, which should be a good
warning sign. We couldnt find any 0.030
wire to try to push through it, so we went back to our hangar and put our
pressure test plate on the bottom of the engine to allow us to back-flow high
pressure helium through the engine with the igniter solenoid removed to clear
it. This seemed to allow it to flow
normally, so we buttoned everything back up for our next try.
At 12:45 we headed out again. Everything went smoothly again, and we had a
perfect ignition and flight. Guidance
and control was perfect, letting us burn the X-mark off the target pad. Landing was perfect, with an instant engine
shutoff based on the ground contact shock signal. When Russ looked under the engine, he did
find that the igniter was somewhat eroded, which was unexpected, but it should still
work. We refueled for the return flight,
but during the igniter bleed we found that it was again almost completely
clogged. That likely explained the
igniter erosion, as the normally-rich igniter had probably crossed through
stoichemetric as the orifice clogged up again.
We were way ahead of the clock, so we had at least a half hour to try to
resolve the issue. What eventually
worked was a paper clip manually filed down to fit in the orifice. Once we had good flow, we finished the
propellant loading and got ready for the return flight.
The engine started with enough of a bang to cause the
vehicle to actually hop off the ground, but the engine was still lit and attached
to the vehicle at the end of the start sequence, so I lifted off to a few
meters altitude. From the exhaust plume
it was clear that the engine wasnt very healthy, but I decided to go ahead and
go for it. I hustled the vehicle up to
altitude, over to the second pad, and down towards the ground as fast as
possible, so if anything happened, it might be a survivable fall. There were some very disturbing impulse
events during the flight over, which were probably pieces of the engine
departing the vehicle. I was letting the
clock run out only a few feet above the ground when, with only a few seconds
remaining, something (probably half of the nozzle retaining ring departing) pushed
the vehicle hard enough to cause it to tilt past the abort limit even with the
gimbal scrambling over to maximum compensation.
The vehicle tilted over and fell to the ground on its side. Total flight time: 82 seconds.
It turned out that nothing was really hurt on the vehicle
besides the engine chamber, and we got the vehicle back up and offloaded
without any problems. Much to our
surprise, none of the shock-watch stickers on the tanks had tripped, meaning
that the body had taken less than 25 G shock.
While it was still functioning fine, the electronics box on top had a
handle bent in significantly, and both the 25 and 50 G shock stickers had been
tripped, but not the 100 G. I always
expected the computer box to take a lot less shock due to the wire rope
isolators, but this side landing seems to have whiplashed it into the ground
even harder. We moved this box to backup
status, and planned on using the one from Pixel for the next attempt.
We chalked the hard start up to some combination of the
eroded igniter or the manually reamed out orifice, and focused on the repeated
clogging of the igniter orifices. We had
never seen this before, but Paul Breed of Unreasonable Rocket http://unreasonablerocket.blogspot.com/
had reported very similar problems with his igniters. The Breeds reminded us of their issues, and
they gave us one of the sintered filters off of their display vehicle to use. This is quite heroic sportsmanship if you
think about it it looked like we had everything else well in hand, and
resolving the clogging problem was very likely going to result in one of the
prizes that they were competing for being taken off the table. We gave them as in-depth a look at our hardware
and operations over the next couple days as possible to try to return the
We put a backup engine and computer back on the module, and
with the filter installed and the brand-new igniter, the flow during the
igniter bleed was clearly better than the previous one had been all day. We still dont know the exact cause of the
clogging it might be the different ethanol that was delivered to the base, it
might have been contamination in the fuel filter that was just added to the
fill hose (we removed it before the second flight), or it might have been
something kicked loose internal to the vehicle over the thousand miles of transit
We again rolled out at 8:00 am sharp. The vehicle again made a perfect flight to
the center of the second pad. The
startup had been with a bit of a bang, but when Russ checked the chamber, it
was still fine. The igniter was perfect,
so we felt that the filter had prevented anything from clogging it, and we
expected the next flight to start fine.
During pre-flight, the igniter bleed was completely unchanged, so we
were very confident that aspect was resolved. Ignition was with a severe bang, and when I
lifted off, it was clear that there was even more significant damage to the
engine than on Saturday, with a lot of raw fuel coming out the bottom. Rather than risk it falling out of the sky at
150 feet, I just set it back down on the pad for an intact-abort. On inspection, the chamber was again
shattered, and a large chunk had already been ejected out of the nozzle, so various
burn-throughs of the aluminum would have been inevitable. Not going for it was the right call.
Our best guess for this hard start was that we were going
significantly faster between flights than we usually do at our test site, and
there might be more fuel remaining in the jacket or soaked into the graphite
than we have at our normal tests. We
decided that we would do a significant helium purge between runs, and possibly
just sit around for a little bit before doing the return flight.
We put yet another new engine on the module, and headed out
for the 12:45 window. Everything went
perfectly smoothly up to ignition, then we had the most violent hard start we
have seen. The engine jacket was ripped
off the injector, and the graphite chamber was completely shattered and thrown
over a wide area. We were done. We invited AST out to the pad to inspect the
aftermath, and we carefully noted where all the pieces had gone. We found a good sized chunk of graphite stuck
in the ground 64 meters from the center of the pad.
We had run out of easy explanations for the engine
starts. This was a brand new engine, so
it could have been some assembly or fabrication error, but we couldnt have
been so unlucky as to have three completely different causes of hard starts in
two days. We think it is likely that our
engine design has always been just on the hairy edge of ignition problems (the
problems with the 2.75 throat versus the 3.25 throat seems to imply that),
and some subtle difference in conditions between Dallas
and Alamagordo was just pushing it over the edge to a three-hard-starts-in-five-tries
level. The altitude might have something
to do with it. A one psi difference
doesnt sound like much, but when I opened the top of my toothpaste at the
hotel, it started extruding out all by itself.
Another rocket developer also commented that a seemingly reliable engine
developed problems after a move to a desert location. The lack of humidity might have played a part,
making it easier to ignite something in the cooling jacket. The ethanol might be slightly different. We just dont know.
There isnt anything I can look back at and say we did
obviously wrong. We had backups for
everything, we had demonstrated the required flight performance many, many
times, and we had made three free flights under experimental permit since the
last XPC, all without incident. The
problems we had at XPC last year were clearly solved our landings were
accurate, and our landing gear didnt break.
I am honestly very surprised that we didnt take any of the prizes. My final estimate before we made the first
flight was 90% chance for level 1, and 60% for level 2. There were a lot of things that branched off
of the if we win the LLC path that we arent going to be able to do now.
It is possible that we can fix up the ignition with some
simple changes to the purging, but at this point I am inclined to make some
drastic changes to the engine design. I
had already been worried about in-air restarts when coming in for a landing
with low tank pressures, so we now have more than enough incentive to make
starting absolutely reliably a top engineering priority. At least two of the hard starts clearly
happened inside the cooling jacket. We
have a much larger volume jacket than is conventional, enabled by the graphite
chamber not actually requiring all that much cooling. This gave us the benefits of very low
pressure drop and a lack of sensitivity to minor tolerance issues, but it also
caused the start sequence to take a lot longer, and provides a lot more volume
for a combustible mixture to form in. We
could go back to an uncooled chamber like we had last year, or possibly even a
simple stainless chamber with heavy film cooling. I dont think we could make a 180 second
flight with one of those engines, but practically everything else that we care
about would still be fine. We have
learned a lot of things about igniters and sealing in the last year that would
allow us to make that class of engine run better than it did for us
before. For the regen engine, I am going
to investigate moving the throttle valve after the cooling jacket to eliminate
the need to fill all that volume. The
raw graphite would leak fuel through the pores, but we have been talking about
chrome plating the chambers to avoid oxidation anyway. A face-shutoff pintle is sounding even more
inviting from a dribble volume standpoint.
None of the vehicle frames are hurt, but we took back three
wrecked engines, and mod1 needs a completely new wiring harness. We are going to take a little time off to recharge,
and when we get back to it, we are going to have to give some more priority to
the Air Force and NASA contract work. I havent
read any of the event coverage yet, or even gone through my rocket related
email yet, because I am still sulking a bit.