It has been a long time since I did an update, but it hasn't
been because nothing has been going on.
Quite the opposite, in fact -- things have been incredibly busy. Outside of Armadillo, the big new for me personally
was that Id Software, my video game company, was acquired by Zenimax
Media. This doesn't have any real impact
on my relative time commitment to Armadillo, since I expect to continue
devoting the majority of my time to software for the foreseeable future, but it
does mean that I have more personal resources to call on if necessary.
I would have had Phil provide some updates, but he has been
overloaded dealing with our FAA/AST waiver applications and other business
logistics on top of actual fabrication work.
Our full time head count is going to be going up some soon, so there may
be hope for more frequent updates in the future.
This will be the technical update, I'll leave it to Phil and
Matt to put together the media and color update. The X-Prize website has some early videos of
the flights, but I expect that there will be some great footage from the NASA
TV crew later.
Last year we had done several 180+ second flights with Pixel
before the Challenge event (as we had the year before that...), but when we
tried to fly during our official window we had a valve movement problem and
burned through the engine, flipping the vehicle onto its side. The damage was minor, and when we got back
home the vehicle passed a hydro test, so we could have just dusted it off and
flown it again, but we has a couple reasons to change direction. The module vehicles fly so much nicer than
the quads -- the control authority is much better, they don't have propellant
trim issues, and they have slosh baffles inside the tanks. We also knew that the new "amateur
rocketry regulations" were due to come into force this year, which would
allow us to fly in many more areas than the experimental permits we had been
getting would, and with the LLC this year allowing you to make your attempt
wherever you wanted, it would be a huge advantage to be able to fly at our home
The problem with this is that there is a firm limit between
what you can fly under waiver and what needs an experimental permit or a full
launch license, which we were not going to be able to get anywhere near our
hangar. That limit is 200,000
pound-seconds of rocket impulse. When
Pixel did a 180 second hover flight, it ran
about 240,000 pound-seconds, and there wasn't going to be any way to
nickel and dime it down by 20%. The mod
vehicles just couldn't do the mission at full propellant load in our normal
blowdown pressurization mode, so we had to add an external high pressure helium
tank and fill the tanks completely full instead of just half full to carry
enough propellant. This became the
The February Armadillo update had a lot of details on the
catastrophic results of our first hover test in this configuration.
There wasn't anything deeply wrong with the setup, but it
was several months before we got around to trying again, since we were busy
with NASA and Rocket Racing League work.
Using a ball valve for programmable high flow pressure regulation works
just fine once the kinks were ironed out.
Based on our later experience, we didn't have enough helium loaded to
fully pressurize the propellant, so it would have settled back down on the
tethers sometime towards the middle of the burn if it hadn't self
destructed. Even if we had the helium,
it probably would have run into fuel depletion at about 160 seconds, with a
fair amount of lox remaining.
Before we tested again, we went on a weight savings program. The legs were the most obvious thing -- we
had designed them to hold up a bolted together quad-module configuration, so
they were much heavier than necessary.
The new legs were smaller, shorter, and thinner, resulting in a savings
of 60 pounds, as well as being easier to land among the boulders and craters of
the level 2 moonscape. We also lightened
up a number of other things, and there was probably another ten pounds we could
have saved if we had to.
The next couple tests were successful, but we found that we
couldn't get enough helium for a full 180 second run into the carbon fiber tank
we were using with just normal helium tanks that come at 2500 psi or so. The painful solution was to add some 6000 psi
helium tanks to our loading procedure, and cascade fill first from the six-pack
of normal bottles, then change over to a regulator on the 6000 psi bottle to
bring the vehicle bottle up to 4000 psi or so.
This was a workable solution, but inconvenient and more expensive to
operate. This was the "single black
Unfortunately, it was still slightly over the 200,000 lb-sec
waiver limit. It could make it, just
barely, if we offloaded propellant to the point that it would sputter out of
propellant at 181 seconds, but then any tiny variance in the process or
environment would have it dropping down early.
There were two paths to take for improvement, we could either increase
the engine specific impulse or reduce the dry weight. We pursued both in parallel.
The engine was essentially the same as we used on Pixel, but
because the Super Mod was several hundred pounds lighter, the engine ran at a
lower throttle level and chamber pressure through the entire flight, which also
meant a lower specific impulse. We
started working on another injector that used the same design, but had a
greater number of smaller injection elements, and we had some new cast chambers
made with 3.25" throats instead of 3.5" throats to increase the required chamber pressure for
hovering. We didn't get to the point of
assembling these, because the weight savings came together ok.
We pulled a little more weight out of the vehicle, but the
only quick thing that could save significant amounts of weight would be
changing out the high pressure helium bottle.
The bottle we were using was a relatively old Lincoln Composites
compressed natural gas tank, and weighed about 60 pounds. After searching around a bit, we had a
conversation with Microcosm about their Pressuremaxx linerless composite
tanks. They were very interested in the
project, and offered to become a vehicle sponsor with discounted tanks. Hence, the "Scorpius Super Mod".
When the tanks arrived, we were very impressed. They were only 12 pounds each, and had a
combined pressure x volume capacity greater than the single 60 pound tank we
started with. Designed for 2200 psi
operation, they would also not require the separate 6000 psi helium
supply. This was the "dual black
tank configuration" that there were a couple pictures of. Unfortunately, on our first loading
operation, we heard a SNAP sound at 1400 psi and one of the tanks started
leaking down. We weren't going to be
able to fly on it, but we continued testing the other tank and it did the same
thing at 1800 psi.
We were quite disappointed, and a little worried about the
schedule. We sent the tanks back to
Microcosm for analysis, and they agreed to build a second set of tanks as soon
as possible. The final verdict was that two
factors contributed to the failures: We had the tanks floating in their mounts
with leather pads on both ends, but we had only allowed about 1/8" for
expansion, and they believed that the tanks could expand up to 3/4" in
length before bursting. The other
factor, which took me a while to accept, was that the tanks had overheated
during pressurization. We were
pressurizing at a rate of about 10 psi a second, which was slower than we had
been pressurizing the other tanks, but being linerless, there were load bearing
fibers and matrix essentially directly exposed to the helium. They suggested that because we were filling
the tanks from the bottom, the helium on the top was getting more and more
compressed and heated with minimal circulation to the rest of the tank. This fit the evidence, in that both failures
were near the top of the tank, and when cut apart, there were clear signs that
the matrix had gotten very hot near the broken area.
While Microcosm was rushing to build new tanks, we had more
tests to do, so we put on four other composite pressure vessels we had laying
around. These were 15 pounds each, so
the weight was the same as the single big black tank, but they were designed
for operation around 2000 psi, so we could use the same loading procedures we
were planning for the Microcosm tanks.
This was the "four green tanks" configuration that we posted a
few videos of, and we did back-to-back 180 second flights with it.
While testing of this configuration, I hit on an idea that
wound up being very useful -- we set up a counterflow heat exchanger that ran
the lox vent through the outside, and the helium fill through the inside. This let us get significantly more helium in
the tanks, and completely quashed any worries about overheating the helium
tanks. They actually start to frost over
at the beginning.
The replacement Pressuremaxx tanks arrived a week before our
contest window. We had a Microcosm
engineer come out and assist us with the acceptance testing and installation,
and everything went fine. The only
change they made was to make them white for better use in the sun, so this is
the final "white tank configuration" that we flew at the LLC. On the vehicle side, we arranged for a full
inch of potential length expansion, and filled the tanks from the top to
prevent the heating piston effect, although our lox heat exchanger makes that a
rather moot point.
We had a legal vehicle, but the propellant depletion was not
completely even, so we wound up with something like 35 pounds of fuel remaining
at oxidizer depletion. We spent three
more flights tweaking in the mixture ratio so that we would have usable
propellant at the end for margin, then one final repeat flight to make sure it
was doing what it was supposed to.
When the vehicle was down to its contest weight, the
throttle was only at about 50% through the entire flight, rather than the 70%
that we were at for most of our earlier long flights. This makes the Isp rather bad, but it also
means that the vehicle could accelerate quite quickly if we were shooting for
altitude. It also caused the hover
control code to modulate the engine a lot more than it used to, so I modified
the control parameters to smooth it out a bit.
I also wound up tuning the attitude control a bit, since it turned out
that our "rocket tug of war" optimized parameters tended to make the
vehicle sway around more at hover than it used to.
We had 15 full loads on the super mod before our Lunar
Lander Challenge attempt. One of those
loads was scrubbed when the original Microcosm tanks failed, and the first
flight ended catastrophically after only a few seconds, but that was still
nearly three million pound-seconds of delivered impulse from the vehicle, not
even counting the previous flights as a normal module. That is around $30,000 in direct testing
Because we built these pads ourselves, we made sure that
they were on a precise east / west line, and exactly 60 meters apart. We build Pad A sturdy, with good thickness
and reinforcement, because we expect to be using it for all of our upcoming boosted
hops, replacing the steel plate in a clearing we have used for the recent NASA
related free flights. The lunar Pad B is
sort of a throwaway, but we made the boulders removable, so we may fill in the
craters later if we need a backup pad.
It cost about $8000 to build the pads and gravel roads out to them. I am very glad I didn't skimp on the gravel,
given the weather we wound up dealing with.
In all of our previous free flights, I had been operating
the launch control computer from inside a truck or van quite close to the
launch location. Since we are going to
be able to do a lot of flights under waiver at our home base, we have set up a
"mission control" inside the air conditioned office space at the
hangar. The launch pad is around 2000'
away, but we changed to some patch antennas for communication, and I was
actually able to increase our transmission bit rate while improving the
Since I didn't have direct visibility to what is happening
on the pad in this configuration, we set up a camera that could see both pads
and give me a view on a monitor. It
wound up taking 2500' of Cat5 cable to make the run, and that is the hairy edge
of feasible for a composite signal. None
of the modern TVs would acknowledge the signals existence, and even on an older
one that would display it, the color was almost completely gone. Still, it was good enough for me to see the
vehicle in the first 30' or so of altitude.
We have used a couple different types of on-board video
transmission systems in past years, and they have all had various issues. This year, I decided to try using IP cameras
and the existing telemetry link. This
turned out to be a success, with some caveats.
The focal planes varied quite a bit from unit to unit on the cheap ones
we used, and the included software had a lot of issues. The cameras just send over jpeg frames, and I
could set up a combination of resolution, update rate, and compression that I
was satisfied with, but the recording feature of the software transcoded the
jpeg stream into, of all things, IV5 format, which turned it into utter
crap. This wasn't a huge issue for us,
because we also included separate HD cameras on the vehicle, but it frustrates
the software engineer in me. I will probably
be integrating direct support for IP cameras in my mission control software so
I can automatically log snapshots during the checklist, and record the flight
video as a full quality mjpg.
We didn't want to risk the Super Mod in free flight, but we
had other flying vehicles on hand to test the traverse and landing. We started out with the methane module that
we had been flying for NASA, first just doing a normal tethered hover over the
lunar pad so I could see what it looked like, then moving to free flights from
Pad A to Pad B. We wound up having a
burn through on the methane engine on the first pad to pad test. The methane engines continue to give us
trouble when throttled down, and we need to do more development if we want to
use them operationally for VTVLs. The
burn through led to the vehicle sliding away from the pad, aborting when it
tilted too much, then shutting down and dropping a few meters to the ground.
Aside from mangling a couple legs, It turned out not to have
hurt much of anything, so we converted it back to an alcohol fueled rocket and
had it flying again that weekend. One
weight savings that we made on the super mod, and also made to the other mod
for equality, was to take out the manifold purges and their plumbing. In hindsight, this wasn't worth the
savings. The engine starts and runs
fine, but it doesn't shut down cleanly, leaving licks of fire going for a
while. I can sort of pump it out by
using the igniter gox solenoid, but it is a mess. This turned out to be more of an issue with
ground touchdowns than for tethered tests, because there is a little puddle of
burning alcohol close to the vehicle. I
didn't want to touch the perfectly working Super Mod at that point, but we are
going to put them back on this week.
My first landing was a few feet off, and I actually bashed
one of the boulders with a foot, but it didn't upset the vehicle much. We spent more time orienting the cameras so
that they were exactly straight down, and the second landing was less than a
foot off the center mark. We will
probably go to cameras that aren't quite so cheap in the future, it was
frustrating that the optics weren't aligned very well with the case.
Our biggest worry was the weather. A week out, it was expected to be raining all
weekend, and the forecast never improved.
We had a local television crew out on Thursday night to witness what we
expected to be our last flight test before the challenge, but the rain rolled
in just as we were preparing for it. We
did manage to get that final test done on Friday morning, and when the judges
and X-Prize representative arrived that evening, I was semi-seriously
suggesting that we just go do the flights right then, instead of waiting for
Saturday. It wouldn't have been valid
without the NASA representative on site, and Peter Diamandis would never have
forgiven me, but it was a very tempting notion.
Saturday morning, it was raining. Lots of people had weather displays going on
phones and laptops, and it looked like there was a clear section that should
pass over us, but it wasn't moving. The
judges and X-Prize officials had all been as reasonable as we could ask for
when we discussed contingencies. If we
aborted due to weather before the first flight, we could start over. If we aborted after one leg was completed, we
could pick up from that point later. If
it rained all weekend, the judges and X-Prize official would be willing to stay
through Monday, although we would also need to convince a NASA rep to do the
After surveying the pads in the afternoon, we were very
worried about getting our trailer stuck behind the crane truck, so we decided
to use a smaller trailer towed by a separate vehicle for the rocket, allowing
the crane truck to go straight in and out to minimize the chance of getting
bogged down. We discussed various
contingencies, including having me fly the rocket over the crane truck if it
got stuck by the lunar pad. We would
have had to offload residuals into a carboy and carry the rocket back, but we
could have managed.
Finally, at 3:00, it stopped raining and we rolled out. It did start misting during our loading
operations, but we felt it was reasonable to keep going. We got the fuel and lox loaded, but when we
were nearing completion of the helium fill, Russ noticed that there was a leak
from the high pressure servo valve packing.
Fifteen previous flights had never seen one. This was tense, because not only was the
contest timer ticking, but our lox was also boiling off at about three pounds a
minute, so if we had a 15 minute delay, I would be very worried about running
out of lox while over the pad.
TODO: valve stem leakage is a chronic problem, we need a
better torque spec and locking mechanism.
The pad crew got it
done in about ten minutes, and we were ready for liftoff. I triggered the ignition.... and got an
igniter misfire. I realized that I had
skipped the checklist step to bleed the gox line to the igniter. I was using the same
"checklist_superMod.txt" file that we had been evolving through all
of our testing, but because this was an untethered free flight, there were
several steps that would be skipped around ignition time, and in the
excitement, I skipped one that I shouldn't have.
TODO: create event-specific checklists.
TODO: modify the ignition timings to gox lead automatically
and / or change to the more reliable igniter geometry on the rocket racer
After I got the engine lit, it lifted off nicely. I had decided to burn off the first 20
seconds of propellant close to the ground at the launch point, because that was
the part that was hardest on the engine, and I wanted to be able to get it down
safely if it had a burn through. After
running that out, I ascended to 55 meters, moved over the 60 meters to the
second pad, then descended to 10 meters to run out most of the clock. With 40 seconds to go, I dropped the altitude
to only 3 meters for the final precision dial in. As I was trying to make the final
adjustments, it didn't seem to be responding correctly. It landed with one foot in the center of the
pad, or about three feet off center. It
was also rolled 90 degrees off from where it should have been, which shouldn't
have been possible, since the roll control keeps it within +/-10 degrees.
When the pad crew got back to the vehicle and we started our
detank procedure, we found that the lox hose couldn't reach the vehicle at the
current orientation, and we didn't want to move the crane truck off the gravel
road. We couldn't move the vehicle until
the judges noted the vehicle position, and the judges couldn't approach until
we removed all the propellant from the vehicle.
We decided to just dump the lox through the engine, which was also
faster. This worked fine, and the fuel
hose was long enough to reach the truck, but it got us off our checklist, which
had a later effect.
I had noticed a couple things out of the corner of my eye
when I was doing the flight: the
telemetry had some bad spots, and the vehicle shadow sometimes looked like it
was at a 45 degree angle, which wasn't normal, but we were in higher winds than
our previous flights, so I hadn't worried too much about it. If I had been aware that I wasn't at the roll
angle I thought i was, I probably would have been able to figure out the right
way to nudge the vehicle for a perfect landing.
As it was, it would have been better if I hadn't even looked at the
on-board camera views.
TODO: draw a heading arrow on the vehicle shadow in my
TODO: multi-person mission control with additional displays,
so we can have another set of eyeballs watching data.
This was extremely tense for me. I reviewed the telemetry and found that it
had fired one pair of roll thrusters early in the flight, but that had caused
it to slowly sail past the point where the other pair should have fired, and it
just kept going. It didn't even try to
fire them. I knew what was the probable
cause for this, but it didn't initially make sense. We had known for a while that one driver
channel on this electronics box was flaky, working sometimes and not working
others. This only controlled one if the
redundant roll thrusters, so it wasn't a big deal, but one of the last things
we did before the contest was to move that pair of roll thrusters to a
different driver channel. Each connector
is either a bidirectional motor or a pair of solenoids, so moving that one
connector moved a pair of roll thrusters.
We did a full 180 second test flight after this change, and everything
worked fine. When I looked back at that
test data, what I found was that the roll thrusters hadn't fired during that
flight either, but it had still stayed perfectly straight all by itself.
We test every actuator before each flight, and the roll
thrusters worked perfectly when I manually fired the bits that they were
on. The problem was that the patch to
the code to remap the channel was wrong.
I made what I thought was the correct fix, and we started the checklist
for the second leg of the flight.
TODO: create command-roll-left and command-roll-right
buttons that can force computer initiated testing the roll control under
As we started the second leg, we got bit by the
circumstances at the end of the first flight.
When we had to improvise the offload procedure, we missed the step in
the final checklist where we vent the residual
pressure in the helium tank. In
the aforementioned actuator test, I cycle the helium regulator servo valve open
and closed. The way we added the
external pressure tanks to the existing mod system was by having a couple hoses
from the servo regulator that could be plugged into the existing pressure /
vent valves. These hoses are free
hanging during the early part of the loading operation, so when I tested the
regulator valve with pressure still in the tank, they flailed around and hit
Russ and Mike in the head. This is the
worst safety incident we have had in almost ten years at Armadillo, and is a
great example of why you should wear safety goggles during the entire process
(which is checked on our checklist).
They both got a good lump from this.
I didn't even find out about it until after the flight when I asked Russ
why he had a pad duct taped to his cheek.
TODO: add an explicit "verify high pressure vent valve
is open" step before the actuator test.
We have this for the main propellant tanks.
TODO: modify the gas manifolds so they have a dedicated,
permanent port for the regulator gas, instead of a manual connection.
As we started going through the checklist for the second
flight leg, I noticed that the pad camera television and the on board camera
display computer were both shut off. We
had had a power outage at the hangar. It
hadn't effected the laptop that I run the main flight control program on, but
it must have also interrupted the wireless radio that communicated with the
rocket. If that had happened during the
flight, the vehicle would have gone into an auto-land from wherever it was.
TODO: get a UPS for everything at mission control.
When we got through loading everything, I made damn sure
that I bled the igniter gox line this time, but I still got an igniter
misfire. I tried again with the same
result. The fuel tank pressure had also
oddly dropped a lot. I realized that I
might have made a mistake in my panicky fix for the roll thrusters, and when
the pad crew mentioned that they had heard a roll thruster fire when I tried to
start the engine, it was clear that I had moved a bit the wrong way.
TODO: take more than 60 seconds to evaluate changes to
flight control software. When
One more build and we were off.
I was prepared to try to manually keep the roll in bounds by
occasionally pressing the actuator bit that fired a thruster if it looked like
it wasn't working, but this flight was completely nominal, and the terminal
guidance also worked properly, so I landed one foot from the center. The offload procedure was also completely
nominal. We had a half hour to spare.
While the judges were verifying the position, the pad camera
display started flashing "low battery", and went dead a few minutes
later. That wasn't critical to any of
the operations, but I would have been distressed if it had happened while I was
TODO: check all camera charge levels before important
It was great to win in front of the home crowd. It is bad to fail, but it is worse to fail in
front of your wife...
The home field turned out to not play a part in the success
this time. I was sort of expecting some
mad dash back to the shop to build something for a second try, but we didn't
need it. Back in 2007 at our first try,
it would have been the difference between losing and winning the level 1, since
we would have been able to just cobble together some rubber bumpers to replace
the unusable shock feet.
The challenge would have been better if it didn't include
the loading and unloading of the vehicles from the transport vehicles. It was silly to have the judges declare our
flight valid, and still have to complete the loading of the vehicle back onto
the trailer. People should be justified
in cheering when the vehicle lands.
If NASA wants to do a Level 3 Lunar Lander Challenge, my
suggestion would be to have it be a single system that makes a 120 second
flight to a lunar pad, has a short stay time, then makes a 120 second flight
back to the launch pad. Contestants
choice to either stage for the flight back, or have a very high performance
vehicle do it in a single stage. This
would be much harder than the current level 2 challenge, and we could not do it
by cobbling together a vehicle from the things we have on hand. It isn't clear that Armadillo would be participating,
because we have enough business prospects that we may be too busy satisfying
customers to compete.
Checklists are very, very important. It is tempting to just assume you know what
you are doing, but experts still make plenty of mistakes. On the other hand, it is important to be able
to improvise when reality throws you a curve.
There have been some interesting articles in recent years about the
value of, and resistance to, checklists in the medical fields. I am quite proud of the checklist integration
in my control software, but we should continue to improve it with integrated
note taking and audio / video logging.
It is clear that we can add significant performance to a
blowdown vehicle design with external pressurization, and this was a useful
experience, but a blowdown vehicle is faster, easier, and cheaper to
operate. We probably won't operate an
externally pressurized vehicle again in the next year. There are too many things that we need to do
that can be done just fine with blowdown.
However, It is very likely that Microcosm tanks will show up in a lot of our design studies in the coming
If we did wind up
doing a lot of work with high pressure helium, we would want to invest in a
high pressure pump so we could transfer helium around from the partially
emptied six-packs that you are left with after a cascade fill. If we started using that, it might also be
tempting to filter and reclaim helium from the propellant tanks, but I don't
think the hassle is justified yet.
Coming Up: higher altitude boosted hops, scientific
payloads, more rocket racer flights, and multi-module configurations.