December 12, 2008 notes:
We are pretty sure we know what caused the valve
movement problem that downed Pixel at the LLC. Our systems run the
propellant valves through a set of relays controlled by a watchdog
microcontroller, which drives the valves closed if the main computer doesn't
continuously toggle a keep-alive signal bit. We found that with the
valves on the bench, we could get interruptions to the current when we tapped
on the relays in the electronics box. We believe that acoustic vibrations
from the operating rocket engine could cause the contacts on the relay to intermittently
This seems plausible for a number of reasons:
We see little spikes in the current trace during the problem times, as if the
relay contacts were only occasionally closing.
The acoustic energy gets much worse for ground liftoffs than when elevated four
feet above the ground on tether stands. We have seen similar problems
with GPS units in the vehicles.
It is worse for Pixel than the modules, because the heavier GLOW needs more
thrust, and the electronics box is closer to the engine.
It wasn't a problem for the Rocket Racer, because the electronics box was
isolated farther away in the cockpit.
We tore one of the relays apart to look at the construction, and there wasn't
even a real spring in them, it just relied on the springiness of the metal
conductor. Our usage of the relay is not energized during normal
operation so that a failure of the watchdog's battery won't bring the vehicle
down, which means there was nothing but the spring tension keeping the current
path open. These electronics boxes are also now three years old, so they
could be getting a bit less springy with fatigue.
We discussed various options for changing the cutoff system, but we still came
down on the side of wanting real, physical relays instead of a purely
electronic solution. Russ found some relays with much stronger springs
for us to replace the current ones with. This is a good example of how
safety systems can have negative consequences. We have seen several other
examples of this over the years, and most people don't really appreciate it.
We still have some concerns about the limit switches inside the valve
actuator. The switch is actually wired up as a DPDT with two diodes, so
there is a tiny spot where the motor is actually disconnected as the switch
changes position, which we can see as a 5 msec divot
in the current traces. We have made a modified actuator circuit with four
diodes and a DPST switch that avoids the issues, but it allows an unloaded
actuator to spin past the stop point on momentum. With a valve on, it
seems to work fine, and we have flown the methane module with it running the
fuel valve with no problems. We are
probably going to replace all the actuator electronics with custom boards in
From a reliability standpoint, it is tempting to remove the valve limit
switches altogether, and just have the computer always keep the valve in the
right range. That would remove a possible engine failure case, where the
limit switch somehow interrupts your control of the valve, as well as also
getting the diodes out of the control path, but it adds problems to our
termination sequences -- we currently just drive the valves closed on a hard
shutdown command, since it is possible to have burned through the feedback
wires while still having control of the valves (speaking from experience), and
we still want to be able to close them off in that condition. We might
consider having a physical stop and just driving the motor for a fixed amount
of time that would be safe to stall for, but that would still mean that we need
to be very careful about not leaving a motor drive enabled when we exit the
flight control program.
We have had a fairly miserable time with the methane
engine work we are doing for NASA. The basic idea was to adapt our
alcohol engine for methane, adjusting the element sizes to get the mixture ratio
correct. Bottom line -- it didn't work.
We made over a half dozen different engines with unlike impinging elements for
methane. If we just used an alcohol engine as-is, it could be made to
burn once we sorted out igniter values and startup sequences (need to chill the
methane manifold as well as the lox manifold, making horizontal testing rather
undesirable), but the mixture ratio was grotesquely rich. Whenever we
shrank the fuel elements down to bring the mixture ratio up, we ran into
combustion stability problems.
We have never had high frequency stability problems with alcohol, even when
deep throttled down to a negligible injector pressure drop, but we saw a lot of
"interesting" things with methane. Our film cooling was getting
badly disrupted, such that even a poorly performing engine would make the
entire chamber rapidly heat red hot. We killed many pressure transducers,
caused several spark plugs to disintegrate, and constantly vibrated swagelock fittings off. We made one run with a vise
grip clamped over the swagelock nut. In the
video playback, you could see the vise grip fall off the engine, which could be
correlated with the pressure signal disappearing on the telemetry trace.
We then put a hose clamp over the nut and torqued it
down until the clamp was solidly bent over the nut. It still came off.
Most of these engine had a perfectly flat reading on the chamber pressure
sensor (in the five or ten seconds before the sensor would die), but the sensor
was only sampled at 200 hz.
However, you could hear the screech audibly in video playbacks.
We tried increasing the pressure drop across the injector by using smaller and
smaller element holes until we couldn't lift off at full throttle and max tank
pressure. It still had problems. We tried varying the feed
pressures between the lox and methane to change the relative injection
velocities. Still had problems. We had
made a couple engines that could run on the test stand at full throttle, but
none of them could dynamically throttle on a hovering vehicle for more than ten
seconds or so without something coming apart.
Finally, we gave up and made an injector with like-impinging elements instead
of unlike-impinging. We immediately had great vehicle flights. The Isp was lousy, but it wasn't much
of a surprise that we would need a longer chamber to give the separated
propellants time to mix. The chamber was actually frosted over in the
upper part, so we added a few inches to the chamber, and performance went up by
20%. The nozzle still wasn't very hot, so we added another three inches,
but the Isp only improved by
5%. With some changes to the injector elements, we have finally brought
the performance up to the level of our lower performing alcohol engine on
module flights, but that still isn't saying much. We have another
injector going in for test this weekend. The injector face also wasn't
getting the heating that the unlike-impinging designs got, so we moved back to
304 from 310 for future like-impinging designs.
It is interesting to note that because a vehicle loaded with methane has a
lower mass ratio than the same vehicle loaded with alcohol due to the different
fuel densities, the blowdown pressurization curve
that is almost perfect on the alcohol module winds up needing a higher initial
pressure and a steadily rising average throttle position when on methane.
We wound up taking off the 80 pounds of LLC payload ballast that we had on the
module to bring the mass ratio up a bit. We may yet wind up investigating
V-cut ball valves for more precise deep throttling. Related to that, we
also have more mixture ratio skew during throttling and more pressure skew
across the blowdown range.
We are also trying a new
fabrication technology for making chambers rapid prototyping investment
casting from http://www.proivc.com/ . This is more expensive than the spun-pipe
based chambers we have been using, but it frees us from the limits of standard
pipe sizes, and makes changing nozzle dimensions easier. The surface finish is rougher than we had
hoped, and there was some slight ovaling in the
chamber, but it is still promising. We
havent fired them yet.
We have made five good 60 second flights with different injector / chamber combinations. They are all pretty boring, but this is
We also tried flying the same vehicle with self-pressurized lox / methane instead
of helium pressurized. It couldnt
generate enough thrust to liftoff, but ignition, shutdown, and throttle up were
all smooth, so this is quite encouraging.
It is clear that the flow is two-phase in the injector, because the
reduction in liquid density alone would not explain the level of reduced thrust
we had. Interestingly, it also didnt
make much more thrust as we took the self-pressurizing level from 150 to 200 to
250 psi. We have a new injector with
2.5x the element area to try next. If we
have good success with this, it opens up several interesting options for us.
We are planning on starting
higher altitude flights at our home base in February, as soon as the brand new
amateur rocket regulations go into effect.
We should be able to fly to 8000 altitude right behind our shop, which
is going to be a HUGE benefit for us. We
can learn a lot, and hopefully get the inevitable tragic loss of vehicle as
we go into a new flight regime out of the way without having to make long road
trips. Coupled with Spaceport America just
getting their spaceport license, 2009 is finally the year where everything is
going to come together.
Even though Pixel was not hurt very badly in the LLC tip-over, we are going to try
to adapt a module for next years level 2 lunar lander
challenge. The module can fly for about two minutes in the current
configuration, so we are adding an external pressure bottle to allow us to fill
the tanks completely full of propellant. We may also need to fabricate
some lighter weight legs, which we would pull in a bit to give it a smaller
footprint for landing in the clear area of the "lunar surface"
We could probably get by with just using a big Tescom
regulator for this purpose, but it was a good opportunity to go ahead and
develop the servo regulator technology that we will need for the much higher
flow rates on passenger carrying vehicles. We tried making a ball valve
controlled pressure regulator several years ago, but backlash on the ball valve
stem gave us problems with the control algorithms at the time. With the
current error-diffusion pulse width modulation that we use, it worked out much
better this time, and we can nicely control tank pressure to an arbitrary curve
over a flight. It is useful to sort of mimic a blowdown
pressure curve to avoid requiring the propellant ball valves to operate down in
the very non-linear cracking range.
For the LLC, we are just hanging the pressure bottle opposite the ballast
weight in the middle of the vehicle, but we plan to mount the pressure bottle
up inside the nosecone for high altitude flights. Launched from a stand
without legs, that should be a 100km capable vehicle. Paying customer
work for the Rocket Racing League and NASA take priority over this work right
now, but we should be ready for a three minute hover test fairly soon.
This is only tangentially related to Armadillo, but
many of you will still be interested. My Tesla roadster, number 30 in the
production run, arrived this month. I used to be known for my crazy
turbocharged Ferraris, and I have a lot of great car stories to tell from those
days, but Armadillo wound up getting me out of the supercar world by taking
over all of my discretionary spending and eventually making me fairly frugal
and money conscious.
When I first heard about the Tesla, it piqued my interest because I had
occasionally looked in on the high performance / homebrew electric car world,
and I personally knew Elon Musk through SpaceX. Elon was daily
driving a McLaren F1 when I first met him, and he has been one of the most open
and honest people I have met in the aerospace world, so I asked him for a
straight up opinion about the prototype vehicle. "It is
blow-your-mind awesome." he replied.
I talked about the car a fair amount, but I hadn't made any six-figure personal
purchases since founding Armadillo, and Armadillo was
still bleeding lots of money at the time, so it probably wasn't going to amount
to anything. Out of the blue, Anna decided to get one for my birthday (my
wife is awesome!). That was a few years ago. Being a software
developer, I am sympathetic to schedule slips, so I didn't fret much about
it. Every month or so, someone would ask me when the Tesla would be getting
here, and I would just say "sometime soon". Finally,
At the peak of its modifications, my twin-turbo Testarossa
had pulled 1010 hp at the rear wheels on a dyno.
This is a level of performance that has basically ruined the experience of most
high performance cars for me. When I later bought a stock F50, my driving
impression was basically "Meh. Pleasantly
quick, I suppose." (I committed a sacrilege and put a mild turbo system on
it to make it live up to its looks). The Tesla wasn't going to knock me
out with neck snapping acceleration, but that wasn't what I was looking for.
For years, my drive to the Id Software office afforded me an almost daily
opportunity to run out fourth gear on the service road, which let the crazy
cars earn their keep. After we moved to a new location, the route just
wasn't there anymore, and I would find that weeks had passed without the
throttle ever going to the floor. For just tooling around town, high
winding, high boost engines just don't come into their own -- it takes a degree
of planning before aiming for 9000 rpm and 25 pounds of boost. They were also, well, "loud" is understating it a bit.
Everyone at the office always knew when I was arriving, and in a lot of
situations, exercising the cars would just be anti-social.
Responsiveness is what I wanted from the Tesla, and it delivers. There is
only a single forward gear, so there is no shifting or being in the wrong
gear. You have the full torque of the motor from a dead stop, with no
power band and clutch to deal with. Your right foot really is an
"accelerator" rather than a "throttle", and it honestly
does transform the driving experience. Coupled with excellent traction
control, there is very little reason not to floor it every time you come away
from a stop, getting smoothly pulled forward like you are on an enormous rubber
band. As Robert Duffy put it, "You are driving a railgun."
The car is obviously very quiet, but the engine has a nice whine as you
accelerate. It doesn't have the wide-eyed, slack-jawed,
religious-experience type of performance that my Testarossa
used to have, but there is a lot to be said for racking up a dozen
happy-car-grins a day with your normal driving.
Internal combustion drive trains, with all the gears, clutch, oil, and exhaust
start seeming remarkably primitive in very short order. I have hopped
back and forth between BMWs and the Tesla for a couple weeks now, and while
there are plenty of creature comforts that are much better in the BMWs, every
time I pull away, I wish I was in the Tesla. I am using it as my daily
driver now, whenever I don't have to haul any big packages for Armadillo.
As I am driving it, the range is only about 150 miles on a charge, but that is
still plenty for what I need, and I just plug it in every night.
I do also confess to enjoying the irony of my driving an electric car. I
am fairly hostile to most of the environmental movement, finding it generally a
modern tribal religion that justifies condemnation and control of others in the
name of protecting the environment. I care nothing at all for the
environment in isolation, only for how it positively impacts human life --
civilization is all about beating the environment into forms that suit us
better. An "electric car" used to be a conspicuous sign of
righteous sacrifice, but you won't get any self-flagellation points for driving
a Tesla. Too much fun.