December 29, 2001 Meeting Notes
I have not been able to replicate the problem that led to
the crash last week. We have lots of
supporting data, but no smoking gun.
To recap, the situation was that after warming all the
engines, I initiated active control and slowly throttled up. The main engine opened up properly, but
stayed at 34% open (not leaving the ground) as I continued to throttle all the
way up. When the joystick was at full
throttle, the flight computer finally acknowledged it and opened the main
engine all the way up, causing it to leap off the ground.
We do not have any way of telling if there was something
special about the 100% joystick mark, or if it just started working again at
that time. I believe that I had it
there for a brief amount of time before it launched, implying that it was a
timing issue instead of a positional issue, but it was all quite rapid, and I
wouldnt be able to swear to it. If I had
logged local joystick state in the log file as well as the returned telemetry
state, we might have gotten a couple more clues.
The telemetry graphs:
The telemetry log shows that the joystick stayed at 34% for
four solid seconds, then jumped immediate to 100%. During that time, all the other sensors were giving data like normal,
so we certainly didnt lose telemetry from the flight computer. From looking at the text dump of the data, the
Y axis was fractionally off center during the dead time, and it also stayed in
the exact same position (0.2 degrees) for the four second span, then instantly
changed to a new value (0.0) at the same time the throttle changed from 0.34 to
1.0, supporting that it was the entire joystick ceasing to update, not just the
While you couldnt see it on the video, there is a very
brief 10G acceleration spike in the telemetry when it broke the tethers. It is only a 10G accelerometer, so it was
likely a bit more than that. We have
never gotten a good positive G reading in our hops, even on this one that took
off pretty hard. The vibration forces
seem to be larger than the linear forces, so it stays very noisy. I am currently only logging the most recent
IMU data 20 times a second, which means that it cant even be averaged properly
because it is only getting 10% of the samples.
Not relevant to this investigation, but something that I intend to
address in the future.
The tank pressure signal on the dual tank system has been
pretty noisy. When it throttled up, the
tank pressure dropped a bit, but when the throttle closed, the pressure
increased back up a bit. It looks like
there may be a venturi effect going on, where the pressure registers as lower
when it is flowing past the transducer.
We should probably move the transducer to one of the top ports where it
just sees gas.
There was a question that the flight computer may have stopped
receiving updates from remotePilot, which could be a networking problem on
either side, but I was able to look at the sequence numbers on the joystick
data in the binary version of the telemetry data and see that they were indeed
continuing to come in as expected, they just werent changing over those four
The flight computer previously immediately exited if it
stopped getting a steady stream of command packets from the remotePilot
application. I recently changed the
behavior to have it continue running the attitude engines, but zero the
throttle, which would cause it to fall straight down instead of possibly
tumbling. I checked this out again, and
it does behave as designed. If it had
stopped getting telemetry packets, the throttle would have dropped to zero.
If the telemetry and flight computer are behaving correctly,
the problem must lie with either the joystick itself, the remotePilot
application, or something in the WinME setup on the laptop.
I theorized that a loose connector might have caused a
temporary loss of updates, so I checked out exactly what happens. If the USB connector is pulled out on WinME while
remotePilot is active, the next joystick read will return an error, which causes
my code to clear all the buttons, which would have stopped the engines. I may want to change this behavior to just
zero the throttle and axis, leaving the button active so it falls with active
attitude control if it was in the air.
If the joystick is plugged back in, normal updates will resume. I did not previously print a warning on
joystick errors, but if that had happened, we would have seen several seconds
of no buttons and no throttle, which was not the case.
It isnt relevant to this investigation, but I also checked
the disconnect behavior with the linux joystick driver. Pulling the USB connector doesnt give any
acknowledgement, and reconnecting it doesnt bring it back to life. I may wind up patching the kernel to give an
error on the next read if the device goes away.
Without a definitive reproduction of the problem, we are
just going to have to be conservative.
I am going to swap that joystick with one that I have at home, just in
case it is a hardware problem. I am
changing the software to allow a separate system to send the joystick packets,
while the laptop just logs telemetry and draws graphics. The joystick system is a stripped down linux
box that will be running NOTHING else.
The tethers are being beefed up in several ways. The vehicle rebuild will have attach points
as strong as the main frame. We are
going to big screw shackles rated for overhead lifting for connecting the
chain. We have run nylon cord through
all the chain links, so there will be about two feet of stretch to absorb shock
before the chains come up hard:
If we were just worried about runaway throttle, it would be
better to connect the tethers at the center of the vehicle, but keeping them at
the outside also provides protection against some classes of attitude engine
failure, and keeps them from getting cooked by the engines in normal operation,
so we are probably going to leave them there.
Bob has ordered the tubing needed to repair the vehicle, but
it wont be here until next Thursday, so we are grounded for at least two
We did our first test of the small engines we are intending
to use to turn a rotor: If we have the
engines spinning at typical rotor speeds, they will see over 4000psi of inlet
pressure, so they need to be much sturdier and higher expansion ratio than our
From left to right: catalyst chamber, anti-channel rings,
catalyst pack and pack compressor, drill press pack cutter, copper gasket,
nozzle. We wound up using a copper
sheet gasket instead of the ring shown here.
We havent measured exactly how much pressure we have been
using to compress the foam packs in previous engines, we have just been
counting discs for consistency. When I
was building this one at home, I added foam discs and just leaned most of my
weight onto the pack compressor. A 0.65
diameter pack with 100 pounds on the compressor is seeing about 300 psi across
the pack. This closed the foam up quite
a bit, to the point that it was moderately difficult to blow through. I dont think we have been compressing our larger
engines nearly this much, but we really need to get a gauge on the press. I was pretty much assuming we wouldnt be
able to use foam in an extremely high pressure engine, and this supports that.
The test firings were fairly rough, and the power was very
low due to the small amount of flow. We
have had turbulent flow issues with testing small motors on the test stand with
the current tank manifold, which we are going to try to isolate next test
We made a pure silver screen catalyst pack to replace the
foam pack. To attempt to get enough
silver in the same area, we really leaned on the press while packing the
screens. It turns out that we
compressed it to almost as much restriction as the foam pack, but it did take
probably over ten times as much force (we need a gauge!) to do it.
We fired the engine that way twice, and it made a bit more
power and ran smoother, but wasnt fully catalyzing, giving a cloudy exhause. Our next test day we will run another
version of this engine with the catalyst pack depth increased from 1.29 to
2.55 and a less densely packed silver screen pack.
I am also going to change the fastemers from stainless
socket head cap screws to brass studs and nuts. They will be weaker, but they will have the same thermal
expansion as the body. The copper
gasket sealed on the first run, but the bolts needed to be re-torqued after it
had been hot. Once we get a reliable
pack combination with this engine, we will do some long duration firings and
see if a re-torqued seal continues to seal over many cycles. Our other motors close at the inlet side,
and use a silicone O-ring to seal. That
seals very well, but the O-rings do degrade with time. Right now, our packs go quicker, so it isnt
a problem, but if we fix our packs to last longer, the O-rings will be the next
thing to wear out, so we want to move to an all metal seal.
The hollow shaft we are going to use for tests was at Russs
house, so we didnt get to fully assemble it, but we have everything we need
for our initial spinning tests:
We are using the second lander frame for a test base, with a
bearing plate bolted where the main motor used to be. We are using a rotary seal from Rotherm rated for 1800 rpm, and
we have a tachometer sensor and DC motor for external spinup if we choose.