June 20, 2004 notes
Good intentions, bad results
If anyone missed the mid-week update I did about the perfect
flight of the small vehicle, go read that first. We had problems trying to hover the big vehicle this weekend.
A minor improvement I have made to the laptop software is to
display the dilution of precision (PDOP) number from the GPS on the status
light, because previously it would have been green even if it was running with
very poor coverage. I need to spend
some time and collect more GPS diagnostic information, like satellite S/N
ratios, but Im not sure if I want to add a lot more traffic to the serial port
during actual run times.
The first problem we ran into was installing the batteries
in the new battery boxes on the big vehicle.
We have started using upset thread lock nuts for a lot of things now,
but when rapidly tightening them down on one of the battery clamp studs, it
galled onto the stud. We use stainless
for everything, because a little mist of peroxide will corrode steel bolts pretty
badly, but galling is a common issue for us.
Using anti-seize prevents this, but we should probably use nylon insert
lock nuts in areas that arent going to be heat affected, and leave the
all-metal locking nuts just for areas near the engine. We had to cut off the studs and weld new
ones in place, all inside the vehicle, because the battery boxes are
permanently bonded in.
When we rolled the vehicle outside, we noticed something off
when I ran an extra check on the jet vane controllers: vane one was moving
about half speed one direction, but full speed the other direction. We puzzled over this a bit, finding that swapping
controllers around always had that actuator displaying the move-slow-one-way
behavior, so we were confident it was the actuator. This was the new one that we had replaced after melting one of
them last test. We were thinking that
we could still test with a slow drive, because it was still moving fast enough
to work, but while exercising it with normal flight movement, Phil looked over
our way and let out a cry of Smoke! A
pretty good quantity of smoke was coming out of the cabin where the electronics
were mounted. One of the motor drives
was fried. Fortunately, we have plenty
of spares of the driver board, so we prepped up another one and replaced it. To make us even more confused, when we
tested the problem actuator directly with a manual switch box, it was full
speed both ways, and when we pulled the valve off and tested it with a
different channel on the burned board, it also ran fine both directions. The new driver board still showed the
slow-one-way problem, so we stopped quickly to avoid smoking it. We finally thought to check the resistance from
the motor drive to the vehicle frame, and found that there was a low resistance
path from one of the drive leads to the motor shaft, which makes its way to the
vanes, conductive graphite bearings, up through the engine, and back to ground
through the spark plug return line.
None of the other valves were like this.
When we pulled the actuator apart, we found out what was
wrong: We make a few modifications to the actuators, primarily removing the
thermal cutout and soldering on any internal quick connect terminals, but we
also usually solder the drive wires directly to the board instead of attaching
them through the little bare wire push down connectors that they are normally
connected with. We found that one of
the wires had been pushed so far through the hole when it was soldered that it
brushed the metal case below it. That
made perfect sense. The actuator was
perfectly normal when bench tested, because the motor connector didnt have a
path to ground. When we drove it with a
manual switch and battery, it could draw enough current to still move full
speed even though many amps were shorting directly to ground. When the motor drive was pushing it, it
would suck as much power as the transistors would give it, burning them
up. We trimmed the wire down, put the
valve back on, and everything worked properly.
We set everything up, and started loading propellant. Because we were collapsing the vacuum hose
last time, we rearranged things so that the venturi vacuum pump was plugged
directly into the vehicle, with the hose going to it carrying 70 psi air. It took us a little while to get the
adjustment setting where it wasnt wasting huge amounts of nitrogen, but the
general arrangement works well. It is
unfortunate that we need a ladder to connect to the vehicle top port, we will
probably try and get all connections at the bottom in the future, using a
fountain tube to get the inlet above the liquid level.
Things looked good when we started pressurizing, but when we
got over 200 psi, it started clearly leaking into the engine. After a little while, the engine popped and
started crackling from internal combustion, which we know is bad for its
lifespan, so we hustled to get things disconnected and ready to go.
Just as I was about to start, I lost the flight computer. It did that same thing last test on the big
vehicle, but it never happened once on the small one, so there is definitely a
problem someplace. The computer came
back again by itself, and we cautiously tried to burn off the propellant.
We had covered the engine facing sides of the 2 foam stand cubes
with adhesive aluminum insulation, hoping that it would keep the foam from
turning into flying molten plastic, but as soon as the engine was throttled up
a bit, the aluminum on all blocks just blew right off. The vehicle then showed the same problem as
last time, even though we were at a higher tank pressure: as soon as it
lightened up, the blocks blew out, but the slow valve and low thrust to weight
had it falling down and bouncing on the tether before it gained positive
thrust. Just as well, because it turns
out than vane zero wasnt moving at all.
I gave it enough throttle to slacken the tether a couple
times, but it obviously wasnt able to control itself with one dead vane. The leaking propellant valves kept the
engine heating up continuously, rapidly bringing the entire nozzle to full red
heat. Just as I finally got all the
propellant burned off at a fairly low throttle, three little fires lit up down
by the nozzle. We got them extinguished
quickly, and took stock of the situation.
The aluminum covering on the foam blocks was a horrible
mistake. Melted plastic last time was
bad enough, but the adhesive on the back of the aluminum was a tar like
substance, and it now covers everything, making about the worst mess you can
imagine. Foam worked so well on the
lander vehicle, but for the big one it looks like we really need to make the
blow-out blocks from metal, and make them rigid, both to avoid melting them and
to avoid the premature blow-out problem.
The little fires were from the RTV we used to seal
underneath the inner bearings. I
expected RTV to ablate instead of burn, but cooking them long enough did set
them on fire. We need to switch to a
Cotronics ceramic sealant. All of the
actuators and wiring seemed to do just fine, even after the extended high heat
Vane zero was stuck in the 45 degree position the entire
time, getting a full heat soak, then getting hammered by liftoff level thrust. This vane was bent after the test. All the others were fine, because the
computer never moves them more than 20 degrees from vertical, and they usually
stay within a few degrees of vertical.
We are going to hammer this one straight and continue using it, but we
will probably move to a better alloy, as well as thickening it, on our next upscale.
When I did a testValves, vane zero did a complete 360 cycle,
then went back to normal 90 degree range, indicating that it had been hung up
on one of the limit switches. We have
seen this a couple times before, and I think I know what may have happened. We had had been doing testValves to check
the speed issue, which cycles the valves all the way each direction and reports
stats. We probably left it in that
position before the test, and when we pressurized the tank up and the engine
started going by itself, the gas flow may have exerted enough force to nudge it
past the limit switch. I should have
the computer always try and leave the vanes at zero degrees when possible. Even though the valve is back in range,
going back and forth properly, the pot feedback range has shifted. We have had this happen to several valves
over the years, and we dont know exactly why.
I will recalibrate, but it is a concern.
We also didnt get thermocouple data on this test, so I
spent some time on Sunday finding out why.
It turns out that when I rebuilt the electronics after the lander crash,
I changed the thermocouple pinout slightly, and I forgot to duplicate the
change in the big vehicle after making it in the new subscale vehicle. A mistake I made correcting it did give me a
useful data point: I had made a
connector to just bridge +12v onto the thermocouple signal wire to give me a
definitive max scale A/D reading. After
I found out that I had the wrong signal wire, I changed it to a different wire,
but I accidentally put it on yet another wrong one. When I plugged my bridge connector in, the laptop lost
communication with the flight computer.
After unplugging it, I could reconnect with it, and found that the
flight computer hadnt rebooted, it had just lost communication. I had accidentally connected the signal to a
ground, and when I bridged +12v to it, the DC/DC converter couldnt supply
enough power to the wireless radio, which shares +12v with the sensors. The computer is on a separate +5v DC/DC
converter, so it was unaffected. This
is good to know. If we lose contact,
but come back to find the flight computer not rebooted, it is probably a
voltage problem with sensors. If I had
been running the sensors on unregulated power, a short like this would have
probably generated more smoke and brought everything down. I finally got the thermocouple on the right
wire, and it works fine now. I need to
check the vane actuators and see if rotating them all the way around ever
causes the pot to sit in a zero resistance range, which would draw bring down
the telemetry. That might be what
We really need to get a non-leaking ball valve before
I am going to write some more automatic checkout code for the
flight computer, so it wont even start if any of the vanes arent moving, or
are out of the calibrated pot range. An
override would be required to operate a malfunctioning vehicle (to vent
propellant, for instance). I am
hesitant to do the same thing with the throttle, although I could, in theory,
alternately open and close the master cutoff and the throttle for full checkout
without venting anything.
There was another anomaly during the flight that we havent
noticed before: the main throttle valve was hunting close to zero even when I
wasnt doing anything. I think this is
a result of recent software changes, and it is benign, but I should fix it to reduce
the stress on the drivers and actuators.
We are still waiting for big shock absorbers to replace the
wire rope isolators. We arent going to
do a ground liftoff until we have them installed.
In other work, we are setting up improvements to our test
stand plumbing, and Phil made a fiberglass nose for the small vehicle, which we
may use as a GPS-transparent ejectable cap to allow us to have an emergency
parachute on the vehicle, if AST will allow it on our waiver.
Good luck to the Scaled team on the flight tomorrow morning! Im a late riser, so by the time I check
news, there should be plenty of writeups on the entire thing