April 8 and 12, 2003 Meeting Notes
There is a nice article on Armadillo Aerospace in the May
2003 issue of Popular Science.
Servo Valve Work
Russ and I definitively solved the distressing
engines-stop-moving problem on Tuesday. I was testing some code for the
initial-cracking calibration test that involved holding the throttle at a
specific position. Russ was in the room with the rocket, and reported
that the valve was buzzing back and forth very rapidly, then it just slowed
down and stopped after about 15 seconds of buzzing in place.
This sure acted like some kind of thermal cutout, but left
to itself, it never started moving again. We figured out that because our
driver board was left in the "close" state, the residual current was
keeping the thermal cutout from ever resetting. If I explicitly set the
driver board to the undriven state and we waited a few minutes, the valve went
back to normal operation. We exactly duplicated the behavior on a second
valve. The reason the stuck valve on Saturday started working again
wasn't because Russ moved it manually, it was because we had powered off the
system, allowing the PTC to reset. The
limit switches were not part of the problem.
Russ found the PTC on the valve servo circuit board that was
causing us problems, so we opened up all four valve cases and bridged across
them. Operating for long periods of time like this will probably get
things heated up, but we tested them for 30 seconds of 180hz cycling without
any performance degradation. On the big vehicle, we can probably drop the
cycle rate to 90hz, since we will have a whole lot more polar moment of
inertial. That should cut the heating rate way down, and make 120 second
flights not a problem.
Problem solved. While we had them open, Russ also
removed the fast-on connectors on the motors, and soldered the wires directly. We have no reason to worry about the ones on
the valves in particular, but we have had one crash tied to a slipping fast-on
in another place, so we might as well preemptively fix them.
We tried getting the one valve with the weird feedback range
to cycle back into the normal range by manually pushing it, but it would always
rotate the full 360 degrees back into the offset position, and the reading
slipped somewhat more after our experiments, so we decided to swap the entire
valve out. The new one has feedback values right in line with the other
The engine that felt weak was indeed just suffering from a
slight opening miscalibration. We manually tweaked the calibration to
compensate for it, and it is now quite close to the others.
We thought that this was the end of our problems, so we were
going to plan for the best and hope to do a hover test on Saturday, then go
right out to our test site for a test launch to 1000 or so. The local FAA gave us clearance, and the
weather was perfect, but it was not to be.
On Saturday, we set up for another captive hover test, and
had the parachutes all installed so we could make sure they arent going to
melt during engine firing. We also
installed a camcorder mount so we could get flight video in the future.
Just as it lifted off, the computer died again. We pulled the electronics out and did some
of the last things we could think of to improve reliability we replaced the
molex power connector to the PC104 stack with a directly crimped line, and we
ran redundant power and ground in through the PC104 pins as well as the
dedicated connector. We added three
more angle bracket supports for the electronics bulkhead in an attempt to
reduce flexing, and we tried to mount the board with bonded-rubber isolators,
but the angle brackets didnt provide a flat enough surface to make that work.
Our current mounting arrangement for the electronics board hasnt
worked out all that well. We have angle
brackets with studs coming out of them mounted to the tube, and the honeycomb
electronics bulkhead needs to fit down over them, then be clamped down with
nuts over big fender washers. This
invariably involves a lot of fussing around to get everything fitting in right,
because the angle brackets are stamped things with rather poor dimensional
control, and the hand mounting of the individual brackets isnt exact. I had been sort of planning on making some
nice milled angle brackets that perfectly conform to the curvature of the tube,
and have a good, true top surface, but I have decided that if we do another
vehicle like this, I am going to mill out a complete 24 OD, 1 wide ring from ½
thick aluminum plate that we can secure into the tube just like our bottom
engine bulkhead. The electronics
bulkhead would then sit completely flat and level on top of this ring. We could tap the ring, which would allow us
to but bolts through from the top, which is a much easier proposition than
aligning on studs from the bottom. We
dont have much trouble taking apart and putting together our nosecone and
engine bulkhead, because it is easy to get one bolt started, then use an awl or
center punch to lever the rest of the holes into position. If we want to do a really good job with
vibration isolation, we could bond a ring of rubber on top of the support ring,
then bond another metal ring on top of that, which is what the electronics
bulkhead would actually bolt to.
We got it all together again, then went out to try the hover
test again. As we were preparing, I
noticed that the pressure reading from the drogue ejection tank was reading off
the scale. I went back and looked at
the data from the previous run, and found that the transducer was giving bad
readings after the warmup, and it jumped an extra volt to the end of the range
during the liftoff. This is the second
pressure transducer of this style that has died like this, and we believe that
in both cases they briefly shorted the main power bus, causing the computer
crash. Russ did an X-ray of the
transducer, and it does have a little circuit board with several frail wires
and poor support inside it, so the general-service transducers, part number
31685k1 from McMaster-Carr are now officially poison. McMaster seems to have stopped carrying the heavy duty
transducers that we havent had any problems with, but I think they are still
available from Omega.
So, the things we changed on the board probably werent
necessary, but they should improve reliability in the future. We had Joseph shake the rocket on the lift,
and drive around the parking lot with it swinging around, even bumping over a
curb with it, and didnt have a single problem.
However, when we fueled it back up for another hover test,
it died again right at liftoff. The
interesting thing this time was that 40 seconds later, the telemetry came back,
with the flight control program still operating. I theorized that this was the power-cycle reboot time for the
Esteem wireless bridge, so I started up a data log and pulled the power lug for
it, then reconnected it. Exact match
40 seconds of interruption, then a return to normal operation.
The Esteem circuit board wasnt mounted all that firmly, so
we fabricated a bracket to hold it better.
When we went to mount it again, we noticed that there was also a
recessed reset button that very likely could have been getting hit be a
vibrating front panel, so we completely removed the front panel.
We fueled up for a third test, but the resetting behavior
was even worse than before one blip of an engine would cause it to
reset. We could wait 40 seconds and
blip the engine again, and it would keep going off like clockwork. If we lowered the vehicle so that one leg
rested on the ground instead of letting it free-hang, it could fire without
resetting. Shaking the vehicle by hand,
either on the ground or off, never caused it to reset, but Russ did get it to
reset once by banging a leg with a big pipe wrench. This is one of the most crystal-clear examples of a vibration
problem I could imagine.
We pulled the parachutes out after this to see how they were
doing, and there was a small burned spot on the parachute bag, even through our
nomex insulating sleeve. The spot was
right at the bulkhead line, where heat was conducted from the base of the
vehicle through the thick aluminum. The
parachute wasnt hurt, and this probably wouldnt have been a problem at all if
we hadnt been blowing lots of propellant out with the vehicle sitting on the
ground, causing lots of convective heating of the base plate, but we are going
to take some steps to improve the conditions.
Adding fastblock insulation to the entire baseplate as well as the
engines will remove most of the heat transfer, and we will probably add some
kind of rigid sleeve for the deployment bag to keep the one spot from getting
We found that tapping on the side of the Esteem reset switch
could cause a reset without even pressing it, so we pulled the board and
desoldered the switch completely, and fueled up a fourth time.
It still shutdown when the engines fired. At this point, we conclude that the Esteem
circuit board probably has a broken trace, and is just very sensitive to a
certain vibration range. This was the
same board that was in the Oklahoma vehicle crash, so it isnt too surprising. We have a spare unit that we can replace it
with, but unfortunately
That was the very last of our 90% peroxide.
We made attempts to hover this vehicle on five separate
weekends now, and it still hasnt been done right, making it the most
frustrating of any of our vehicle generations so far, but we have a good understanding
of what has been happening:
Driving the servo valves hard caused us two different sets
of problems: Russs original quad driver
board burned out transistors when it was rapidly cycled, and the valves
circuit breakers would trip after 15 seconds of very fast back and forth.
Shorted out pressure transducers seem to have caused two
The old model Crossbow IMU died on us during test. This part had been in the Oklahoma crash.
The Esteem board seems to also have been damaged in the
Oklahoma crash, and caused several hangs during test.
It does seem like we had some intermittent power problem
that was vibration sensitive.
There seems to be a moral here about using parts that have
been through crashes: DONT. The Esteem
is a $2000 part, and the Crossbow is an $11,000 part, and they are both
many-week lead-time items, so I had some motivation to reuse them, but we
probably shouldnt have.
We have done a whole lot of fixes to the system that
probably werent directly related to any of the real problems, but should
improve reliability over any of our previous systems. We have less connections between things on the board than ever
before, and higher quality components are being used throughout ring terminals
for all connections on our custom boards, and mil-spec double crimp connectors
Some flaws in the software have been pointed out as a result
of the various anomalies, and a couple more are still to be fixed.
On the positive side, the architecture we are using now has
no inherent scaling limits (unlike our solenoid based previous systems), and
once it is all proven out, should be the basis of our X-Prize vehicle and
future launch vehicles. If we had more
peroxide, I think we would be flying next weekend.
We had really hoped to get off one good altitude flight
before running dry, but we just didnt make it. We have been in conservation mode for months now, leaving a 4,000
lbf monoprop and 1,000 lbf biprop engine sitting on the floor, because we
wanted to save all our peroxide for flight tests instead of engine tests. The biggest strategic error I have made was
not going ahead and becoming an anchor tenant for X-L Space systems instead of
letting them shut down. It was a tough
decision, because we would have had to be purchasing peroxide for nearly a year
before we really needed the extra volume, which would have meant sitting on a
big storage tank. At the time, I also still
thought it was going to be reasonably easy to deal with FMC once they realized
we were a serious customer making >$100,000 orders.
Our Degussa sales rep was ready to sell to us, but once it
went to their legal department, we hit the same wall as with FMC. We have made in-person visits to their
corporate offices, and we are just finishing up a full presentation for FMC and
Degussa about our company, our plans, and why it is in their best interest to
sell us peroxide, but there is no firm list of actions that we can take that
will guarantee a proper response from them.
I am funding a small operator to begin running a
concentrator for us, which will get us back in the air, but they cant possibly
provide the quantity we need for the X-Prize flights (10,000 gallons or so for
the full development program).
We are probably going to be at least two months without 90%
peroxide. In that time, we will
continue fabrication work on the full size vehicle, and we are going to do more
tests with alternate propellant engines.
Changing to a bipropellant will complicate our task quite a bit, but if
we do work through it, it gives us more performance headroom and drastically cuts
down the cost of flights. (Look at the
silver lining, damnit