November 12, 13, 15, and 16 (busy week!) meeting notes
We prepared for and conducted our first remote flight test
at the Oklahoma Spaceport facility in Burns Flat this week. Several lessons were learned.
We built a checklist for our flight operations, which was a
very good idea. Going through it before
setting off caught several things we almost forgot.
The five hour drive from Dallas to Burns Flat was rougher on
the equipment than expected. The tarp
we put over everything to keep people from staring was damaged by the wind in
several places, and the wooden cradle we transport the vehicle on actually
broke one of its 2x4 support bars. We
are probably going to arrange some hoops for the trailer so we can tarp it like
a covered wagon in the future. I may
consider an enclosed trailer with a suspension in the future.
Everyone was extremely helpful in Oklahoma, and we set up in
the middle of a service road well off from the main airport runway. Our expected altitude with only five gallons
of peroxide was under 1500, and our parachute drift range with 13 mph winds
was only about 2000, so we had plenty of room. Bill Kourie from OSIDA stayed with us to communicate with the air
traffic control tower during our launch activities.
Our setup was a bit slower than we expected, but everything
got done fairly smoothly. The VOX on
the radios we brought was more trouble than it was worth, often triggering with
wind noise, but this was our first time using radio communication.
We did a full water test, then loaded up five gallons of
peroxide. The engines all warmed
quickly, and ran perfectly clear, even though it was in the mid 50s.
When we were cleared for our launch, I smoothly throttled up
the engine over a two second period.
The vehicle tilted a little bit on liftoff, but seemed to straighten out,
but it then continued tipping, eventually tipping all the way over and flipping
into the ground from a hundred or so feet up.
There was still peroxide left in the vehicle tank, but all
the pressure had drained out by the time we reached it. We tipped it up to allow the remaining
peroxide to drain down into the main engine and slowly catalyze away, then we
carried the vehicle back to the road to run some low pressure water through it
to clean it up.
We drove the remains to our bunker to strip off the good
parts, and left the main body there.
The telemetry cut off only four seconds after throttle-up, indicating
that the computer died, but there was very valuable data.
Immediately after liftoff, there was a +Z angle rate kick,
probably caused by the funny takeoff aerodynamics underneath the tail
flare. The piece of aluminum sheet metal
we put under the rocket for ground protection was folded in half and crumpled
up after liftoff, which was completely unexpected. You can briefly see that in one of the liftoff video frames. The rate peaked at 22 deg/s, with the
opposite attitude engine full on, then it started coming back down. The liftoff
test last week did not show this behavior, but the feet were changed, and the
surface was different this time. It is
also possible that the main engine mount was slightly distorted by the
The Crossbow stopped updating 1.25 seconds before telemetry
The vertical acceleration was right at one G when the
Crossbow stopped updating, and very smooth.
This was slightly higher than expected, indicating about 600 pounds of
thrust from the engine at 280 psi takeoff tank pressure. The plumbing on the test stand was
definitely limiting performance compared to the straight shot on the
vehicle. The welded catalyst pack
continues to perform very well.
The battery voltage started dropping rapidly at this point,
but the computer continued operating for another 1.25 seconds, until the
battery voltage reached 9v, at which point telemetry ceased. The 15v power converter for the Crossbow
probably suffered a voltage drop before the 5v power converter for the main
computer. The main engine feedback
potentiometer reading fell off as the 5v supply dropped below 5v, and the
engine pressure transducer started falling off faster than it should as the
supply voltage dropped below 10v. All
of this points to a general power system failure, rather than just a computer
power failure (which has triple redundant connections to the main power supply
from the manned lander work).
During the last 1.25 seconds of operation, the computer
continued using the last valid Crossbow data, which caused it to hold the same
two attitude engines on, which built up momentum on all three axis. Presumably the attitude solenoids all closed
when the computer died and stopped sending an active signal to the solid state
relay boards, but quite a bit of momentum could be built up in that time. The main engine would remain in the
full-open position. As the vehicle did
its flip, you could see it slowing down while it was pointed upwards.
The flight control code has in the past had stop-all-engines
behavior when the crossbow stops updating, but on this flight there was no cutoff
checks, which was a mistake. If there
had been, the rocket would have just dropped from about 20 in the air, and
suffered much less damage. The exact
timing for deciding the crossbow isnt working is a tough judgment call, but a
quarter second should certainly be enough time to decide that the attitude engines
should cut off. The decision to cut the
main engine is harder, because the vehicle should be able to continue flying as
an unguided, aerodynamically stabilized vehicle if it is going fast enough, but
right-off-the-pad, it could turn into a land shark.
There was one GPS update after liftoff, showing it at three
meters above the ground, but with only a small vertical velocity. The processing latency on GPS velocity and
position may be different.
My initial thought was that something had shorted, perhaps in
the motor drive feedback or pressure transducer, which have power running to
them from the main bus. When we opened the
electronics box, the cable to the battery positive terminal was not connected. The battery still had full voltage in it, so
we believe that the terminal came off during the flight, causing the voltage
drop that led to the failure, rather than during the crash. It is unfortunate that it seemed to work
during the water test and warm-ups, but the drive from Texas probably loosened
the connection to the point that it was barely hanging on. The batteries have slip-on connectors, which
have bothered me for quite a while, but screw terminal batteries are not
available until much larger sizes. We
are going to drill our own screw terminals in the lugs of future batteries, and
possibly solder them as well.
The important thing is that the Crossbow IMU survived,
because that costs more than everything else put together, and can have an 8
week lead time. I am going to buy a
backup, in case we arent so lucky next time.
Crossbow is now offering (but not shipping yet) an improved fiber optic
gyro IMU with half the drift rate, but they jacked up the price a few thousand
The main tank actually seems to be ok, but we are not going
to trust pressurizing it again.
The fiberglass nose and tail cones were both broken.
The engines casings for the parachute tower still look OK, I
guess they bent away before the body hit them.
The tower was mangled, of course.
The pressure transducer at the top of the tank was broken.
Our aluminum engine frame at the base was bent a fair
One attitude engine broke the jet holder fitting off inside,
but we can probably remove it.
The main engine servo valve had the half inch pipe fitting
permanently bent in it, but we were able to swap that section of the valve with
scrap from a valve broken in a different way, so it seems to have been saved,
but we havent leak checked it yet. The
plastic connectors on the valve were very brittle from the cooking they took on
our hover tests that stuck to the ground, and broke when disassembled. We are going to run Tefzel wire all the way
to the valve motors in the future, instead of using the supplied pigtail
All the plumbing survived, except for the two fittings that
jammed in engines.
All the engines look ok, but we will have to carefully check
that the main engine hasnt bent its inlet connector.
The WinSystems SBC computer seems dead. The memory SIMM was ripped out of the
socket, which also partially detached, and even after reseating everything, it
wont boot. The flash drive still works
fine in another system, which saves me the effort of building a new linux
system from my last backup.
The antenna connector on the Esteem wireless unit is broken,
but the unit looks OK. Taking the case
apart showed that we can save a large amount of electronics area and several
pounds by just mounting the guts and ditching the case.
Both batteries have cracked cases, although neither one
spilled any acid gel in the box.
The fan over the power supplies was wrecked.
The A/D breakout board was smashed by the batteries.
New Vehicle Work
We are going to proceed with the next vehicle design, as if
this test had succeeded, rather than rebuilding an identical vehicle. The major change is to move to four large
engines that are differentially throttled, instead of the single large engine
and four solenoid controlled attitude engines.
This goes back to the control style of our very first lander, and is
motivated by the fact that we are bumping up against vehicle size limits for
being controlled by the thrust we can get from solenoid based attitude engines.
The vehicle will pay much more attention to streamlining,
with the intention of being capable of supersonic flight. The nose will be 10 or 15 degrees, and we
will be using a honeycomb composite constructed box fin arrangement for
stability instead of the tail flare.
There will be no external protrusions or loose cables along the sides. We are going to try a rear parachute
ejection system, with an intentionally crushable top nose section
The propulsion system will have a master cutoff valve, run
by a separate watchdog computer. We
have talked about this for ages, but not yet implemented it. If implemented on the last vehicle, it would
have dropped it from a much lower altitude.
We are going to make many changes in the electronics to
There will be a backup 9600 baud telemetry radio, in
addition to the Esteem 802.11b.
No more solid core wire for DB connectors, move to 22 ga stranded
Tefzel wire. All 18 gauge wire is
already Tefzel, but I had been using solid wire for soldering serial cables,
which is a known poor practice. I am
moving to mil-spec double-crimp terminals for all flight hardware, instead of
the single-crimp industrial terminals we have been using.
Mount all the electronics, except for the inertial unit, on
a vibration isolated board.
New A/D breakout board
The breakout board that WinSystems sells for their A/D board
takes up a lot more space than necessary, and uses bare wire screw terminals
for input, so we are going to replace it with a custom board that is smaller
and takes ring terminals.
16 signal inputs with #6 ring terminals, one ground is
common to all signals measured.
The range is +/- 10V, so we need to cut the main battery
voltage in half before sampling. It is
a toss up if this should be done on the A/D breakout board, or on the power
supply board. There should be a grid of
holes for soldering in random resistors or capacitors to modify signals.
The grounds are common to all the signals, so I think all we
need is a single ground ring terminal that we will run back to the power
The connector going to the A/D board is a 26 pin ribbon
cable with the following pinout:
1: ch0 2: ch8
3: ch1 4: ch9
5: gnd 6: gnd
7: ch2 8: ch10
9: gnd 10: gnd
11: ch3 12: ch11
13: gnd 14: gnd
15: ch4 16: ch12
17: gnd 18: gnd
19: ch5 20: ch13
21: gnd 22: gnd
23: ch6 24: ch14
25: ch7 26: ch15
Trivial microcontroller that watches a continuous signal
from the main computer, and uses a private motor drive to open the master
cutoff valve only when the main computer is healthy.
One optically isolated digital line from the main computer
Private +12v / GND
Two #6 ring terminals to control the master cutoff servo
valve (the main computer will still read the pot feedback of that valve)
Power supply board
Multiple, diode isolated batteries for redundancy, with an
additional port for running on external power
External charging ports for each battery, so the electronics
dont need to be taken out of the vehicle for charging.
Short run from batteries to boards, no in-line power
switch. Use the power pin on the DC/DC
power converters for switch-on. Use
redundant switches to prevent a switch glitch under vibration from turning
Run nothing from the unregulated power supply, except for
the A/D line for current voltage level.
We previously ran a couple things from the unregulated 12v supply, like
the Esteem wireless unit, and the pressure transducer. It is possible we were losing telemetry
momentarily earlier than the computer died, depending on the details of their
Instead of running wires from the power supply board to
jumpered barrier strips for distribution as we previously did, build plenty of
terminals directly onto the power supply board. At a minimum:
Lots of grounds.
Unregulated +12v: Battery A/D line
+5v: computer (two lines)
+5v: 6 motor drive potentiometer feedbacks
+5v: several spares
+6v: laser altimeter
+12v: pressure transducer
+12v: Panel-PC LCD display
+12v: Several spares
-12v: Panel-PC LCD display
+15v: Crossbow IMU
We might want to use a higher voltage for the IMU, as the
range is 15v-30v, and we have been warned by someone about running avionics at
their minimum recommended voltages. Todays
result seem to corroborate that it is closer to going out than the rest of the
Current draw signal for telemetry? If we ever have a short somewhere, this would be helpful in
Isolated voltage signals for each battery? If we dont have that, telling when a
battery has failed will be difficult.
Our current solid state relay board still has bare wire
terminals (although they are high quality ones that havent yet given problems),
it still has the old power supply on it that we dont use, and one bit on the
input connector is flaky, so it needs to be replaced.
Isolated voltage signal for A/D telemetry?
Isolated continuity checks for each actuator? The motor valves can be self-tested by
watching the potentiometer feedback, but solenoids and pyro would need a
low-current test signal. The actuator
battery needs to be completely isolated from the main battery to avoid noise
problems, so a continuity sensor would need to be isolated as well.
We have known needs for up to six solid state relays and six
motor drives, so building for eight and eight is probably good planning.