The easiest way to set up the high priority shell is to completely ignore the direct graphical interface and use either a serial port connection or a network connection.
If you can, use the serial port since in that case you can boost the priority of the serial port driver (devc-8250 I’m guessing, otherwise devc-*) up before you get into your lock up situation.
slay -P 200 devc-ser8250
If you can’t use the serial port, then I find boosting the network interface driver (io-net/io-pkt) and inetd to be the next easiest.
slay -P 200 io-net
slay -P 200 inetd
Now open a serial port or telnet in and check the priority of your connection. Assuming all went well when you do a pidin you should see your shell listed with a high priority.
Now run your stuff … if you find that things lock up and your shell is still hosed, then your issue is more serious than a runaway process running READY. If you system is locked up from the UI, but the shell is still responsive, then pidin (in the high priority shell) will give you and indication of what is going on … post the results here if you need help interpreting them.
If things are locked up and you don’t have a shell, then you might give the instrumented kernel tracing a go assuming you have some location where you can persistantly store the data. Take a look at the options for tracelogger and then start tracelogger as a high priority task. Assuming you manage to get a log file, you can use the System Profiler in Momentics to look at the results.