Hi…
I want to thank every body that contributed to this thread. I compiled
everybody’s contribution into a lump article. I did not specify who
mentioned what, but the contributor’s name are below. Please feel free
to correct what is henceforth written including your name and
occupation, and if a contributor does not want the name displayed,
please let me know. Again, I will want to circle this article around my
colleagues, and if you do not wish to have your name displayed for
whichever reason, please do let me know.
Comments and corrections are welcomed, sought after, necessary, and
encouraged. Thanks. 
Bests…
Miguel.
\
\
- Contrary to what most people think, Neutrino code is easier to
maintain than Linux, the reason being that Neutrino is itself the RTOS
Kernel which has one single developer: QSSL.
-Linus (Linux creator) essentially manages and keeps veto privileges
over the Linux kernel.
With regard to this Linux kernel, between 1.0, 2.0, 2.2 and 2.4 the
driver interfaces changed. Also, MAJOR subsystems changed as well. The
reality is that the Linux kernel is done by people who do things because
it is fun, not because a customer needs it. As a case in point, take a
look at the packet filtering interfaces under Linux. Between every
major revision that has had such capabilities (2.0, 2.2, and 2.4) it has
totally changed. The c library is another case-in-point. Unless you
are following the kernel closely you will have issues. I would say that
Linus makes sure that the kernel doesn’t suck, but he has said many
times that he is willing to change interfaces as needed and will not
provide binary backward compatibility when he does allow such changes.
Regardless, some may argue that Linux is generally pretty consistent
within major revisions. Either way, QNX can stand on it’s own merits
without needing to look too hard at Linux, WinCE, etc. It is difficult
to not choose QNX when looking at technical aspects of the OS.
-When compared to embedding Linux, it’s much, much easier to get QNX up
and running on your target device because of the tools and support that
QSSL provides. Linux community is very large indeed, but it has no
definite source of support. Tools for driver creation in Linux are non
existent, this in sharp contrast to Driver Development Kits (DDK) freely
available for Neurino/QNX 6.
2. A fundamental difference between Neutrino and Linux is at the kernel
and driver level as it pertains to driver development. In Linux, the
drivers are intrinsically intertwined with the kernel, that is, drivers
in Linux run inside the kernel space. A direct consequence of this is
that if you develop a new driver in Linux with no previous support for
the desired functionality already set up in the kernel, then you have to
compile the driver along with the complete kernel with the associated
room for error. In contrast, Neutrino is actually the Kernel (which no
one ever compiles), and any developed drivers are compiled completely
independent of the Kernel. This results in a stable, unchanging and
dependable kernel, and drivers that operate as another processes as far
as the kernel is concerned.
- Indeed Linux device drivers can be compiled and inserted/removed
into/from a running kernel without recompiling the kernel or rebooting,
but only if you already have support for that subsystem in place. For
example, if you didn’t build SCSI support into your first kernel you
CANNOT insert a SCSI driver module at some point in the future. Same
for sound, isdn, usb, etc. Whence, in Linux, if the driver is properly
set up, then it may not be necessary to recompile the kernel. For
example, if the developer would know that the project needs a scsi
driver, then the developer would set up the Linux kernel appropriately.
However, a major tradeoff is, therefore, that in Linux (embedded or
otherwise) if you want support for a certain device or kernel feature,
you must include this support in the kernel whether you subsequently use
the added kernel feature or not. This is a big deal for embedded systems
in general. Developers don’t want to clutter their system with things
they aren’t using but when developers want to add something new they
don’t want to have to worry about changing the primary point of
stability associated with the kernel. QNX microkernel architecture takes
care of this point by keeping a stable and reliable kernel outside the
developers hand while keeping driver development outside the kernel
space and in user space. That is, QNX drivers run as regular processes
and are easy to start/stop. Moreover, developers can debug the drivers
using ordinary tools such as gdb rather than using the kernel debugger
over a serial port like in Linux.
- In the final analysis, writing/debugging drivers will have to be done
regardless, and doing this from a user space is by far a better option
as opposed to doing the same thing from the kernel space. This is a
fundamental point. Thus we could argue that a perceived lack of wide
driver availability for QNX is not as important as the fact that
available Linux drivers do operate in the kernel space, and that the
later is the greater of the two evils.
Thus a fundamental point is that driver development under QNX/Nto is
going to be a lot easier and quicker to do because you are just writing
another standard C/C++ program, using the same debugger and toolset as
your non-driver code. Since the driver is a standard process you will
tend to find that you have to reboot to OS far less often which greatly
speed up the Code, Compile, debug (!reboot) cycle.
In addition, one important thing that people miss is, in QNX it is not
just debugging easiness. Since in QNX drivers can be mostly/completely
user space programs they don’t have any restrictions typically being
imposed on kernel space code. So it is actually easier to design and
code.
- Another reality is development/testing cycle. Those can be very long
and changing any major part of product would require full re-testing to
ensure quality, at least in companies following SEI standards. In our
case it means new stuff must pass 3 different testing groups (developers
tests, QA tests, customer on-site tests) and it takes long time because
each group of tests has to simulate high load during long enough time.
That means we must submit new stuff for testing few months before
deadline and we can’t resubmit it every week as problems arise since
testers time/capacity is limited and every version would have to be
tested from scratch… Then if we miss delivery dates or stuff breaks
after testing there are finance charges… So, nobody wants to even
think about introducing major disturbances like new kernel versions.
In that reality it is very nice to have a system where kernel hardly
ever changes, no matter how much new functionality you add. Look at USB
support - QNX added it without even touching kernel/process manager.
Most other Oses required major kernel surgery for that. Kernel
stability by itself is a major plus when choosing QNX over any other
embedded OS, real time or otherwise.
- In conclusion, the existing Linux (Win32) drivers are not useful from
the real time side not so much because they are in the kernel but
because they are in a monolithic kernel, therefore, they inherit its
poor worst-case performance. Desktop OSs are optimized for average case
performance; by stuffing more and more into the kernel. To get good
worst-cast performance, take everything you can out of the kernel; that
is, make it a micro-kernel (or exec). And, of course, make what little
remains pre-emptable.
CASE STUDY EXAMPLE: “When I was working on one servo controller for a
class 3 medical device, every function within the same address space had
to be “stub tested” meaning that each function had to be called with
each parameter in three states; either “in range”, “out of range high”,
or “out of range low”. For each test, the entire address space was
automatically scanned for changes, and changed addresses automatically
translated into symbols. We then had to produce documentation written
and signed by an engineer explaining why each symbol changed for each
test case, and certifying that this was expected behavior. We had 750
functions in one address space with an average of three parameters per
function (750x3x3 = 6750 tests - the documentation for each test was 2-3
pages - each time the code within the address space was modified). If
we were able to use QNX on the servo controller we would only need to
have done this for each process that changed (it would have had a
significant impact in reducing work).”
-
Linux is very reliable, but Neutrino based code is more so for the
following reason: In Linux if a driver dies, so does the kernel => end
of story ==>> you need to reboot the system. In Neutrino if a driver
dies, the kernel continues on, and another process can restart the dead
driver => inherent robustness ==> never need to reboot the system.
-
Perhaps a most compelling argument for use of Neutrino is that Linux
and Neutrino are both POSIX compliant, which means that code developed
for Linux is 90% compatible with code generated for Neutrino. The
difference is, of course, that Neutrino is a true, pre-emptable hard
real-time OS which Linux is not.
-
Neutrino is a true RTOS that can be used as a soft real-time OS if
need be. Linux can be used as a soft real-time OS which can never
be used as a hard RTOS. Using Neutrino in embedded applications just
makes sense because Neutrino was designed for embedded applications from
its birth.
-
In QNX, if you need real-time, you are programming to the posix layer
directly. Some of the real-time Linux extensions involve a real-time
executive which runs the Linux kernel as a process. This means that for
real-time task, you’re programming towards the executive rather than the
posix, and possibly running your real-time process in kernel space
rather than in protected mode.
-
Regular Linux and its drivers are not adequate for hard real time; the
Linux community clearly confirms this fact by the existence of RTLinux.
No matter how plentiful, RTLinux drivers are on the wrong side of the
dual OS setup. That is, they are essentially useless.
NOTE: Dual OS means that there are two kernels, a realtime OS (or exec)
in back running the desktop OS/kernel in front as a task. Thus
determinism is gained by subjugating the desktop OS to the RT OS (or
exec). This dual nature means that the learning curve advantage of
Linux for programmers, the familiar POSIX API, is lost because the back
kernel presents a radically different API. Ditto Win32 API for dual OS
approaches with Microsoft OS. At the opposite side of the OS spectrum,
in QNX there is a single micro-kernel upon which all OS services are
built, that is, real time, human interface…
Linux et. all. drivers are in the regular Linux kernel, not directly
accessible from the RT exec. There are variations of pipes that can
communicate back to front and front to back but the undeterminism of the
desktop OS remains. Moreover, real-world applications of RTOS involve
interfacing to oddball hardware (mine, missile, robots, and the like).
A significant virtue of an RTOS is ease of writing/debugging drivers.
Linux (or any Unix) offers considerable improvement over Microsoft OS in
this respect but is miles behind what can be done with a micro-kernel
such as QNX. Drivers in user space are hard to beat; the micro-kernel
driver approach is orders of magnitude better than monolithic driver
development.
- The dual OS approach of RTLinux (or Microsoft OSs) is now, once again,
a hot topic. This is the third time around for this idea. In the late
70s, Hans Lachlama at Bell Labs first tried this approach with a RT
kernel and Unix as a task under it. Within a couple of years, the
approach was found wanting and abandoned in favor of Plan 9. Again, in
the late 80s it surfaced and again was eventually abandoned. It seems
unlikely to me that this third time around will be charmed. Rather,
this idea resurfaces periodically whenever the previous crop of
technical people move on to management (thus attaining nearly complete
memory loss) and a new crop, lacking corporate memory, arises. In a
way, one could argue that dual OS approach is to software as gallium
arsinide is to hardware: “It’s the wave of the future; always has been
and always will be.”
As a further critique of the dual OS, applying more to those with
Microsoft OS, consider this. Either the desktop OS is critical to
operation of the real time system or it’s not. If it is, e.g., drivers,
then how can its lack of determinism be reconciled with real time
deadlines? If not, why have it? If (when, in the Microsoft case) it
blue screens, the entire real time system must be rebooted. If you want
the familiar look/feel or whatever, set the desktop OS up on another box
communicating by ethernet or the like. Then write your communications
software to provide a “circuit breaker” against indeterminism. Boxes
are cheap; failures of real time systems are not. NOTE: Venturecom and
others are trying the same architectural (dual OS) approach with
Micosoft Windows {NT|2000} as used for RTLinux.
- Basically, with that dual approach you are stuck with a primitive
executive. That won’t ever change because there’s no point extending its
functionality beyond some level, or why would you need Linux in the
first place.
NOTE: there are somewhat different ‘dual’ approaches. Look at L4-based
systems for example. They are pretty much real time microkernels running
Linux as a ‘server’. Difference is, they were designed as real time
microkernels in the first place and only then started to run Linux on
top, whereas RTLinux and RTAI are just hacked Linux. So they are less
primitive but still not quite full POSIX.
Full POSIX+realtime is very hard thing to do because POSIX demands some
functionality which really takes LOT of efforts/experience to implement
without screwing determinism totally. Notice how that author suggests
that may be putting ‘preemption points’ into kernel would make it more
deterministic. What a great idea. QNX kernel is fully preemptable, all
system calls can be preempted at any point even in a middle of message
passing. Interrupts are disabled only for few opcodes and can actually
be nested. Interrupt handlers are very short because most of work is
done by drivers on user level. Actually you can have no interrupt
handlers at all under QNX6 (by attaching events instead of functions).
NOTE: If appropriate for embedded systems, some classic Unix systems,
notably SunOS5 allow you to do what’s written above too although with
larger (but still more or less deterministic) latencies and they have
real time schedulers as well. So, if I wanted just soft real time with a
mainline OS in not-so-constrained environment I’d use Solaris rather
than Linux/RTLinux/RTAI.
NOTE: For more insights you might want to read the famous ‘Linux is
obsolete’ thread, which is discussion between Linus and his teacher (I
believe Tannenbaum is his name) who wrote Minix (long before). In short,
the teacher was rather disappointed by stone-age monolithic design of
Linux kernel and went as far as saying that Linus would not pass his
exams with work like that…
-
QNX is a LOT smaller which translates into big savings on a per unit
basis. It may cost more to use QNX for development but if you can halve
your hardware requirements on a million units…
-
On choosing Linux vs. QNX.
If the developers have previous Linux experience, sure it matters. If
they don’t then it is ‘flock effect’. Everyone likes to stay with
‘mainline’ and follow the flock. Such approach has many non-technical
advantages. Nobody will blame you for your choice because it is
‘everyone’s choice’. You don’t have to prove or ‘sell’ anything to your
management - magazines do that. Magazines also keep reminding you how
great your choice is, for people with lack of self-confidence it is
important just like for women to keep hearing how loved they are.
Finally, if the flock goes wrong way then it is flock to blame, not you
personally.
-
Some may argue that Linux problems with respect to real time are
generic OS problems (which is not true). QNX itself is the best proof.
-
At the root of it all, all of the things that are “different” about
QNX are there to support real-time, and fault tolerance, and that this
is what needs to be learned whether using RTLinux or QNX (it’s just
easier with QNX since real-time/fault tolerance is designed in).
- QNX 2 → 6 are remarkable for it’s elegance and simplicity. Look at
the kernel calls. There are maybe 50 calls - most directly mapping to
POSIX calls and 10 or so are message passing related. In Neutrino there
are 4 orthogonal services:
- message passing
- synchronization
- scheduling
- timer
These services are sufficient to build fully distributed hard real-time
applications on. It’s small well debugged and has to be looked at as a
software component that yes we have to pay for. However this component
should be a lot easier to learn and master than other UNIX variants or
the mess of API’s that MS provides. I would think that this power and
elegance would attract software engineer types once they see it. I mean
who wants to work with that massive old architecture that LINUX uses …
I wonder what fraction of Linux programmers actually look at even a
small fraction of the code.
It is true that complexity appears to have increased a bit in the 2 → 4
→ 6 evolution, but the kernel has gotten more focused and the calls are
more powerful. They are a lot simpler than competing systems especially
when you have to add a second OS layer to provide real-time services.
The addition of threading to QNX6/Nto had a big impact on message
passing with the addition of the thread id and channel concept. Earlier
versions hid this with by transparently mapping a node/process id pair
into a virtual circuit (VC). But the channel is very similar to the VC
and a lot more flexible due to threading support.
The QNX6 resource manager is a change from QNX 2 & 4, but more powerful
and fits nicely into the POSIX way of doing things.
- In the ‘Embedded Systems’ magazine there is an article ‘Real-Time
Linux’ by Alex Ivchenko
(http://www.embedded.com/story/OEG20010418S0044). This quote shows why
Linux et. all are on the wrong (indeterminate) side of the dual OS
setup. The quote itself is important as it comes from a strong Linux
advocate. There is this portion:
-
quote -
“If you’re writing a device driver instead of working with one that
ships
with the data-acq hardware, it’s also important to realize that a
real-time
Linux driver must follow more strict limitations then a Linux driver.
For
instance, when a real-time task makes a call, the driver shouldn’t do
any of
the following:
a… Allocate or free memory
b… Use printk() or similar I/O routines
c… Make blocking calls
d… Copy data to or from the user space or somehow invoke the VMM
(virtual
memory manager)
e… Use spinlocks or other synchronization objects between real-time
code
and Linux kernel
Obviously, the behavior of a real-time driver should be deterministic.
Recall that real-time Linux treats the Linux kernel as the lowest
priority
task. When your real-time task receives control, Linux itself can be in
an
unpredictable state such as in the middle of a spinlock or interrupt
handler. Thus, you can’t rely on Linux services.”
-
end of quote -
Nice indeed. What you can do then except for banging ports directly?
Sounds like you need to do everything inside interrupt handler. You
might be on time with first interrupt but what if they come too fast for
your handler to complete? Probably the developer disabled all the
interrupts while servicing the one interrupt… In any case this is
unacceptable.
As previously said, with that dual approach you are stuck with a
primitive executive. That won’t ever change because there’s no point
extending its functionality beyond some level, or why would you need
Linux in the first place.
Authors:
Rennie Allen, ??
Dean Douthat, ??
David L. Hawley, ??, (D.L. Hawley and Associates)
Igor Kovalenko, embedded systems engineer??, (Motorola)
Chris McKillop, software engineer (QSSL)
Miguel Simon, research engineer, (SAIC)
Kris Warkentin, software engineer (QSSL)
\
\
my opinions are mine, only mine, solely mine, and they are not related
in any possible way to the institution(s) in which I study and work.
Miguel Simon
Research Engineer
School of Aerospace and Mechanical Engineering
University of Oklahoma
http://www.amerobotics.ou.edu/
http://www.saic.com
