Qnet and realtime


Our company is looking to build a distributed real time embedded control system, and as we are at the very beginning of development, we are looking for an operating system that suits our needs. Among many other reasons, one particular thing that drew our attention toward QNX is Qnet, which seems like the thing we are looking for. First of all, I am a complete newbie to QNX, having learned everything I know about it by skimming through parts of the documentation on QNX’s website. My question would be, since Qnet uses a protocol different from TCP/IP, does that mean, that Qnet provides realtime behavior? The Qnet chapter of the guide on qnx’s website, doesn’t mention anything about Qnet’s being realtime. And if not, since, from what I understand, Qnet’s been designed in such a fashion, that most of the communication layers are swappable, is it possible to modify it to be 100% deterministic, by say, exchanging the actual transport protocol, with something suitable(something like RTMAC in the RTnet linux framework)? And would something like this require changes at the driver level?
Sorry for asking so much questions, but we would really like to know what we are up to, and what level of effort would such a thing would require, so we could decide for or against using it.

Yes, it does. Since Qnet simply extends the kernel, priority is conveyed with the message, and priority queuing applies. Priority inheritance even works across the network (all of this just falls out of the design, since Qnet is only extending the kernel itself).

Qnet is deterministic, as long as the transport itself is deterministic.

The driver only needs to be written. For instance Qnet runs over rapid-IO, and since rapid-IO has a deterministic arbitration scheme, it is completely deterministic.

Since Qnet is designed from the ground up to simply extend the QNX kernel across the network, and since the QNX kernel is designed from the ground up as a real-time kernel, Qnet is quite unlike any other kind of networking technology, in that it is inherently deterministic.

And there lies the rub. Interestingly enough, the original QNX network for QNX 2.+ used Arcnet, a protocol that was designed from the ground up to be deterministic. Slow by today’s network speeds, but deterministic. Even the relatively lengthy process of adding a node to the network was spelled out in the specification.

Ethernet, on the other hand was not designed this way. The original implementations of Ethernet had no worst case transport delay. Worse yet, its efficiency had a peak at around 40% of bandwidth, after which additional traffic caused a lowering of bandwidth. Even so, it is not hard to put together an Ethernet network that is effectively real-time. Switches can remove the problem of collision, and so if you know the bandwidth requirements of each node in your system, and have enough bandwidth in the network, you can assume a limited delay. Likewise if you are just using Ethernet as a point to point network, it is very predictable.

I guess I should not finish this without mentioning that a deterministic implementation of Ethernet was invented, I think it was called Anylan, but was not successful in the market and I think is gone. It required specially designed cards, and I think switches as well.

While switched Ethernet is deterministic, it is very difficult to control. In order to determine worst case latency, you need to know the properties of the switch (how it arbitrates simultaneous ingress on multiple ports where each packet is destined for a single egress port - it can’t just “blend” them, so only one of them is not delayed). While it is almost certainly deterministic, since there is no standard, you have to determine this data manually for each switch you intend to deploy.

Yes, quite so. Effectively deterministic really isn’t deterministic, is it now ;-).

I think the universe is deterministic it’s just beyond our grasp.

Einstein is with you on that, but I think Niels Bohr, and Werner Heisenberg might have a bone to pick with you (mind you if they’re right then an Ethernet packet could very well turn into lava lamp, before it gets to the switch :slight_smile:

I dont know the exact source, but during the ‘opening’ process of TDP, they mention in some slides that Qnet can be fine tuned by directly accessing the rapid I/O card. The refer to a development kit, the kit still is not available. I was wondering if it would be hard to have a 2 wired interface (like RS232 or similair) that uses the qnet approach, or USB maybe?

Well I guess I can’t help but jump in when there’s something I know about. First of all, according to quantum mechanics, whenever you measure something, for example an objects position, the result is a pure probability. For macroscopic events, the probability that the object will be a measurable distance from where Newtonian mechanics expects it to be, is 1 + or - a very very very small number. Often such an event has a low probability even given that the measurement is repeated continuously for the lifetime of the universe.

But some still think that there is an underlying mechanism, which implies determinism. This can only be considered a very theoretical idea. Chaos theory puts to rest any idea that this determinism can be of any use. I once heard that if you hit a billiard ball very hard, and it bounces back and forth on a billiard table, and you want to predict it’s ultimate position, that by the time it has bounced 20 or more times, you would need to include the gravitational effects of nearby stars in order to get an accurate answer. I can’t say I’ve seen the math, and this may be an exaggeration, but you get the idea.

My apologese to any purists who would rather I stick to Qnet.

Isn’t it where the problem lies, you cannot “see” something without affecting it and that is what we yet cannot predict. (There is this nice experiment about affecting the path a photon took 1 millions years ago but how you look at it today, very funky).

But that doesn’t mean it’s not deterministic.

Isn’t it where the problem lies, you cannot “see” something without affecting it and that is what we yet cannot predict. (There is this nice experiment about affecting the path a photon took 1 millions years ago but how you look at it today, very funky).

But that doesn’t mean it’s not deterministic.

There are a lot of misnomers about quantum mechanics, and this is one of them. With Newtonian mechanics it is also the case that you can’t detect a moving object without affecting its trajectory. That’s because photons have momentum and energy. You are right that this would in principle be deterministic and predictable.

It isn’t that you can’t see something without affecting it. It’s that what you are looking for, position or velocity, has no reality until you affect it. Quantum mechanics does not give objects a position or velocity in time the way Newtonian mechanics does. Instead an object is completely described by a wave function, which only tells you about the probability of an objects properties. As far was we know, the wave function of an object does not go to zero almost nowhere. You cannot measure a wave function directly. Instead you do an experiment, and the wave function collapses giving you some information, such as a position. But by the time you get this information, it is no longer correct. Even worse, getting this information has affected the object, so it has a new changed wave function and you can’t predict its future properties in time.

This all might seem quite strange. Why is this of any value at all? Well the probability of a macroscopic object being in some location, is closer to its Newtonian derived location than we can measure. So at a macroscopic level, it makes the same predictions, an important feature.

At the microscopic level it explains things that Newtonian mechanics does not. Here’s an example. If you have a point source of light, and you project it through two narrow slits in a barrier, on the other side you see bands of light and dark. That is because after the wave of light hits the barrier, two waves come out of the two slits, and they diffract. If you think of two periodic water waves coming at each other, when they hit, you get larger peaks and troughs, but there are always places where the waves cancel each other in between. This is what the dark bands are.

Now what happens if the light source only sends photons toward the barrier at a very slow rate, say one a second. Each photon is still described by a wave function that splits at the barrier, and recombines on the other side in the same band pattern. So if you have sensitive detectors, you discover the same bands if you wait long enough.

You might well wonder, when a single photon goes through the barrier, which slit does it go through? It must go through one or the other. You could close off one of the slits, and then the pattern of bands would go away. You could keep both slits open, but put some kind of detector near one of the slits to detect the photon as it flies by. But this would change the wave function, and you again would no longer get the bands.

There are some very spooking things too. You might have read about something flippantly called, spooky action at a distance. This again is a bit of a misnomer, but it does describe what we observe. Two particles can have their wave functions connected in a very special way. For example, a hydrogen atom in its ground state, the state with minimal energy, has a proton and an electron whose spin states are in opposite directions. We don’t know what that direction is, but it must be opposite, or the atom would have some extra energy that could be ejected as a photon. It turns out that there is a way to increase the distance between the proton and the electron, without changing the spin states. They remain opposite to each other. There have been such experiments where the two particles have been separated by as much as a mile.

Now, as I’ve mentioned before, the direction of the spin is just a probability, having no exact reality. So until you do an experiment on one of the particles to find out what it is, it has no reality. However, once you do this experiment, you immediately know the exact spin state of other particle. It’s as if the information has jumped between the two particles, no matter how far apart they are. The information jump is immediate, and not limited by the speed of light.

The phenomenon known as entanglement explains the non-local feature of Quantum mechanics. That is, the information about wave function is not contained in some space local to the object. Compare this with something like temperature, which is local to some position.

The source for all of QNet is released, so the TDP TDK is not necessary.

Here is the landing page for Qnet (it is actually nothing more than a tiny module in the network stack)

community.qnx.com/sf/wiki/do/vie … _wiki_page

So to sum it all up: Qnet is not fully deterministic when collisions can occur on the bus, so a raw ethernet network cannot be used. On the other hand, when using a switch, collisions can be avoided. So, that means by inserting the switch between the nodes makes the network determinstic, if the switch does its work deterministically.
But the inclusion of the switch is costly, introduces latency (possibly tens of milliseconds) and besides, what does an embedded want less, than one more component to fail? :slight_smile:
I don’ really know about rapidio, is it used in actual industrial embedded devices (such as for a backplane bus), or is it more like a distributed computing thingy?
The best solution would be to use a low-cost bus that is already available on the most of the computers, in this case, Ethernet, but modify the drivers in such a way that it becomes deterministic(this is what, for example rtnet for linux does). The question is, is there such an effort going on for QNX, or would we have to write (or possibly port) everything?
Another (quite expensive) solution would be to use some existing high-speed fieldbus system, and modifying Qnet to use it. Concerning this, I would be more than happy if you could offer me some guidance (some tutorial, code examples or something) on how to implement a Qnet “medium”, so that we could evaluate, which path is worth taking.

Even when using a switch QNET is not deterministic in real life. I mean yeah maybe the protocol could be, to some degree, but for example to figure out the transmission time of say a perticular message you must now only take into consideration:

  • the work load of the receiver
  • Error on the mediums.
  • the work load of the transmitter
  • The network traffic, yeah a switch can deal with collision but 10 1G port spitting data into 100Mbit will not be good.

In the days of 68000 it was easy to figure out how a program would run, could the number of instruction and number of cycles per instructions and your done. I remember when Psos was around they would specify how much “cycles” each function call would take, now that’s deterministic all right ;-) Today this is impossible to do.

As for answering your question, I think you would have to write a layer between the stack ( QNET ) and the network driver to implement something like that on ethernet.

I don’t know how rtnet does it but I think to get deterministic on ethernet you’d have to send packet at fix interval and have the protocol “ride” into these time slot created by the steady stransmission of these packet.

By the way good switches will have latency far bellow milliseconds. Some will only need the time to receive the source and destination MAC addresses which at 1Gig doesn’t take a long time…

You said “the inclusion of the switch is costly” well if Ethernet is what you want to use you have no other choice but to use a switch ( Hubs are hard to find these days) , unless you plan to connect only two devices together which makes Ethernet deterministic since there won’t be any collision.

Secondo if you go with your own protocol all the machines on the network must use that same protocol otherwise inclusion of other protocol/ethernetpacket will make the network uncontrollable (what if I ping flood your machine). That means your machine will only be able to interface with other machine that have this same protocol, greatly limiting connectivity.

The devil is in the details ;-)

Yes, exactly.

Here is the rapid IO website: rapidio.org/home

From the FAQ:

Q: What is RapidIO technology?

A: The RapidIO architecture is an electronic data communications standard for interconnecting chips on a circuit board and circuit boards using a backplane. The RapidIO architecture is designed to be used for the processor and peripheral interface where bandwidth and low latency are crucial. The RapidIO architecture is partitioned into a three-layer hierarchy of logical, transport, and physical specifications, which allows scalability and future enhancements while maintaining compatibility.

There is nothing that can be done in the driver to make shared bus ethernet deterministic. The exponential backoff algorithm is implemented in hardware and can’t be disabled (when shared bus is turned on, the backoff algorithm is, as well). You’d have to build special hardware that operated a shared bus in half-duplex, but then all you’d have done is substitute an off-the-shelf switch with something of your own design).

Equating rtnet and Qnet verges on absurdity.

For example: in rtnet, there is no priority inheritance over the network which makes vulnerable to priority inversion. I can’t imagine someone practiced in the art of distributed hard real-time would even consider a technology that can’t prevent priority inversion (fundamental tenet of real-time).

FYI, rtnet does not modify the drivers in some magical way that conveys determinism to shared bus ethernet (that it is technically impossible, since the backoff algorithm is implemented in hardware and can’t be altered by software). The reason rtnet provides special drivers, is because they can’t use the standard Linux stack (which is completely non-deterministic), not because they are modifying the driver.

If you feel that rtnet provides sufficient determism, then you can certainly use Qnet with the standard drivers (rtnet provides far less determinism than what you get “out-of-the-box” with Qnet and standard ethernet drivers).

I think it is important to point out, that we may be talking about two entirely different classes of application here. QNX is a true fault tolerant distributed real-time operating system. We are worried about theoretical worst case latencies that might occur once every 10 years, whereas if you are considering rtnet to be a viable option, you can’t possibly be operating at this level of criticality (rtnet would be doing good if it ran for 10 years, let alone guarantee a worst-case latency over that period).

Firstly: The code to provide a “driver” to Qnet is only necessary should you want to implement a custom device (and only if you are seriously concerned about determinism). If OTS ethernet works for you; you don’t need to do a thing.

Secondly, writing a custom driver for Qnet is really easy (since it is not a general purpose driver). Essentially you only need to write a read() function and a write() function. The major complexity is the name resolver (which is the piece of code that maps node name to MAC address - i.e. performs the same function as ARP in tcp/ip).