Getting physical addresses

QNX Newsgroups (Archives) qdn.public.qnxrtp.os
1

I’m writing a Neutrino driver for a PCI card with bus mastering (DMA)
capabilities. The big piece I’m missing is figuring ou the physical
address of a piece of memory.

Most of the time, I get a pointer (virtual address) to a buffer and the
size of the buffer. From this, I need to get a set of physical
addresses, one for the start of each page on which the buffer lives.

There is a special situation where I know that the buffer is physically
contiguous, and therefore only need to get the physical address of the
first page.

The end goal is to take this physical address and give it to the bus
master engine on my PCI card so the card can DMA to or from that buffer.

I’ve been unable to find the routines or other documentation which tells
me how to do this. Any clues or pointers to docs would be appreciated.

And while I’m on the subject, is there some sample Neutrino pci driver
code lying around anywhere? Hey, I can dream, can’t I?

Thanks in advance,
Eric

2

In qdn.public.qnxrtp.os Eric Berdahl <berdahl@serendipity.org> wrote:

I’m writing a Neutrino driver for a PCI card with bus mastering (DMA)
capabilities. The big piece I’m missing is figuring ou the physical
address of a piece of memory.

Most of the time, I get a pointer (virtual address) to a buffer and the
size of the buffer. From this, I need to get a set of physical
addresses, one for the start of each page on which the buffer lives.

A couple steps – you have to allocate DMA safe memory, then get the
address.

Pretty simple:

ptr = mmap( 0, size, PROT_READ|PROT_WRITE|PROT_NOCACHE,
MAP_ANON|MAP_PHYS|MAP_NOX64K, NOFD, 0 );
posix_mem_offset( ptr, nofd, 1 &phys_addr, NULL );

Where phys_addr is a 32-bit value for the physical offset.

-David

3

Hey David, aren’t PCI bus masters immune to the ancient 64k barrier thingie? I
thought that was a limitation of the standard PC DMA controller only, although
I suppose if someone had a weak PCI bus master it might suffer from the same
limitation…

-Warren


“David Gibbs” <dagibbs@qnx.com> wrote in message
news:8tus8d$ale$1@nntp.qnx.com…
| In qdn.public.qnxrtp.os Eric Berdahl <berdahl@serendipity.org> wrote:
| > I’m writing a Neutrino driver for a PCI card with bus mastering (DMA)
| > capabilities. The big piece I’m missing is figuring ou the physical
| > address of a piece of memory.
|
| > Most of the time, I get a pointer (virtual address) to a buffer and the
| > size of the buffer. From this, I need to get a set of physical
| > addresses, one for the start of each page on which the buffer lives.
|
| A couple steps – you have to allocate DMA safe memory, then get the
| address.
|
| Pretty simple:
|
| ptr = mmap( 0, size, PROT_READ|PROT_WRITE|PROT_NOCACHE,
| MAP_ANON|MAP_PHYS|MAP_NOX64K, NOFD, 0 );
| posix_mem_offset( ptr, nofd, 1 &phys_addr, NULL );
|
| Where phys_addr is a 32-bit value for the physical offset.
|
| -David

4

In qdn.public.qnxrtp.os Warren Peece <warren@nospam.com> wrote:

Hey David, aren’t PCI bus masters immune to the ancient 64k barrier thingie? I
thought that was a limitation of the standard PC DMA controller only, although
I suppose if someone had a weak PCI bus master it might suffer from the same
limitation…

I know they’re immune to the <16M address limitation, and I know they can
cross the 64K mark if you ask for more – but I think they still like their
memory 64K aligned – I’m not sure though. That was the suggested incantation
I was given when I asked how to do this.

(And, for ISA, or in MAP_BELOW16M.)

-David


-Warren



“David Gibbs” <> dagibbs@qnx.com> > wrote in message
news:8tus8d$ale$> 1@nntp.qnx.com> …
| In qdn.public.qnxrtp.os Eric Berdahl <> berdahl@serendipity.org> > wrote:
| > I’m writing a Neutrino driver for a PCI card with bus mastering (DMA)
| > capabilities. The big piece I’m missing is figuring ou the physical
| > address of a piece of memory.
|
| > Most of the time, I get a pointer (virtual address) to a buffer and the
| > size of the buffer. From this, I need to get a set of physical
| > addresses, one for the start of each page on which the buffer lives.
|
| A couple steps – you have to allocate DMA safe memory, then get the
| address.
|
| Pretty simple:
|
| ptr = mmap( 0, size, PROT_READ|PROT_WRITE|PROT_NOCACHE,
| MAP_ANON|MAP_PHYS|MAP_NOX64K, NOFD, 0 );
| posix_mem_offset( ptr, nofd, 1 &phys_addr, NULL );
|
| Where phys_addr is a 32-bit value for the physical offset.
|
| -David

5

In article <8tus8d$ale$1@nntp.qnx.com>, David Gibbs <dagibbs@qnx.com>
wrote:

In qdn.public.qnxrtp.os Eric Berdahl <> berdahl@serendipity.org> > wrote:
Most of the time, I get a pointer (virtual address) to a buffer and the
size of the buffer. From this, I need to get a set of physical
addresses, one for the start of each page on which the buffer lives.

A couple steps – you have to allocate DMA safe memory, then get the
address.

Pretty simple:

ptr = mmap( 0, size, PROT_READ|PROT_WRITE|PROT_NOCACHE,
MAP_ANON|MAP_PHYS|MAP_NOX64K, NOFD, 0 );
posix_mem_offset( ptr, nofd, 1 &phys_addr, NULL );

Where phys_addr is a 32-bit value for the physical offset.

I’ll echo Warren’s question. PCI devices should not be subject to the
64k limit, so I wouldn’t expect to have to specify MAP_NOX64K. Am I
mistaken?

I believe posix_mem_offset is the routine for which I was looking. Am I
correct in assuming that the phys_addr returned by posix_mem_offset is
the physical address I give to my PCI-based DMA engine as the target
address (i.e. I don’t have to convert that from a cpu-physical to a
pci-physical address)?

Reading the docs on posix_mem_offset, it seems that I could use the
routine serially on any generic buffer, regardless of its physical
continuity. For example, if I had a physically fragmented buffer and I
wanted to do a scatter-gather DMA operation (assuming I had a DMA engine
that supported such an operation), I could do something like the
following:

while (0 != bufferLen)
{
// Get the physical base address and length of largest
// contiguous block
posix_mem_offset(buffer, bufferLen,
&bufferPhys, &contigLen, NULL);

// add a descriptor to the scatter-gather list for a
// block starting at bufferPhys and continuing for
// contigLen bytes – exercise for the reader

// Advance to the next block
bufferLen -= contigLen;
buffer += contigLen;
}

Do I understand correctly?

Thanks in advance,
Eric

6

In article <8tv47t$fdo$1@nntp.qnx.com>, David Gibbs <dagibbs@qnx.com>
wrote:

In qdn.public.qnxrtp.os Warren Peece <> warren@nospam.com> > wrote:
Hey David, aren’t PCI bus masters immune to the ancient 64k barrier
thingie? I
thought that was a limitation of the standard PC DMA controller only,
although
I suppose if someone had a weak PCI bus master it might suffer from the
same
limitation…

I know they’re immune to the <16M address limitation, and I know they can
cross the 64K mark if you ask for more – but I think they still like
their
memory 64K aligned – I’m not sure though. That was the suggested
incantation
I was given when I asked how to do this.

To be general, 64K alignment is probably not a bad idea. Not all PCI bus
masters have this requirement. It is sometimes a matter of efficiency,
though (but not for my particular hardware :slight_smile:

And 64K size limitations are not really a problem, particularly if
you’re working with multimedia cards who think nothing of spending >1MB
on a video frame.

(And, for ISA, or in MAP_BELOW16M.)

Ah! I was confusing MAP_BELOW16M with MAP_NOX64K.

Thanks for the clarification,
Eric

7

In qdn.public.qnxrtp.os Eric Berdahl <berdahl@intelligentparadigm.com> wrote:

In article <8tus8d$ale$> 1@nntp.qnx.com> >, David Gibbs <> dagibbs@qnx.com
wrote:



I’ll echo Warren’s question. PCI devices should not be subject to the
64k limit, so I wouldn’t expect to have to specify MAP_NOX64K. Am I
mistaken?

See previous response.

I believe posix_mem_offset is the routine for which I was looking. Am I
correct in assuming that the phys_addr returned by posix_mem_offset is
the physical address I give to my PCI-based DMA engine as the target
address (i.e. I don’t have to convert that from a cpu-physical to a
pci-physical address)?

Yes, this is the value you give to the DMA engine. It is a linear
address in physical address space. Both the CPU and the PCI bus should
be using the same addresses for everything, otherwise you’ll have real
problems. (At least, that is my understanding of things.)

Reading the docs on posix_mem_offset, it seems that I could use the
routine serially on any generic buffer, regardless of its physical
continuity. For example, if I had a physically fragmented buffer and I
wanted to do a scatter-gather DMA operation (assuming I had a DMA engine
that supported such an operation), I could do something like the
following:

while (0 != bufferLen)
{
// Get the physical base address and length of largest
// contiguous block
posix_mem_offset(buffer, bufferLen,
&bufferPhys, &contigLen, NULL);

// add a descriptor to the scatter-gather list for a
// block starting at bufferPhys and continuing for
// contigLen bytes – exercise for the reader

// Advance to the next block
bufferLen -= contigLen;
buffer += contigLen;
}

Yes, you should be able to do this. I vaguely remember there being a problem
with the contiguous length reported by posix_mem_offset(), though. I think
it may not have given the largest possible value – that is, somethings that
were actually contiguous didn’t get reported as such.

-David

8

Both the CPU and the PCI bus should
be using the same addresses for everything, otherwise you’ll have real
problems. (At least, that is my understanding of things.)

I think this is only true for standard Intel PC architecture with only one
PCI bus. And it is probably not true on many RISC architectures. For a
detail treatment of this subject, refer to a Windows NT/2000 device driver
book, which has a much more universal view of DMA.

I vaguely remember there being a problem
with the contiguous length reported by posix_mem_offset(), though. I
think
it may not have given the largest possible value – that is, somethings
that
were actually contiguous didn’t get reported as such.

I reported this problem earlier and I have not received a technical answer
yet. First of all, posix_mem_offset() returns “not implemented” in my QRTP
distribution. I have to use the older (no longer documented) mem_offset()
call instead. Secondly, the contiguous length reported is never bigger than
one page, even there are multiple physically contiguous pages. And finally,
the contiguous length reported is not correct (at least not in the way I
interpret the documentation) for the last page (or is it the first page). I
think if the virtual buffer ends in the middle of a physical page, the
contiguous length for the last segment is incorrectly reported as a whole
page. Since I can’t find the official definition of posix_mem_offset() (the
standard is not finalized yet), I can’t really argue what the correct
behavior should be.

-Kim

9

In article <8u9ln7$4pm$1@inn.qnx.com>, “Kim Liu” <kliu@terayon.com>
wrote:

Both the CPU and the PCI bus should
be using the same addresses for everything, otherwise you’ll have real
problems. (At least, that is my understanding of things.)

I think this is only true for standard Intel PC architecture with only
one
PCI bus. And it is probably not true on many RISC architectures. For a
detail treatment of this subject, refer to a Windows NT/2000 device
driver
book, which has a much more universal view of DMA.

If the CPU bus is the PCI bus, then CPU-physical and PCI addresses are
identical by definition. However, most if not all motherboards do not
use PCI for their CPU bus. On those boards, there is a CPU-PCI bridge,
which translates protocol and addresses. In such hardware, the
CPU-physical may or may not be the same as the PCI address.

In Neutrino, you can see the acknowledgement of this in the definition
of the pci_dev_info structure in hw/pci.h. In that structure, you’ll see
fields called, for example, CpuBaseAddress and PciBaseAddress. These
fields are the addresses of the same item on the PCI card, one from the
perspective of the CPU-physical bus and one from the perspective of the
PCI bus.