shmem above 4gig

[maschoen]

Shared memory above 4Gig.

Recently I’ve found that a QNX 6.6 system with 16 Gig seems to have an odd limitation. I can run programs above the 4Gig limit by starting a program

on -P pgm

However such a program can only create shared memory segments with memory below the 4Gig limit. I don’t understand why this should be. Going to QNX 7 is not an option for my customer.

Has anyone else come across this? Is there a way to make it work?

[Tim]

Presumably they are using the 64 bit calls for opening/mapping shared memory correct? Or are you trying a small sample program and also failing.

Tim

[maschoen]

@Tim

Presumably they are using the 64 bit calls for opening/mapping shared memory correct? Or are you trying a small sample program and also failing.

Tim

No 64 bit calls. No reason to. QNX 6.6 is using a 32 bit flat model and that’s adequate for what we want to do. The problem is just as I stated. When allocating shared memory segments, if the system runs out of room below 4Gig the allocation fails.

The program I’m running is intended to check all memory at startup. This is how it works.

1. Create shared memory segments of a reasonable size, say 2Gig until all memory is used.
2. One by one mmap in a segment, check the memory and un-map it.
3. Release all the segments.

This runs fine but stops creating shared memory segments around 2-3Gig.

I have a work around. I could create separate processes that each allocate a large chunk of memory using malloc(), say 3.5 Gig and check the memory. I would have to synchronize them so that after checking, the program stays in memory until memory runs out and all the memory has been checked.

I’m asking because if it were just a matter of setting some bit somewhere, my work would be done. I don’t know of any logical reason why QNX would restrict shared memory to the bottom 4Gig.

[Tim]

Maybe I’m a bit confused here.

How do you ‘see’ 16 gigs of RAM in a 32 bit O/S (4 gig limit)?

We’ve recently started using QNX 7.0 but only in 32 bit mode at the moment (to save on recompilation issues in 64 bit O/S land). On a machine with 8 gigs of RAM installed QNX 7 32 bit is only reporting 2 via ‘pidin in’. I can imagine via some flag to proc32 that it might be possible to report 4 but I don’t see how it could report 8.

Also doesn’t the CPU itself matter? If it’s only a 32 bit CPU the MMU would be limited to 4 gigs. You didn’t mention the CPU being used but I figured I might as well ask in case it’s a hardware limitation.

Still curious to know how you expect to see 16 gigs in 32 bit land.

Tim

[Tim]

Hmm, googling around I’ve found these mentions of stuff about mmap and more than 4 gigs

memory=type1[:type2[:type3[:type4]]]
Specify the typed memory pool or pools to use. For example, memory=sysram&below4G:sysram says to try sysram with the below4G modifier, and if no such region exists, then try plain sysram. (The same option works on systems with more or less than 4 GB of RAM.)

    It's up to the startup to set up typed memory. Use pidin syspage=asinfo to see the list.
    Generally you don't need to specify the memory option, in which case io-blk.so uses the normal mmap() pool; but on a system with more than 4 GB of RAM, it's mandatory.
    You might have to quote this option, in order to prevent the shell from interpreting special characters such as an ampersand (&).

For more information about typed memory, see "Typed memory" in the Interprocess Communication (IPC) chapter of the System Architecture guide.

qnx.com/developers/docs/6.6. … lk.so.html

The mmap() and mmap64() functions map a region within the object specified by filedes, beginning at off and continuing for len, into the caller’s address space and returns the location. The mmap64() function is a large-file support version of mmap().
In QNX Neutrino 6.6 or later, the large-file support functions and data types appear in the name space only if you define _LARGEFILE64_SOURCE when you compile your code. For more information, see “Classification” in What’s in a Function Description?

The object that you map from can be one of the following:

a file, opened with open()
a shared memory object, opened with shm_open()
[b]a typed memory object, opened with posix_typed_mem_open()[/b]
physical memory — specify NOFD for fildes

qnx.com/developers/docs/6.6. … /mmap.html

qnx.com/developers/docs/6.6. … emory.html

qnx.com/developers/docs/6.6. … _open.html

Seems like mmap by default uses sysram (lower 4 gigs) and not ram (all of RAM) So maybe you need to do some posix_types_mem_open calls to establish names for the other 4 gig blocks of memory beyond the default sysram. Which you can then mmap and verify are OK?

I’ve never done any of this before but it seems like you can do it via those posix calls. Definitely curious to know if you get this to work.

Tim

[maschoen]

[quote=“Tim”]
Maybe I’m a bit confused here.

How do you ‘see’ 16 gigs of RAM in a 32 bit O/S (4 gig limit)?

The 16gigs is real memory. I only see 4gig virtual at a time. Where that is depends on the GDT or LDT tables.

There were two things we needed to do to run a program in the memory above 4gig.

1. startup_apic needed a parameter, I think -x to look for the extra memory.
2. There’s a flag one needs to use when spawning a program to allow it to use this extra memory. Running a program this way has the same affect

on -P mypgm

Once running the program doesn’t know where it’s real memory is (unless it looks hard). I think the reason for this flag requirement has to do with hardware drivers that could not deal with memory above 4gig. This is actually backwards for the way it should be implemented. A program that needs memory below 4gig should have to set a flag, not the other way around.

We’ve recently started using QNX 7.0 but only in 32 bit mode at the moment (to save on recompilation issues in 64 bit O/S land). On a machine with 8 gigs of RAM installed QNX 7 32 bit is only reporting 2 via ‘pidin in’. I can imagine via some flag to proc32 that it might be possible to report 4 but I don’t see how it could report 8.

Check out startup_apic.

Also doesn’t the CPU itself matter? If it’s only a 32 bit CPU the MMU would be limited to 4 gigs. You didn’t mention the CPU being used but I figured I might as well ask in case it’s a hardware limitation.

I’m not entirely sure about this. A Pentium is probably limited to 4gig. Even a 64 bit processor needs the extra hardware addressing lines to work. The processor we are using is 64bit i7 so no problem there.

Still curious to know how you expect to see 16 gigs in 32 bit land.

Simple, I don’t expect to see more than 4 at a time.

[maschoen]

@Tim

Seems like mmap by default uses sysram (lower 4 gigs) and not ram (all of RAM) So maybe you need to do some posix_types_mem_open calls to establish names for the other 4 gig blocks of memory beyond the default sysram. Which you can then mmap and verify are OK?
Tim

mmap is Posix. I’m not sure if I need mmap or mmap64(). Doesn’t really matter. It’s ftruncate that fails. You create a shared memory segment using shm_open() and then you allocate it using ftruncate().