Remote File Access With Emacs

Andrew Thomas <andrew@cogent.ca> wrote in message
news:oprth4dznow9rgrq@nntp.qnx.com

Interestingly, the very many operating systems supported by emacs either
provide a system allocator that does maintain contiguous virtual memory
or provide a functioning sbrk() that allows gmalloc to work. While I
agree
with you in principle, I don’t see much evidence for it in practise.

Partly because they have the luxury of doing so - if they actually need the
physical memory backing the virtual address space, they can swap it out and
the next reference will fault it back in on demand - we not going to do that

This is the gmalloc route. It fails. I cannot remember how, frankly,
but as I said, it has to do with sbrk() not really obeying its own
documentation.

Well, actually you can do a hunk allocator without sbrk - just mmap() in
hunks of memory and provide a slab like object allocator from that. The
fact you do your own allocator already means you have the memory map, and
can do away with sbrk all together.

Generally speaking, I am willing to put in a few days of work to port
emacs to QNX6, but I am not willing to totally rewrite its memory
allocators or ELF dumping code. I was rather hoping that QNX would
try to support emacs, at least passively - it’s one of the most popular
GNU
tools.

Well, putting features into the allocator to get some desktop domain program
ported isn’t what we want to target IMHO. In this narrow case, we can put a
feature in to control the rate of release back to the system (the
pathalogical case being never) which can help the port. But if the choice
has to come between a ‘traditionally conforming’ sbrk() and a more efficient
allocator - the choice isn’t hard.

It is unfortunately also a pretty severe test case for
“compatibility”, where compatibility is defined as
“conformance to a legacy assortment of random side-effects”. We don’t
have
to like it, but it’s the world we live in.

I don’t debate that - but then one needs to understand that one day those
‘side-effects’ won’t be around.

FYI - thanks for all you input Andrew, much appreciated!

-Adam

On Wed, 6 Aug 2003 18:01:46 -0400, Adam Mallory <amallory@qnx.com> wrote:

Andrew Thomas <> andrew@cogent.ca> > wrote in message
news:> oprth4dznow9rgrq@nntp.qnx.com> …

Interestingly, the very many operating systems supported by emacs
either provide a system allocator that does maintain contiguous virtual
memory
or provide a functioning sbrk() that allows gmalloc to work. While I
agree with you in principle, I don’t see much evidence for it in
practise.

Partly because they have the luxury of doing so - if they actually need
the physical memory backing the virtual address space, they can swap it
out and the next reference will fault it back in on demand - we not going
to do that.

I don’t understand the correlation between the last two paragraphs.
I didn’t think that contiguous virtual memory in any way implied a
swap implementation or a relationship with physical memory.

Well, actually you can do a hunk allocator without sbrk - just mmap() in
hunks of memory and provide a slab like object allocator from that. The
fact you do your own allocator already means you have the memory map, and
can do away with sbrk all together.

I don’t do my own allocator. I’m configuring emacs to use one
of the many memory allocation schemes that it has available to it.
It requires one of two possibilities:
a) a functioning sbrk() that can be used in a malloc() replacment
that all library functions use to the exclusion of other memory
allocation techniques. QNX seems to fail here by not respecting
malloc() replacements. Many people have asked for this.
b) a memory allocator that can be optionally made to maintain
contiguous virtual memory, and a way to determine the beginning
and end of that virtual memory.
These are not ridiculous requirements.

Well, putting features into the allocator to get some desktop domain
program ported isn’t what we want to target IMHO. In this narrow case,
we can put a feature in to control the rate of release back to the system
(the
pathalogical case being never) which can help the port. But if the
choice has to come between a ‘traditionally conforming’ sbrk() and a more
efficient allocator - the choice isn’t hard.

This is more than just emacs. Like I said, Common LISP environments,
some scheme and smalltalk environments, and many C/C++ garbage collectors
require similar capabilities.

In any case, this is not a choice between ‘traditionally conforming’ and
‘efficient’. Both are possible simultaneously. If QNX actually allowed
drop-in replacments for malloc() then all that would be required is to
fix sbrk(), and the job is done. If you use sbrk() to grab system memory,
you can have a ‘traditionally conforming’ allocator. If you do not, then
you can have an arbitrarily efficient one.

QNX publishes a library with sbrk() in it. This function does something,
just not a correct thing. You could end this line of questioning by
removing sbrk() from the library, but as long as you continue to ship it,
I am just hoping you will also make it work.

It is unfortunately also a pretty severe test case for
“compatibility”, where compatibility is defined as
“conformance to a legacy assortment of random side-effects”. We don’t
have to like it, but it’s the world we live in.

I don’t debate that - but then one needs to understand that one day those
‘side-effects’ won’t be around.

Not in the lifetime of Linux, is my guess. As long as you want to claim
that QNX is compatible with existing UNIX software, supporting some of
those side-effects is a necessity. This particular side-effect is just
one of the harder ones. And you nearly have it. Why not take the
extra baby step here?

Cheers,
Andrew

Andrew Thomas <andrew@cogent.ca> wrote in message
news:oprtljcvj2w9rgrq@nntp.qnx.com

I don’t understand the correlation between the last two paragraphs.
I didn’t think that contiguous virtual memory in any way implied a
swap implementation or a relationship with physical memory.

The fact that if you have a virtually contiguous heap, in QNX also means you
have physical backing store for it (ie. real memory). So if we avoid using
‘real’ memory by releasing it back to the system, then we do so. Other OS’s
(FreeBSD, Linux etc) have the luxury of not doing that (ie releasing
directly back via munmap()) because if the system needs more ‘real’ memory,
it’s possible to just take away the backingstore, while still leaving the
heap virtually contiguous (although nothing is behind it).

I don’t do my own allocator. I’m configuring emacs to use one
of the many memory allocation schemes that it has available to it.
It requires one of two possibilities:
a) a functioning sbrk() that can be used in a malloc() replacment
that all library functions use to the exclusion of other memory
allocation techniques. QNX seems to fail here by not respecting
malloc() replacements. Many people have asked for this.
b) a memory allocator that can be optionally made to maintain
contiguous virtual memory, and a way to determine the beginning
and end of that virtual memory.
These are not ridiculous requirements.

No one said they were ridiculous. When you have fixed resources you must
prioritize where you spend them - as such getting an emacs port isn’t high
on the list. I’m not suggesting that you do the implementation of the
allocator, I just think emacs could stand to be a tad more portable.

This is more than just emacs. Like I said, Common LISP environments,
some scheme and smalltalk environments, and many C/C++ garbage collectors
require similar capabilities.

The key word is ‘some’ - ‘some’ software is written in less portable
manners, that’s life too. And I agree that we should do what we can to ease
the pain, no agruments there.

In any case, this is not a choice between ‘traditionally conforming’ and
‘efficient’. Both are possible simultaneously. If QNX actually allowed
drop-in replacments for malloc() then all that would be required is to
fix sbrk(), and the job is done. If you use sbrk() to grab system memory,
you can have a ‘traditionally conforming’ allocator. If you do not, then
you can have an arbitrarily efficient one.

I’m not sure why you say we don’t support ‘drop in replacement’, you can use
any allocator you like. Just ensure that symbols for malloc/free etc
resolve to your allocator first, then resolve against libc.

QNX publishes a library with sbrk() in it. This function does
something,
just not a correct thing. You could end this line of questioning by
removing sbrk() from the library, but as long as you continue to ship it,
I am just hoping you will also make it work.

Well, I’ll impose on you once more to tell me what exactly you find broken
and I’ll do my best to see what can be done about it.

Not in the lifetime of Linux, is my guess. As long as you want to claim
that QNX is compatible with existing UNIX software, supporting some of
those side-effects is a necessity. This particular side-effect is just
one of the harder ones. And you nearly have it. Why not take the
extra baby step here?

I agree that we’ll do what we can, and I don’t recall anyone saying ‘no’.

Cheers,
-Adam