Notification when filesystem gets modified.

QNX Newsgroups (Archives) qnx.qnx4
1

Hi developers,

Is there a way for a process to somehow get a message when an object (and
also which object) in the filesystem (Fsys’s namespace) gets modified?

Background:

I wrote an application which mirrors a directory tree via FTP to a remote
machine. The approach I use is to periodically scan the whole local tree
to see if anything was modified and keep a record about the objects. But
as you know an interrupt driven approach is usually more efficient than
polling. So it’s better to get a message saying that a certain object just
finished getting modified and that it can be mirrored immediately.

One way to do this is to create a resource manager with it’s own namespace
that maps onto the directory tree. However if the object gets modified
outside of the resource manager’s namespace, it won’t know about it.

So I’m looking for other options also. Is there perhaps a way to somehow
interface with Fsys? All the daemon needs to know is what object got
modified and that this modification is complete.

Thank you for any tips you might have.

regards,
rick

2

“Rick Lake” <ow@private-domain.nl> wrote in message
news:cko80p$cps$1@inn.qnx.com…

Hi developers,

Is there a way for a process to somehow get a message when an object (and
also which object) in the filesystem (Fsys’s namespace) gets modified?

No,

I asked for this feature in the QNX4 days but was told it wasn’t a good idea
and that instead I should write my own “watcher” that would poll the files
and check for modifications.

Background:

I wrote an application which mirrors a directory tree via FTP to a remote
machine. The approach I use is to periodically scan the whole local tree
to see if anything was modified and keep a record about the objects. But
as you know an interrupt driven approach is usually more efficient than
polling. So it’s better to get a message saying that a certain object just
finished getting modified and that it can be mirrored immediately.

One way to do this is to create a resource manager with it’s own namespace
that maps onto the directory tree. However if the object gets modified
outside of the resource manager’s namespace, it won’t know about it.

So I’m looking for other options also. Is there perhaps a way to somehow
interface with Fsys? All the daemon needs to know is what object got
modified and that this modification is complete.

Thank you for any tips you might have.

regards,
rick

3

Mario Charest <nowheretobefound@8thdimension.com> wrote:

MC > “Rick Lake” <ow@private-domain.nl> wrote in message
MC > news:cko80p$cps$1@inn.qnx.com…

Hi developers,

Is there a way for a process to somehow get a message when an object (and
also which object) in the filesystem (Fsys’s namespace) gets modified?

MC > No,

MC > I asked for this feature in the QNX4 days but was told it wasn’t a good idea
MC > and that instead I should write my own “watcher” that would poll the files
MC > and check for modifications.

The right way to do this is with a resource manager. That would be easy
in QNX6.

Way back when, both Geoff Roberts (Australia) and myself had QNX4
mirroring utilities. They both worked by polling but were quite fast.

The slick thing about mine was that if you had two servers that were
mirroring each other, you could take one down for maintenence, for days if
necessary, and as long as the other system stayed up, when you plugged the
other server back in it would automatically update itself.

Mine is no longer available, sorry. I don’t know the state of Geoff’s

4

Mario Charest <nowheretobefound@8thdimension.com> wrote:

“Rick Lake” <> ow@private-domain.nl> > wrote in message
news:cko80p$cps$> 1@inn.qnx.com> …
Hi developers,

Is there a way for a process to somehow get a message when an object (and
also which object) in the filesystem (Fsys’s namespace) gets modified?

No,

I asked for this feature in the QNX4 days but was told it wasn’t a good idea
and that instead I should write my own “watcher” that would poll the files
and check for modifications.

was afraid of that. guess I’ll have to dive into the “iomanager” examples;
thanks anyway.

regards,
rick

5

“Rick Lake” <ow@private-domain.nl> wrote in message
news:ckqt2h$dq5$1@inn.qnx.com…

Mario Charest <> nowheretobefound@8thdimension.com> > wrote:

“Rick Lake” <> ow@private-domain.nl> > wrote in message
news:cko80p$cps$> 1@inn.qnx.com> …
Hi developers,

Is there a way for a process to somehow get a message when an object
(and
also which object) in the filesystem (Fsys’s namespace) gets modified?

No,

I asked for this feature in the QNX4 days but was told it wasn’t a good
idea
and that instead I should write my own “watcher” that would poll the
files
and check for modifications.


was afraid of that.

Yeah and kind of sad given the mantra “polling is bad” :wink:

guess I’ll have to dive into the “iomanager” examples;
thanks anyway.


regards,
rick

6

Rick Lake wrote:

Is there a way for a process to somehow get a message when an object (and
also which object) in the filesystem (Fsys’s namespace) gets modified?

You would need to write an IO manager to do this. Have it attach at the
root of the directory hierarchy of interest (eg “/” or some subset). It
will then get all _IO_OPEN messages; you can then scan these for ones
which look like they will modify an object in question (going for write
permission, an O_CREAT, or an unlink, etc). In all cases redirect the
actual request to the real filesystem (using the EMORE/symlink trick;
as others have said this is harder to do in QNX4 than QNX6 where an
ENOSYS suffices). But you now have a hint at which object (the pathname
in the message) is about to be modified, so schedule a later examination
or re-copy of it. [In this scheme you sacrifice some doubt as to
if/when it is modified for the convenience and performance of not having
to process/mirror the requests yourself if you didn’t redirect the IO.]

7

“John Garvey” <jgarvey@qnx.com> wrote in message
news:ckut11$bj4$1@inn.qnx.com…

Rick Lake wrote:
Is there a way for a process to somehow get a message when an object (and
also which object) in the filesystem (Fsys’s namespace) gets modified?

You would need to write an IO manager to do this. Have it attach at the
root of the directory hierarchy of interest (eg “/” or some subset). It
will then get all _IO_OPEN messages; you can then scan these for ones
which look like they will modify an object in question (going for write
permission, an O_CREAT, or an unlink, etc). In all cases redirect the
actual request to the real filesystem (using the EMORE/symlink trick;
as others have said this is harder to do in QNX4 than QNX6 where an
ENOSYS suffices). But you now have a hint at which object (the pathname
in the message) is about to be modified, so schedule a later examination
or re-copy of it. [In this scheme you sacrifice some doubt as to
if/when it is modified for the convenience and performance of not having
to process/mirror the requests yourself if you didn’t redirect the IO.]

Isn’t this scheme responsible for the drop in performance cause by fs-pks
which result in it being remove from 6.3?

8

Hi John,

Thank you for this suggestion. Is this in theory or have you or someone
else actually done this?

Also, am I correct in assuming that by attaching to “/” you will get
_IO_OPEN messages to ALL resource managers? (I.e. also those directed to
Socket for NFS mounted directories, Dev for “/dev/*” devices, etc, and
also the pseudo paths created using the “prefix” util.)

And also (please forgive me if this is an RTFM), can you give me a hint
as to what the “EMORE/symlink” trick is?

regards,
rick

John Garvey <jgarvey@qnx.com> wrote:

Rick Lake wrote:
Is there a way for a process to somehow get a message when an object (and
also which object) in the filesystem (Fsys’s namespace) gets modified?

You would need to write an IO manager to do this. Have it attach at the
root of the directory hierarchy of interest (eg “/” or some subset). It
will then get all _IO_OPEN messages; you can then scan these for ones
which look like they will modify an object in question (going for write
permission, an O_CREAT, or an unlink, etc). In all cases redirect the
actual request to the real filesystem (using the EMORE/symlink trick;
as others have said this is harder to do in QNX4 than QNX6 where an
ENOSYS suffices). But you now have a hint at which object (the pathname
in the message) is about to be modified, so schedule a later examination
or re-copy of it. [In this scheme you sacrifice some doubt as to
if/when it is modified for the convenience and performance of not having
to process/mirror the requests yourself if you didn’t redirect the IO.]

9

Registering as ‘/’ won’t get you everything, only those things that
aren’t attached to another explicit prefix space a.k.a. another
io/resource manager. And, under QNX4 all io/resource managers must
control a unique prefix space.

The “EMORE/symlink” trick goes something like this… When a resource
manager gets an IO_OPEN, IO_STAT, etc. on what it knows to be a symlink,
it replies with EMORE and the full path string to which the symlink
points. The kernel’s “path name manager” will then redirect the IO_*
call to the appropriate resource manger that controls the prefix space
for the “path string” returned in the reply.

So what I think John is say is… mount your “real” file system as say,
“/ROOT” and then register your io/resource manager as “/”. Whenever
your io manager gets an applicable IO_* request, record pertinent
information (for later mirroring investigation) and reply with EMORE,
prefixing the request “path string” with “/ROOT/” in the reply (like it
and everything else in this prefix space is a symlink). The kernel’s
path name manager will then redirect the request to the “real” file
system for all the real work and heavy lifting.

-Rob

Rick Lake wrote:

Hi John,

Thank you for this suggestion. Is this in theory or have you or someone
else actually done this?

Also, am I correct in assuming that by attaching to “/” you will get
_IO_OPEN messages to ALL resource managers? (I.e. also those directed to
Socket for NFS mounted directories, Dev for “/dev/*” devices, etc, and
also the pseudo paths created using the “prefix” util.)

And also (please forgive me if this is an RTFM), can you give me a hint
as to what the “EMORE/symlink” trick is?

regards,
rick

John Garvey <> jgarvey@qnx.com> > wrote:

Rick Lake wrote:

Is there a way for a process to somehow get a message when an object (and
also which object) in the filesystem (Fsys’s namespace) gets modified?

You would need to write an IO manager to do this. Have it attach at the
root of the directory hierarchy of interest (eg “/” or some subset). It
will then get all _IO_OPEN messages; you can then scan these for ones
which look like they will modify an object in question (going for write
permission, an O_CREAT, or an unlink, etc). In all cases redirect the
actual request to the real filesystem (using the EMORE/symlink trick;
as others have said this is harder to do in QNX4 than QNX6 where an
ENOSYS suffices). But you now have a hint at which object (the pathname
in the message) is about to be modified, so schedule a later examination
or re-copy of it. [In this scheme you sacrifice some doubt as to
if/when it is modified for the convenience and performance of not having
to process/mirror the requests yourself if you didn’t redirect the IO.]

10

Rob Hem <rob@spamyourself.com> wrote:

Registering as ‘/’ won’t get you everything, only those things that
aren’t attached to another explicit prefix space a.k.a. another
io/resource manager. And, under QNX4 all io/resource managers must
control a unique prefix space.

Ah, of course! (I should have realized this; silly me). Guess I would have
to deal with NFS mounts differently, since they get modified by processes
on the remote host.

The “EMORE/symlink” trick goes something like this… When a resource
manager gets an IO_OPEN, IO_STAT, etc. on what it knows to be a symlink,
it replies with EMORE and the full path string to which the symlink
points. The kernel’s “path name manager” will then redirect the IO_*
call to the appropriate resource manger that controls the prefix space
for the “path string” returned in the reply.

So what I think John is say is… mount your “real” file system as say,
“/ROOT” and then register your io/resource manager as “/”. Whenever
your io manager gets an applicable IO_* request, record pertinent
information (for later mirroring investigation) and reply with EMORE,
prefixing the request “path string” with “/ROOT/” in the reply (like it
and everything else in this prefix space is a symlink). The kernel’s
path name manager will then redirect the request to the “real” file
system for all the real work and heavy lifting.

Cool trick :slight_smile:

And if the object in question really is a symlink, it will eventually just
be handled as a nested symlink, right?

-Rob

[…]

11

Rick Lake wrote:

Thank you for this suggestion. Is this in theory or have you or someone
else actually done this?

No, this is real (from a local disk-caching server I wrote called
“shadow”). Of course that was 4-5 years ago and I am fuzzy on details.
If did work though (would copy remote files to your local disk as you
accessed them - if not already cached - and then redirect to that
local/faster copy, let writes go through to the original and invalidate
local copy, and periodically re-check the local copies for being stale).

Also, am I correct in assuming that by attaching to “/” you will get
_IO_OPEN messages to ALL resource managers? (I.e. also those directed to
Socket for NFS mounted directories, Dev for “/dev/*” devices, etc, and
also the pseudo paths created using the “prefix” util.)

QNX4 does longest-prefix matching, so attached paths like “/dev” (or
anything output by “prefix”) will go direct to that server, not you.

QNX4 only allows a single server to attach to a unique pathname, so you
will have to do something; either put your disk at “/hd” and attach
yourself to “/”, or (my preference, but trickier) leave your disk at “/”
and attach to a subset of that which you will be mirroring ("/mirror").
This depends if you need notification of the entire disk, or if your
application is structured so that you are interested only in a subset
(or can be so arranged that you will work on only such a subset).

And also (please forgive me if this is an RTFM), can you give me a hint
as to what the “EMORE/symlink” trick is?

Something like this:

uselen = strlen(usepath),
pathlen = strlen(msg.hdr.open.path) + 1;
msg.hdr.open.type = EMORE;
msg.hdr.open.version_cycle = 0x80, msg.hdr.open.eflag = 0;
memmove(&msg.hdr.open.path[uselen + 1], &msg.hdr.open.path[0], pathlen);
memcpy(&msg.hdr.open.path[0], usepath, uselen);
msg.hdr.open.path[uselen] = (pathlen > 1) ? ‘/’ : ‘\0’;
bytes = &msg.hdr.open,
nbytes = sizeof(msg.hdr.open) + uselen + 1 + pathlen;

Basically you would construct a new pathname, made up of where you put
your hard disk mount (“usepath” in above example), plus the incoming
pathname, and EMORE says to the pathname resolver that it was a symlink
and you have supplied the new path (of course it possibly wasn’t a
symlink, but you have given a new path off to the real disk file; the
_IO_OPEN message will now get re-sent to that server (Fsys)).

12

John Garvey <jgarvey@qnx.com> wrote:
[…]

Basically you would construct a new pathname, made up of where you put
your hard disk mount (“usepath” in above example), plus the incoming
pathname, and EMORE says to the pathname resolver that it was a symlink
and you have supplied the new path (of course it possibly wasn’t a
symlink, but you have given a new path off to the real disk file; the
_IO_OPEN message will now get re-sent to that server (Fsys)).

Thank you very much for this elaboration. (As I said previously: a cool
trick :slight_smile: One thing I’m missing is how to know when the requesting process
is finished with the modification. The actual server (Fsys) knows this,
because it gets the “open” and eventually the “close” message. But, IIUC,
my resource manager would still need to poll the modification time of the
object and assume it’s safe to copy after N seconds of idle time, thus
introducing an unnecessary delay. A big plus, though, is that there’s no
need to poll the whole mirrored directory tree. That makes this solution
very attractive.

Thanks again and thanks to all that responded.

regards,
rick

13

Rick Lake wrote:

Thank you very much for this elaboration. (As I said previously: a cool
trick > :slight_smile: > One thing I’m missing is how to know when the requesting process
is finished with the modification. The actual server (Fsys) knows this,
because it gets the “open” and eventually the “close” message. But, IIUC,
my resource manager would still need to poll the modification time of the
object and assume it’s safe to copy after N seconds of idle time, thus

There are still some details for you to work out :wink: Something does
need to poll for modification completion, but it is now targetted to
specific files rather than the whole directory tree. I would suggest
that this not be done by the server process (as it could impact
responsiveness to ongoing redirections). One option may be to have a
process, create a pipe, fork, with one process becoming the message
sniffer/handler, andwrite pathnames to the pipe where the second process
performs the poll and copy (pull pathnames from pipe onto some active
list and then periodically scan and process it [using the real filename
to bypass the redirector]). I don’t think there is an API in QNX4 to
give you the number of writers to a file, so you’d either have to delay
or do some periodic stat() calls until the file size and mtime got
stable.