safe priorities?

Hiya!
I was diagnosing some relatively new code and was wondering about it’s
effect on the file system. The process only reads data from the disk, but
others in the system would be updating data. The priority for the process
was set to be higher than Fsys. I’m wondering if this could cause
corruptions in the file system if it takes any CPU time way from Fsys.

Is it safe to have a process at the same priority as Fsys, or is it best to
keep processes just below Fsys’s priority?

TIA!
Ron

Ron Cococcia <cococr@cs.rpi.edu> wrote:

Hiya!
I was diagnosing some relatively new code and was wondering about it’s
effect on the file system. The process only reads data from the disk, but
others in the system would be updating data. The priority for the process
was set to be higher than Fsys. I’m wondering if this could cause
corruptions in the file system if it takes any CPU time way from Fsys.

Is it safe to have a process at the same priority as Fsys, or is it best to
keep processes just below Fsys’s priority?

Actually, unless you’ve done something quite strange, Fsys doesn’t actually
have a priority of its own. It floats to the priority of the requesting
process. Now, the drivers have a priority – it happens to be 22. Is
that what you’re talking about?

Anyway, in general there isn’t a problem with running above Fsys’ priority –
it shouldn’t cause corruption or any such thing.

-David

QNX Training Services
dagibbs@qnx.com

OK, tx. If I run at a priority higher than another process that is doing
disk writes, could it possibly cause corruption?


“David Gibbs” <dagibbs@qnx.com> wrote in message
news:9kci9f$l7m$1@nntp.qnx.com

Ron Cococcia <> cococr@cs.rpi.edu> > wrote:
Hiya!
I was diagnosing some relatively new code and was wondering about it’s
effect on the file system. The process only reads data from the disk,
but
others in the system would be updating data. The priority for the
process
was set to be higher than Fsys. I’m wondering if this could cause
corruptions in the file system if it takes any CPU time way from Fsys.

Is it safe to have a process at the same priority as Fsys, or is it best
to
keep processes just below Fsys’s priority?

Actually, unless you’ve done something quite strange, Fsys doesn’t
actually
have a priority of its own. It floats to the priority of the requesting
process. Now, the drivers have a priority – it happens to be 22. Is
that what you’re talking about?

Anyway, in general there isn’t a problem with running above Fsys’
priority –
it shouldn’t cause corruption or any such thing.

-David

QNX Training Services
dagibbs@qnx.com

Or, if I run a process higher than Proc, could that do bad things to other
processes requesting disk access? This is probably the one case I worry
about (very avoidable) but I want to know if it’s safe.

Ron


“Ron Cococcia” <cococr@cs.rpi.edu> wrote in message
news:9kcilq$lec$1@inn.qnx.com

OK, tx. If I run at a priority higher than another process that is doing
disk writes, could it possibly cause corruption?


“David Gibbs” <> dagibbs@qnx.com> > wrote in message
news:9kci9f$l7m$> 1@nntp.qnx.com> …
Ron Cococcia <> cococr@cs.rpi.edu> > wrote:
Hiya!
I was diagnosing some relatively new code and was wondering about it’s
effect on the file system. The process only reads data from the disk,
but
others in the system would be updating data. The priority for the
process
was set to be higher than Fsys. I’m wondering if this could cause
corruptions in the file system if it takes any CPU time way from Fsys.

Is it safe to have a process at the same priority as Fsys, or is it
best
to
keep processes just below Fsys’s priority?

Actually, unless you’ve done something quite strange, Fsys doesn’t
actually
have a priority of its own. It floats to the priority of the requesting
process. Now, the drivers have a priority – it happens to be 22. Is
that what you’re talking about?

Anyway, in general there isn’t a problem with running above Fsys’
priority –
it shouldn’t cause corruption or any such thing.

-David

QNX Training Services
dagibbs@qnx.com

Ron Cococcia <cococr@cs.rpi.edu> wrote:

OK, tx. If I run at a priority higher than another process that is doing
disk writes, could it possibly cause corruption?

Again, it shouldn’t do so – you will pre-empt Fsys, but that is
expected behaviour. That is, Fsys expects to get pre-empted in
the middle of doing stuff. In fact, from Fsys’ point of view, it
shouldn’t notice that it got pre-empted.

Now, if you NEVER give Fsys (or its drivers) a chance to run, then
you could run into some problems… corruption not being the main
one, just no data moving being the main one.

Are you seeing some corruption? Or is this just making sure?

(The best (worst) way to corrupt data from Fsys’ point of view is to hit
the power button while it is updating meta-data, e.g. directory information,
the bitmap, or the inodes file. 2nd best (worst) way is while it is updating
a file itself.)

If you are seeing corruption, can you describe what type of corruption
you are seeing?

-David

QNX Training Services
dagibbs@qnx.com

I was just trying to relate the beginnings of corruption in our product with
some changes to system settings. I hadn’t received any major corruptions
until 1 customer couldn’t boot his unit (Lost Link). After that, it seemed
that corruptions were becoming more common. One thing I just realized is
that it might have started happening after I changed a priority of one
process to just higher than Proc (reduced Proc to 26, bumped my process to
29). Internally I haven’t noticed anything weird, but I was worried (even
before making the change) that some unsuspecting side effect might rear it’s
ugly head.

The product we make can be interrupted by power (unfortunately) and my guess
is that’s the reason the corruptions are happening. But from the internal
point of view, if something I was doing could possibly prevent a file system
update from happening when it needed to be, I wanted to know.


“David Gibbs” <dagibbs@qnx.com> wrote in message
news:9kck0p$lu5$1@nntp.qnx.com

Ron Cococcia <> cococr@cs.rpi.edu> > wrote:
OK, tx. If I run at a priority higher than another process that is
doing
disk writes, could it possibly cause corruption?

Again, it shouldn’t do so – you will pre-empt Fsys, but that is
expected behaviour. That is, Fsys expects to get pre-empted in
the middle of doing stuff. In fact, from Fsys’ point of view, it
shouldn’t notice that it got pre-empted.

Now, if you NEVER give Fsys (or its drivers) a chance to run, then
you could run into some problems… corruption not being the main
one, just no data moving being the main one.

Are you seeing some corruption? Or is this just making sure?

(The best (worst) way to corrupt data from Fsys’ point of view is to hit
the power button while it is updating meta-data, e.g. directory
information,
the bitmap, or the inodes file. 2nd best (worst) way is while it is
updating
a file itself.)

If you are seeing corruption, can you describe what type of corruption
you are seeing?

-David

QNX Training Services
dagibbs@qnx.com

Ron Cococcia <cococr@cs.rpi.edu> wrote:

I was just trying to relate the beginnings of corruption in our product with
some changes to system settings. I hadn’t received any major corruptions
until 1 customer couldn’t boot his unit (Lost Link).

That sounds like unexpected power-off type corruption.


The product we make can be interrupted by power (unfortunately) and my guess
is that’s the reason the corruptions are happening. But from the internal
point of view, if something I was doing could possibly prevent a file system
update from happening when it needed to be, I wanted to know.

In general, except at power-off time, no particular filesystem update is
time critical. Fsys maintains a consistent in-memory state.

Complete starvation is, of course, bad – but that doesn’t sound like what
is happening.

But, a partially updated on-disk state when a sudden power-off hits can
and will cause corruption. Of course, if you do delay disk updates, you
can slightly increase the likelihood that your filesystem will be corrupted
in case of a sudden power-off. But, since the high priority thing is
generally pre-empting both Fsys & the client requesting the write/change,
I’m not sure this will have any detectable affect.

Make sure you are NOT running Fsys with the -a option. It can greatly
increase the likelihood of corruption in case of unexpected power loss.

-David

QNX Training Services
dagibbs@qnx.com

Ron Cococcia <cococr@cs.rpi.edu> wrote:

Or, if I run a process higher than Proc, could that do bad things to other
processes requesting disk access? This is probably the one case I worry
about (very avoidable) but I want to know if it’s safe.

If you’re runnig above Proc, and READY, then nobody else is requesting
disk access, because you’ve pre-empted them. If you’re blocked, then
it doesn’t matter. In essence, this should not have any effect and
is not dangerous.

(Unless, of course, your high priority process starves the rest of the
system of CPU time…)

-David

QNX Training Services
dagibbs@qnx.com

Hi

I’ ve run QNX 4 systems with priorities all over the place. No problem.

HOWEVER . . .

One possibility of perceived corruption might be that if a lower priority
process tries to write a large block of data to a file, and only gets half
done(I.E. it requires multiple messages to Fsys), then a higher priority
process preempts Fsys and tries to read that whole block of data from the
same file, it will end up reading the new data from the first part of the
block and the old data from the last part of the block.

Note: The file won’t actually be corrupted, but the reading process may see
it as corrupted. It this what you’re seeing?

Bill Caroselli

“Ron Cococcia” <cococr@cs.rpi.edu> wrote in message
news:9kcfr5$k40$1@inn.qnx.com

Hiya!
I was diagnosing some relatively new code and was wondering about it’s
effect on the file system. The process only reads data from the disk, but
others in the system would be updating data. The priority for the process
was set to be higher than Fsys. I’m wondering if this could cause
corruptions in the file system if it takes any CPU time way from Fsys.

Is it safe to have a process at the same priority as Fsys, or is it best
to
keep processes just below Fsys’s priority?

TIA!
Ron

In an application at FAAC, we run our executive program above Proc and have no
problems. In fact, it is necessary to do so to avoid some latency problems
associated with starting/stopping processes.

In another application at another company where we are writing financial data to
disk and cannot afford to lose/corrupt it, we use a fairly long write-behind time
but have the entire system on UPS with interrupt on power line loss. When
batteries are running low, we shut down the application, flush disk cache and then
shutdown QNX. A monitored UPS is the only surefire way, that I know of, to avoid
data loss/corruption.

David Gibbs wrote:

Ron Cococcia <> cococr@cs.rpi.edu> > wrote:
I was just trying to relate the beginnings of corruption in our product with
some changes to system settings. I hadn’t received any major corruptions
until 1 customer couldn’t boot his unit (Lost Link).

That sounds like unexpected power-off type corruption.

The product we make can be interrupted by power (unfortunately) and my guess
is that’s the reason the corruptions are happening. But from the internal
point of view, if something I was doing could possibly prevent a file system
update from happening when it needed to be, I wanted to know.

In general, except at power-off time, no particular filesystem update is
time critical. Fsys maintains a consistent in-memory state.

Complete starvation is, of course, bad – but that doesn’t sound like what
is happening.

But, a partially updated on-disk state when a sudden power-off hits can
and will cause corruption. Of course, if you do delay disk updates, you
can slightly increase the likelihood that your filesystem will be corrupted
in case of a sudden power-off. But, since the high priority thing is
generally pre-empting both Fsys & the client requesting the write/change,
I’m not sure this will have any detectable affect.

Make sure you are NOT running Fsys with the -a option. It can greatly
increase the likelihood of corruption in case of unexpected power loss.

-David

QNX Training Services
dagibbs@qnx.com

Dean Douthat <ddouthat@faac.com> wrote:

When batteries are running low, we shut down the application, flush
disk cache and then shutdown QNX. A monitored UPS is the only surefire
way, that I know of, to avoid data loss/corruption.

What he said.

You can reduce the chances, and recover from most errors in other cases,
but what he said is true.

Oh ya, also, lock away the power button on the computer. :slight_smile: You don’t
want someone accidentally hitting it. :frowning:

-David

QNX Training Services
dagibbs@qnx.com

I fully agree, though UPS and polite shutdown is not always
possible/practical.

You also need to be aware that there is a problem floating somewhere
in Fsys (4.24 and 4.25) that can arbitrarily trash disks. John Garvey
and I were hunting it down over 2 years ago but it still exists (because
I see it more frequently) and there are others that have posted it but
not gotten any response.

I’ve seen it manifested in multiple ways - all relating to .bitmap
allocation:

  • files being extended overwrite files being loaded/executed
  • files being extended overwrite bad things - .boot, .inodes, root
    directory
  • it hops partitions and drives - writing to hd1t80 I get data from one of
    these files in something on hd0t78… This spans SCSI and IDE drivers!

Why now more than before - longevity (I believe) - files are getting larger
and fragmented with some approaching the 2gig limit.

  • One situation is running out of disk space - keep opening and trying to
    write (we were doing ~40 at a time) and if you keep trying at some
    point it will step on something… We fixed the “feature” that was doing
    this
    be I still don’t think Fsys should have done what it did.
  • 2nd is exceeding the 2gig limit on a sequential write and not stopping.
    We were backing up databases from an NT system and it exceeded
    2gig without anybody noticing… we had a couple of lost links/misallocated
    blocks after each backup - give it time and it self-destructs… We also
    fixed this in our code but it still shouldn’t happen.

This is the only ongoing complaint our support group has today
regarding QNX…

David - I am re-running this through our sales channel to get it (hopefully)
fixed once and for all… I need it on 4.24 and 4.25.

Jay

Dean Douthat wrote in message <3B6ABE4F.5365D919@faac.com>…

In an application at FAAC, we run our executive program above Proc and have
no
problems. In fact, it is necessary to do so to avoid some latency problems
associated with starting/stopping processes.

In another application at another company where we are writing financial
data to
disk and cannot afford to lose/corrupt it, we use a fairly long
write-behind time
but have the entire system on UPS with interrupt on power line loss. When
batteries are running low, we shut down the application, flush disk cache
and then
shutdown QNX. A monitored UPS is the only surefire way, that I know of, to
avoid
data loss/corruption.

David Gibbs wrote:

Ron Cococcia <> cococr@cs.rpi.edu> > wrote:
I was just trying to relate the beginnings of corruption in our product
with
some changes to system settings. I hadn’t received any major
corruptions
until 1 customer couldn’t boot his unit (Lost Link).

That sounds like unexpected power-off type corruption.

The product we make can be interrupted by power (unfortunately) and my
guess
is that’s the reason the corruptions are happening. But from the
internal
point of view, if something I was doing could possibly prevent a file
system
update from happening when it needed to be, I wanted to know.

In general, except at power-off time, no particular filesystem update is
time critical. Fsys maintains a consistent in-memory state.

Complete starvation is, of course, bad – but that doesn’t sound like
what
is happening.

But, a partially updated on-disk state when a sudden power-off hits can
and will cause corruption. Of course, if you do delay disk updates, you
can slightly increase the likelihood that your filesystem will be
corrupted
in case of a sudden power-off. But, since the high priority thing is
generally pre-empting both Fsys & the client requesting the write/change,
I’m not sure this will have any detectable affect.

Make sure you are NOT running Fsys with the -a option. It can greatly
increase the likelihood of corruption in case of unexpected power loss.

-David

QNX Training Services
dagibbs@qnx.com