protecting filesystem from power failures

what are the protections and option we have with QNX 6.3.2 when it comes to protecting filesystem with sudden power outtage?
What happens to the filesystem/files where power is out during write?


It’s pretty robust. You could add chkfsys in the startup procedure. In 10 years we never had a problem.

Or check the new qnx6fs in 6.4.x for even more robusteness.

I understand that is is pretty robust but I guess not robust enough in my case because I do see couple of times some of our db files (its just a binary files) corrupt due to computer turning off during writing to that file. This is how I am starting the driver right now.
devb-eide eide wcache blk cache=10m,automount=hd0t77:/

but thinking of adding the “commit=high” option, would this help?

It is quite robust, and yes “commit=high” will help, but there is a trade off between performance and integrity. There are two issues, data and meta-data.

QNX’s file system has always put a high priority on meta-data. It is possible in theory to put meta-data on the disk in an order where the file structure is always logically consistent and I believe that the current QNX file systems does this. A hard drive write cache can defeat this, but it does reduce the possibility to a very small window in time.

The strategy of sending meta-data to disk quickly cannot solve a basic problem with data. If you turn off the disk while there is “dirty” data in a write cache somewhere, your database can become corrupt. There are strategies that get around this, but at the expense of performance. You can pre-write your updates to disk, and then apply them. I know that MySQL does something like this.

QNX is not immune to problems with meta-data nor data. Meta-data problems should only occur when increasing the size of a file so one strategy is to pre-write your files to their largest possible size and keep track of their logical size yourself. Chkfsys should clean up any meta-data problems at startup, but it is always possible that a file will end up with corrupt data in sectors that never were written. By shortening the time between when data is written to the write cache, and when it gets to disk, you will impact the performance.

Here are some strategies that may mitigate problems:

  1. Use RAID’ed disk
  2. Put your system on a UPS
  3. Have a procedure for shutting down where you first close the database and then sync the disk
  4. Have a procedure where anyone (yourself included) has to put your fingers in a wall socket if you fail to follow the procedure in 3).

Note: 1 and 2 do not protect you from user error.

if I use Flash File System or Embedded Transactional File System (ETFS) how would this be different compare to QNX4 File System ?
My embedded system uses either the IDE flash card or Compact flash card. Also I don’t have sql db running, what I meant by db in my case is just a file read and write. I guess my simple question is what would happen to the file when the power goes out in the middle of a write? How would these different file system handle it? What can I do in software to possibily minimize data corruption when this happens?


You can’t use ETFS on a hardisk (being a real one or flash based). ETFS is for FLASH based hardware that provide direct access.

Here is a way to minimize problems if you are worried that your system could crash in the middle of a write. Pre-write the file as big as you will need it with some known value, eg. zeros.

If you use Compact Flash, it is contra-productive to your goal, reliability. Compact Flash by nature ‘hides’ what it really does to the OS. It looks like an IDE Hard Disk, but it isn’t one. It means that it claims to have written data, but in reality it hasn’t, yet. You must use commit=high, or use the 6.4 fs-qnx6, which is a copy on write filesystem, which either presents the old or the new version of the data. But even then, the file system is just software which has to rely on hardware to state the truth about writing data. The fs-qnx6 filesystem uses a ‘sync’ command to ensure that data is really written. CF tends to respond to ‘ok, done’ to this kind of command even if it is still writing…

thanks for all your input, I’ll give it more testing and report back.

Yes taking mechanical HD aside Compact Flash is probably the worst device you can use if reliability is important.

Okay I have decided to not do commit=high because this will be a global setting affecting all writes to the disk which I don’t want. But I do want to control writes to certain files to have the same affect as commit=high. would below code essentually do what I think it does? I am not clear on setting O_DSYNC vs O_SYNC vs O_RSYNC, I know it is documented but its kind of hard to follow and how it really behaves if I use combination of those flags. Again all I want is for the write() to write directly on to Compact Flash instead of cache and or a buffer.

if ((filedes = open(f->path, O_WRONLY | O_TRUNC
|O_DSYNC | O_SYNC | O_CREAT,0664))== -1) {
printf(“Could not open/create file %s\n”, f->path);

wrnum = write(filedes,p,len);

As far as I know that is the best you can do from within a program. Im not sure its as good as commit high though. Also given you are using Compact Flash I am not sure it really matters.