Dev.random ENOMEM problem [fixed]


I don’t know if this was already addressed, but Dev.random (public domain
software) had a problem that after a time it would return ENOMEN when
trying to open it’s /dev device. This was the cause:

An OCB table is kept, which is an array of pointers (Ocb *ocbs_[FDMAX]).
Every time an _IO_OPEN message arrives, an OCB struct is malloc()'d and the
returned pointer is assigned to an entry in this OCB table.

Now when an _IO_CLOSE message arrives, this struct is free()'d again.
However, the pointer value was retained in the OCB table. The next time an
_IO_OPEN message arrives, the next entry in the table is used, because the
previous one still contained a value. In due time the table fills up and
an ENOMEM is returned.

The fix is quite straight forward: every time the OCB struct is free()'d,
also clear the entry in the OCB table (i.e. assign NULL to it).

Here’s a possible fix. All this is in devrand.c. The function FdUnMap() is
the function which free()s the OCB struct. It should also clear the ocbs_
entry. I.e:

int FdUnMap(pid_t pid, int fd)
int idx = 0;
Ocb *ocb = FdGet(pid, fd, &idx);

if (ocb->links == 0)
ocbs_[idx] = NULL; // this fixes the problem

And FdGet should be modified accordingly to fill idx with the
correct ocbs_ index. I.e:

Ocb *FdGet(pid_t pid, int fd, int *idx)

if (idx != NULL)
*idx = index; // return the index to the caller, i.e. FdUnMap()

return ocbs_[index];

All other instances of FdGet() calls, can just pass a NULL as the idx
argument, since the idx isn’t used in those calls.