CS 111 Sprint 2010

Scribe Notes for LECTURE 6 - 4/15/2010

by Ameen Askar and Yuanjia Wu

File Descriptors issues and Piping:

What can go wrong with file descriptors?

When working with file descriptors, a lot of things can go wrong. We always need to makesure that we are working with the correct file descriptor, the file is open and not locked by another process, and finally close the file descriptor when we do not need it anymore.

Let us consider this example:

1  fd = 942; 
2  buf = "hello\n";
3  size=5; 
4  int main(void){
5      write(fd,buf, size);
6      return 0; 
7  }

In this example, most likely, the file was not open and write will return -1. This is because the fd is a very high number and usually there will not be 942 files that are open in the same time. It is important to note that the file descriptor returned is the lowest possible number. For example:

If file descriptors 0,1,2,3,4,5,9,11,12 are open (where: 0 is stdin, 1 is stdout, and 2 is stderr)

Then next open will return file descriptor equal to 6 because it will return the lowest possible number for the file descriptor. The error we get from write is only -1, but since there is many possible errors for that to happen, write like many other system calls store the error number in errno which refer to more details about the failure. In the above example it will return EBADF.

How do we handle such an error ? In this example, write returns the numbers of bytes it read.

1  int main(void){
2      if (write(1, "hello\n", 6) != 6){
3          write(2, "ouch!\n", 6); 
4          return 1;
5      }
6      else
7          return 0;
8  }

Another error that might happen is if the file or device we are writing to is full. For example: ./a.out > /dev/full //this always fail because /dev/full is always full.

DANGLING FILE DESCIPRTORS:

These are more tricky errors that might happen:

Sometimes two file descriptors variable in the same process have the same value.

1  int f = open (.....);
2  if (0 <= f){      //if g was closed before and the f now has
3      close(g);    //the same value. We close it (closing f fd) 
4      write(f, .....);    //now writing to it will fail EBADF
5  }

But if we have an h=open(....) before the write(f,....), maybe h will be equal to f again and if it is pointing to another file. This is called dangling file descriptors.

What if we are writing to a device and it that device vanishes:

This happen all the time when reading and writing to removable device. Fortunately the read and write fail with a code that enable us to hold and try again later or exit. It is all referenced by the errno. Here is an example that also shows how we can have file descriptors leak:

1 for(int i=1; i<argc; i++){
2      int fd=open(argv[i], O_RDONLY);
3      if (0<=fd)  
4          read(fd,buf,sizeofbuf);
5      process(buf);
6      close(fd);    //removing the close will cause fds leak.
7  }
8  fd = open ("/dev/usb2", O_RDONLY);
9  //if device vanishes then we try to read
10 read(fd,....);    //this will fail with some other errno (EAGAIN)

Or what if we are trying to read from a device that does not have data available:

Say we are trying to read from keyboard and there is nothing in the buffer. Depending on the system we can get an error with code EAGAIN to try again. Or we can have the read wait until there is data and then return. However, waiting can cause problems since no data may come in for along time and then the program will be waiting forever.

1 int fd = open("/dev/kbd", O_RDONLY);
2 char c; 
3 if(0<=fd)  
4    read(fd, &c, 1);

MORE ERRORS: READ and LSEEK calls can fail but in a nice way:

If we are trying to read from a file that has an end (file /etc/passwd contains 713 bytes ,) read will start returning 0 after we finish reading the whole file. lseek to an invalid positive number will return 0 and lseek to negative position will give an error.

read OR lseek call to:

fd = open("/etc/passwd", O_RDONLY)

Return value



read(fd, buf, 1024);

//gives an error

713

read(fd, buf, 1024);

0

lseek(fd, 239483,0);

0

lseek(fd, -1, 0);

ERROR


Race conditions when writing or reading from same file:

Now we have the condition if two processes are using the same file. They will have different file descriptors since it is different file descriptor tables but the two fds are pointing to the same file. The output in this case will be unexpected. This is why we introduce locks later one.

Ex1:

if cmd1 does write(1,"hello",5)

cmd2 does write(1, "there",6)

(cmd1 & cmd2&) > /tmp/log

The possible result is:

hello there

there hello

htteloroe

Ex2:

(cmd1 > /temp/log &

cmd2 > /temp/log &

)

Offsets are different in this case we have more trouble when doing lseek.

Piping vs. Temp files:

In many applications we need to pass a lot of information that may not fit on RAM from one process to another. To do this we normally create temporary files, many things can go wrong with this.

EXAMPLE: now let us look at creating a temporary file for "Sorting" application:

int fd = open("/tmp/sorttmp", O_RDWR|O_CREAT, 0600);

write to fd

-

-

read from fd

close(fd); //it is better to check for error, if(close(fd)<0) error();

unlink("/tmp/sorttrap") //unlinking an open file will waste memory

//(invisible file). This is why we close it first.

In this code we have a problem if two processes sorting are using the same file to fix it:

1 for(;;){
2    int fd; 
3    char fbuf[1000]; 
4    genRandfielName(fbuf,sizeoffbuf); 
5    if(fd=open(fbuf, O_RDWR|O_CREAT|O_EXCL,0600)){
6         continue;
7         return -1;
8     }
9 }

NOTE: O_EXCL will not open file if already exists. This fix the problem partially. However, what if we are creating the file at the same instance in the two processes. We still have the same file for the two processes. Let us look at different ways of doing temp files.

Better ways to create temp files:

Better way to create temp files:

1- fd = tmpfile() we can have function that will generate random file

2- new syscall opentmp(O_RDWR) :

- nameless file (some problems -> disk space management)

+O.S. can choose file location

+O.S. can free files on close, automatically

3- Locks on files.

4- Pipes.

File locks:

Locks are a very good way to make sure that a file is not used by more than one process at a time. We can lock the whole file or just lock a segment of it depending on the need. Processes can wait for locks and specify what kind of lock it needs. This is done through the system call fcntl which stands for file control and to get the lock we call acquire_lock("/file_name").

To create a temp file using locks we would have the following:

acquire_lock("/temp");

generate file name

try it out

create file

release_lock("/tmp");

fcntl(fd, F_SETLK, p)

Note that F_SETLK is an option of the lock we have:

  • F_SETLK: set a lock, it fails if already locked.

  • F_GETLK:

  • F_SETLKW: set lock waiting

This can cause dead lock with errno: EWOULDDEADLOCK

Also, p in the function has the type (struct flock const *), this structure has:

l_type: lock type

  • F_RDLCK read lock

  • F_WRLCK write lock

  • F_UNLCK none

l_whence: 0,1,2 means start, here, or end of file

l_start: specify segment to lock

l_len: specify the length of the segment to lock

l_pid: specify who has the lock


PIPES:

Pipes are kind of temp files. They are nameless and they have bounded size. Two processes use pipes by: one will by reading from the pipe and the other one is writing to that pipe. Pipes are structured by having a circular buffer. The write will hang if the pipe is full and the read will return 0 when no data to read. If reading is faster the read part will hang for more data to read. If the write part is faster it will hang for the read to read some and set that region into unused space.

Pipes are used in shell by having | the vertical bar between processes:

Du | sort -n

How do we create pipes:

1  pipe (int * fd);     
2    //do some work
3    //with the pipe 
4  close(fd[0]);    //close read and write parts
5  close(fd[1]);

The pipe() function returns -1 if it fails to create it or 0 if it was successful.

Now to implement piping in the example du | sort -n when shell is process 0

du is process 1

sort is process 2

There are three ways to do the piping.

  1. P0 -- P1 -- P2 pipe from 1 to 2

  2. P0 -- P2 -- P1 pipe from 2 to 1

  3. P0 -- P1 and P2 pipe from 0 to 1 and 2


Here is how to do P0 -- P1 -- P2:

1  pid_t p1 = fork();     //fork process
2  if (p==0){             //If child
3    int fd[2];           //var to store pipe
4    pipe (fd);           //create pipe
5    pid_t p2 = fork();   //for another process
6    if (p2 == 0){
7          if (fd[1] != 0){          //if we have pipe
8               dup2(fd[1], 0); //duplicate pipe out into 0 (stdin)
9               close(fd[1]);   //close fd[1], A copy is in stdin
10          }close(fd[0]); 
11           execvp("sort",-------) //run sort in the child procs
12   }
13 }






CS111 Operating Systems Principles, Spring 2010, UCLA. Prof. Paul Eggert, April 15, 2010.

Valid HTML 4.01 Transitional