Table of Contents

  1. RPC Performance Issues
  2. Network File Systems
  3. NFS Problems

RPC Performance issues

Distance:

The first Problem concerning Remote Procedure call performance is the issue of Distance. Since the speed of light is a fixed constant, there will always be some propagation delay in a distributed system where the individual nodes are large distances apart. One solution to this problem is simply to move the machines closer together.

An example of this type of solution in action can be found on wall street where competing trading firms try to acquire real estate closest to wall street in order to ensure their trades are executed as fast as possible. Obviously, this solution isn't suitable for everyone since its impossible for every node to be close to each other. Therefore more creative approaches are uses in reality such as:

Pipelining:

Pipelining is strategy in which multiple requests are sent simultaneously without waiting for a response for the previous request. The immediate upside is that it enables us to exploit parallelism via asynchronous connections. That is to say, while one request is being processed, instead of waiting for that response to return, we instead assume success and send the next response in order to ensure the server always has something to do.

This strategy is frequently employed in the HTTP protocol in order to speed up webpage load times on residential (slow) connections. However, this strategy is also not without its share of draw backs. The first problem is the increase in server load that pipelining requires. This is because the server is constantly performing tasks for the client that are at times redundant or unnecessary due to a failure or dependency in an earlier request. Therefore if the individual requests are unlikely to be dependent on server feed back, pipelining will improve efficiency.

Caching:

Caching is a method by which the client tries to anticipate the servers response to a given request. This is typically done by storing a previous response to a similar request from the server locally on the client. This enables us to reduce redundant operations over a network and thereby take advantage of locality to speed up the process.

One problem observed with caches is the issue of stale state. This occurs when the local copy on the client is not up to date with the current copy on the server. In order to alleviate the potential for displaying an outdated cached item, there are a number of solutions we employ:

  1. Expiration Date: Cached items are stored with a set date upon which they are no longer valid. This time frame of validity can be set by the server such that shorter validity times correspond to more frequent updates.
  2. Status Query: Another potential way to alleviate the problem of a stale cache is to implement a low overhead method of checking whether the local cache is still up to date. The down side however is that we still must wait for a response to travel over a network which will always have some propagation delay. Given the relative size of most requests to the speed of most modern broadband connections, the question then becomes whether or not its better to simply just re-download the server response.
  3. Push Protocol: You might be asking, wouldn't it just be easier to simply notify the clients of server updates? Push Notification allows just that. When a webpage for instance is updated the server can notify the clients that their cache has become stale and that they should reacquire data. One draw back of push notifications is that the client must be on and listening for a server push notification. In the case of mobile computing, this can mean more power consumption. Another drawback is simply increased server load.

Pre-fetching:

The final strategy employed to improve RPC performance issues is pre-fetching. This is when the client tries to anticipate what the user will do next and send the request to the server before the client actually requests that data. An example of this could be when the client recognizes dependencies of a particular request and automatically pre-fetches them before they are needed by the client.

Network File Systems

Network File Systems allows us to abstract the physical location of the disk containing the information from the users. This enables us to have multiple machines contribute to the same file system. However, there are a number of issues that must be taken care of from an operating systems's point of view

Mount Tables:

Mount tables are used to determine what file systems we have accessible and where they are accessible from. Notice however that since there could be multiple file systems of the same type, there exists the potential for inodes with the same name. Therefore we must list not only the inode number, but also the file system to which it belongs to. When a client attempts to explore a particular directory, the kernel first consults the mount table and jumps to the appropriate location. This also enables the potential to conceal aspects of the file system from users by mounting over top of them. When a client attempts to access the file system, the mount table redirects them to the new file system instead of what was previously underneath.


Problems:

Potential Problems with a mount tables include the possibility of loops existing. Although historically the mounting of a file system twice is forbidden, certain file systems such as LOFS allow it. Another potential issue is the potential for a user to, for example, mount something over top of /ect/passwd thereby allowing them to instantiate their own password file and gain root access. The typical remedy for this situation is simply to only allow mounting as root.

Process Structure:

Consider the following linux process structure:

Struct Process
    - Struct task
        -Struct filestruct (contains the file descriptors to open files)
            -Struct file
                -Struct inode (points to open files)

--------------------------- Virtual File System Layer ------------------------------
Struct file operations
    - Functions: read and write
Struct inode operations
    - Functions: Unlink, remain, ect.

The question becomes why the need for so many structs? This is because file systems are modeled after object oriented programming but are written in C instead of C++.

File Handles:

As stated earlier, we when using several file systems on a networked system, we need a way to distinguish a inodes from one another. To solve this problem, file handles were created. A file handle is simply an inode number + a device number on a server. An example might be a 96 bit number that is partitioned such that the first 32 bits represent the device number on a server and the remaining 64 bits represent the file on the device.


Since the file handle is distributed on the server-side, this affords us a level of abstraction in that the client doesn't actually see the details of how the network file system is arranged. The server simply interprets the file handle and knows which device and inode we are talking about.

Why File Handles?

System calls in traditional linux machines use file descriptors which are a small set of numbers which the kernel uses to keep track of open files. However, in a networked file system, we want the clients to survive server crashes. If we were to use file descriptors we would encounter broken pipes resulting in crashes. To alleviate this problem, we use file handles instead. This allows the clients to continue to operate normally during server crashes with the only concession being performance.

Stateless NFS:

Since Network File Systems were designed with client resiliency to server crashes in mind, NFS uses a stateless protocol. That is, the server can be viewed as simply program connected to a disk where no important state information is stored by the program. This means that there is no way for the server to know if a user still has a file open or not. To alleviate this complication the clients must keep track of all the processes that are using that file.


If the client issues the command to delete a file on the server (link count drops to 0), but the file was still open on the client machine then we encounter a problem since the open file will no longer be available. In this scenario, NFS cheats by renaming the file to some obscure name such as “NFSd9d34”. When the last process using that file is closed, NFS then deletes “NFSd9d34”. In the rare possibility that the server should crash before the client had a chance to delete the obscure file, then there become a lost file that which would eventually need to be manually removed

NFS Protocol:

Network File Systems use requests that look suspiciously like Unix system calls. Some examples of these requests are the following:

MKDIR

  1. Parameters:This command makes a new directory on the remote file system taking 3 parameters. The first parameter is a file handle that points to the parent directory of the file to be created. The second parameter is the name of the new directory to be created. The final parameter is the attributes of the directory such as its permissions
    Return Type: The server responds by sending a file handle to the new remote directory.

REMOVE

  1. Parameters:This command takes a directory file handle and the name of a file and deletes that particular file.
    Return Type: The server responds with the status of the delete operation.

LOOKUP

  1. Parameters:This command takes a directory file handle and the name of a file.
    Return Type: Returns the file handle and the attributes of a file. This function is analogous to the stat command.

READ

  1. Parameters:This command takes a file handle, and offset in the file and a size of data to be read
    Return Type: The server then responds with the data stream requested

Note: Refer to table 4-1 in the book for more NFS commands

NFS Problems

When dealing with a distributed file system among multiple nodes, the issue of file integrity becomes of particular importance. The two main problems fall into the category of either synchronization issues, or reliability issues.

Synchronization:

Consider the following scenario: Client 1 gets the time of day and writes to a file. Client 2 also gets the time of day and reads from the file Client 1 wrote two into a buffer. Similarly to the time-travel lab, it is possible that client 2 may see that the clock moved backwards in time. The reason for this is again due to the stateless nature of NFS protocol. Even from a single client, writes might not be executed in the order specified. For this reason, NFS does not have read-to-write consistency.

NFS does however have close-to-open consistency. This means that the client waits for all write responses from the server before a close is performed. However, close is a system call. There is no CLOSE in the NFS protocol. This is because NFS doesn't care about file descriptors, just file handles. This means that when the client actually calls close(file descriptor), it is possible for close to return -1 with the errno set to EIO. For this reason it becomes of particular importance to check the close return values for potential write fails on an NFS server. There is one alternative option and that is to enable the Synchronous write flag, but this is rarely done since it essentially eliminates pipelining resulting in poor performance.

Reliability Issues:

Media faults can and will occasionally occur with any file system. On distributed or networked file systems the chances of failure increase proportionally to the number of devices involved. When disk blocks go bad, or devices report read/write errors, there needs to be some sort of mechanism to ensure the our data is not lost. The first solution to data corruption was to use logging to mark areas that were compromised. The issue then became, well what happens if the log is corrupted? This gave rise to double logging or redundancy which is now better know today as a RAID, a Redundant Array of Independent Disks.

RAID:

Redundant Arrays of Independent Disks are very common in systems where performance or integrity are desired. RAIDs exploit the properties of systems with multiple physical drives in order to boost integrity and performance. The multiple configurations for several types of raids are covered here:

RAID1:

The most basic RAID for ensuring data integrity is RAID 1. In this configuration, data is written to 2 or more mirrored drives such that if one drive should fail, there is always a copy intact. When data is read, it can be from any of the disks since they are identical. RAID systems assume a single point of failure, if a drive fails, it is replaced and copied immediately before multiple failures can occur.

RAID0:

aid 0 on the other hand sacrifices data integrity for an increase in performance. In this RAID 0 disks are configured in series in either one of two forms. The first form is concatenation, whereby 3 little disks are behave as if they are 1 large disk drive. The second form is striping, where by disk sequential disk blocks are placed on separate disks. This enables us to do sequential reads and writes in parallel given sufficient bus bandwidth.


RAID4:

One problem with raid 1 is that it effectively increases the cost by a factor of 2. Raid 4 attempts to offer a cheaper alternative using what is known as a parity disk. Consider the following 4 disk array:

Drives A-D each hold storage data where as drive E is a parity disk consisting of the xor of disks A-D. In order to perform a write in raid 4, we first write to a data disk, then we read from the parity disk and before finally writing the new parity value. The benefit of this configuration is that should there be a read error on disk C for instance, we can recover the data by simply doing A^B^D^E where E=(A^B^C^D). This type of system is used on seasnet since it is easy to add additional disks.

RAID5:

Raid 5 is similar to raid 4 with the exception being that a Raid 0 configuration is applied to the data section in order to boost performance. It is worth noting however that on both Raid 4 and Raid 5, there is the potential for the parity disk to become a bottle neck. For this reason, often times the parity disk is also mirrored.


Failure Rates:

Disk failures are bound to happen in any system, but it is important to factor in disk failure trends when considering the usage of a RAID. Disk failure rates typically take the form of a bathtub curve such that the highest rates of failure are present either very early or several years later into a disks service life.



 

Notice however, that if we were to use a RAID in a system such as the mars rover where swapping out faulty drives is not an option, the initial failures rate would remain low up until the first disk failure, upon which the chances of failure increase exponentially.