Documentation/filesystems/orangefs.txt

ORANGEFS
========

OrangeFS is an LGPL userspace scale-out parallel storage system. It is ideal
for large storage problems faced by HPC, BigData, Streaming Video,
Genomics, Bioinformatics.

Orangefs, originally called PVFS, was first developed in 1993 by
Walt Ligon and Eric Blumer as a parallel file system for Parallel
Virtual Machine (PVM) as part of a NASA grant to study the I/O patterns
of parallel programs.

Orangefs features include:

  * Distributes file data among multiple file servers
  * Supports simultaneous access by multiple clients
  * Stores file data and metadata on servers using local file system
    and access methods
  * Userspace implementation is easy to install and maintain
  * Direct MPI support
  * Stateless


MAILING LIST
============

http://beowulf-underground.org/mailman/listinfo/pvfs2-users


DOCUMENTATION
=============

http://www.orangefs.org/documentation/


USERSPACE FILESYSTEM SOURCE
===========================

http://www.orangefs.org/download

Orangefs versions prior to 2.9.3 would not be compatible with the
upstream version of the kernel client.


BUILDING THE USERSPACE FILESYSTEM ON A SINGLE SERVER
====================================================

When Orangefs is upstream, "--with-kernel" shouldn't be needed, but
until then the path to where the kernel with the Orangefs kernel client
patch was built is needed to ensure that pvfs2-client-core (the bridge
between kernel space and user space) will build properly. You can omit
--prefix if you don't care that things are sprinkled around in
/usr/local.

./configure --prefix=/opt/ofs --with-kernel=/path/to/orangefs/kernel

make

make install

Create an orangefs config file:
/opt/ofs/bin/pvfs2-genconfig /etc/pvfs2.conf

  for "Enter hostnames", use the hostname, don't let it default to
  localhost.

create a pvfs2tab file in /etc:
cat /etc/pvfs2tab
tcp://myhostname:3334/orangefs /mymountpoint pvfs2 defaults,noauto 0 0

create the mount point you specified in the tab file if needed:
mkdir /mymountpoint

bootstrap the server:
/opt/ofs/sbin/pvfs2-server /etc/pvfs2.conf -f

start the server:
/opt/osf/sbin/pvfs2-server /etc/pvfs2.conf

Now the server is running. At this point you might like to
prove things are working with:

/opt/osf/bin/pvfs2-ls /mymountpoint

You might not want to enforce selinux, it doesn't seem to matter by
linux 3.11...

If stuff seems to be working, turn on the client core:
/opt/osf/sbin/pvfs2-client -p /opt/osf/sbin/pvfs2-client-core

Mount your filesystem.
mount -t pvfs2 tcp://myhostname:3334/orangefs /mymountpoint


OPTIONS
=======

The following mount options are accepted:

  acl
    Allow the use of Access Control Lists on files and directories.

  intr
    Some operations between the kernel client and the user space
    filesystem can be interruptible, such as changes in debug levels
    and the setting of tunable parameters.

  local_lock
    Enable posix locking from the perspective of "this" kernel. The
    default file_operations lock action is to return ENOSYS. Posix
    locking kicks in if the filesystem is mounted with -o local_lock.
    Distributed locking is being worked on for the future.


DEBUGGING
=========

If you want the debug (GOSSIP) statments in a particular
source file (inode.c for example) go to syslog:

  echo inode > /sys/kernel/debug/orangefs/kernel-debug

No debugging (the default):

  echo none > /sys/kernel/debug/orangefs/kernel-debug

Debugging from several source files:

  echo inode,dir > /sys/kernel/debug/orangefs/kernel-debug

All debugging:

  echo all > /sys/kernel/debug/orangefs/kernel-debug

Get a list of all debugging keywords:

  cat /sys/kernel/debug/orangefs/debug-help