Implementation
OMEIS is implemented as a CGI program written in C and running behind Apache. The omeis program can also be accessed using a command-line interface. The server is stateless, process-per-request, but supporting concurrent read/write operations on the same files. All files are mmaped and shared between processes. Consistency of the underlying files is maintained with ranged cooperative file locks for reading and writing. The Pixels files contain only raw pixel data in XYZCT order. The server also maintains dimensionality information to avoid bothering the back-end for anything but authentication. Implementation of statistics-based intensity scaling requires maintaining statistics information as well. The dimensionality and statistics is contained in a companion file with the same name as the Pixels file and a '.info' extension. The stored Convert commands are in a second companion file with a '.convert' extension. Statistics are calculated when the FinishPixels call is made. Accessor methods (GetPlaneStats and GetStackStats) are provided to return these statistics to the caller.
The image server maintains separate directory trees for files (uploaded via UploadFile) and Pixels. They do not share the same ID space, so a PixelsID and a FileID may be equal. The last used ID is stored in counter files (Files/lastFileID and Pixels/lastPix). To get a new ID, the file is opened and write-locked, the last ID is read, the ID is incremented and written back to the file, the lock is released, and the file is closed. The directory trees to hold the Files and Pixels are generated dynamically by converting the ID to a string and using one directory level for every 3 digits. FileID=1234567890 will have a path of Files/123/456/789/1234567890 (returned by GetLocalPath). The filename and SHA1 digest of the uploaded file (the value of the 'filename' parameter in the 'File' form variable) is be stored in Files/123/456/789/1234567890.info. Pixels are treated similarly, with the companion/header file being stored in Pixels/123/456/789/1234567890.info (for PixelsID=1234567890).
OMEIS enforces a unique constraint on all files and pixels through the use of SHA1 digests. Attempting to store identical files and pixels will not generate errors, the IDs returned will simply be those of the existing identical files and pixels. This is done by using Berkeley DB. This is a B-Tree that stores PixelsIDs and FileIDs indexed by SHA1 digest to allow rapid lookups of pixels and files by their digests. In the future, this lookup may be exposed to the external interface (i.e. return a FileID or PixelsID from a SHA1 digest).
Pixels are considered identical if they have the same content (and thus SHA1 digest). Identical files have the same digest and same filename. If the files' digests are the same but their filenames are different, they are assigned different IDs. However, with the help of symbolic links only one copy of their contents is stored on hard-disk. For all intents and purposes, such as DeleteFile method calls, the files are considered separate.
Because all files and pixels are mmaped and shared between omeis processes, this implementation will benefit significantly from lots and lots of RAM. The RAM will be used by the OS to cache the mmap'ed files using the CPU's virtual memory system, and paged in and out of disk as necessary. The omeis processes themselves have a minimal RAM footprint even when dealing with very large files, so lower RAM will only hurt performance, not functionality.
