OMEIS is implemented as a CGI program written in C and running behind Apache. The
omeis program can also be accessed using a command-line interface. The server
is stateless, process-per-request, but supporting concurrent read/write
operations on the same files. All files are mmaped and shared between
processes. Consistency of the underlying files is maintained with ranged
cooperative file locks for reading and writing.
The Pixels files contain only raw pixel data in XYZCT order. The server also
maintains dimensionality information to avoid bothering the back-end for
anything but authentication. Implementation of statistics-based intensity
scaling requires maintaining statistics information as well. The dimensionality
and statistics is contained in a companion file with the same name as the
Pixels file and a '.info' extension. The stored Convert commands are in a
second companion file with a '.convert' extension. Statistics are calculated
when the FinishPixels call is made. Accessor methods (GetPlaneStats and
GetStackStats) are provided to return these statistics to the caller.
The image server maintains separate directory trees for files (uploaded via
UploadFile) and Pixels. They do not share the same ID space, so a PixelsID and
a FileID may be equal. The last used ID is stored in counter files
(Files/lastFileID and Pixels/lastPix). To get a new ID, the file is opened and
write-locked, the last ID is read, the ID is incremented and written back to
the file, the lock is released, and the file is closed. The directory trees to
hold the Files and Pixels are generated dynamically by converting the ID to a
string and using one directory level for every 3 digits. FileID=1234567890 will
have a path of Files/123/456/789/1234567890 (returned by GetLocalPath). The
filename and SHA1 digest of the uploaded file (the value of the 'filename'
parameter in the 'File' form variable) is be stored in
Files/123/456/789/1234567890.info. Pixels are treated similarly, with the
companion/header file being stored in Pixels/123/456/789/1234567890.info (for
PixelsID=1234567890).
OMEIS enforces a unique constraint on all files and pixels through the use of
SHA1 digests. Attempting to store identical files and pixels will not generate
errors, the IDs returned will simply be those of the existing identical files
and pixels. This is done by using Berkeley DB. This is a B-Tree that stores
PixelsIDs and FileIDs indexed by SHA1 digest to allow rapid lookups of pixels
and files by their digests. In the future, this lookup may be exposed to the
external interface (i.e. return a FileID or PixelsID from a SHA1 digest).
Pixels are considered identical if they have the same content (and thus SHA1
digest). Identical files have the same digest and same filename. If the files'
digests are the same but their filenames are different, they are assigned different
IDs. However, with the help of symbolic links only one copy of their contents is
stored on hard-disk. For all intents and purposes, such as DeleteFile method calls,
the files are considered separate.
Because all files and pixels are mmaped and shared between omeis processes,
this implementation will benefit significantly from lots and lots of RAM. The
RAM will be used by the OS to cache the mmap'ed files using the CPU's virtual
memory system, and paged in and out of disk as necessary. The omeis processes
themselves have a minimal RAM footprint even when dealing with very large
files, so lower RAM will only hurt performance, not functionality.
