Agenda

Remember: Agenda must be complete (with estimated times) on the day before the meeting. Any additions after that must go at the bottom (AOCB)

1. Accepting minutes from last meeting

2. Introduction of visitors:

   • Dr Jos Koetsier
   • Prof Malcolm Atkinson

3. Matters Arising (<10 mins)

   • Melissa: adjust version number and backport
   • Ola: writes tickets/test about search issue
   • Chris: test web startup on Windows at home
   • Ola: fix or remove enum tab
   • Everyone: Go through delete scenario (21)
   • Andrew: talk to Ruben about ome-tiff options - done

4. Analysis, distributed and otherwise: Donald & Chris's presentation (45 mins)

5. Any other business (<5 mins)

Notes

1. Accepting Minutes:

   • Accepted

2. Introduction of Guests:

   • Dr. Jos Koetsier:
   • Prof. Malcolm Atkinson:

3. Matters Arising:

   • Nothing new (moved foward to Thursday)

4. Analysis Presentation:

   • Donald & Jos have been working on distributed analysis 'on the side'
   • Introduction to 'what is OMERO'
     • Centralized and ubiquitous format/importing of many microscopy data types
     • remote client interfaces (java and web) to allow visualizing and analysis
     • scripted and distributed analysis of data
     • Problems:
       • Structure image data
       • Annotate analysis results
       • Attach results to images, datasets, etc.
     • Solutions:
       • Structured annotations using name spaces
       • Storage of tabular data (HDF)
       • Interfaces for other types of clients (matlab, C++, Python, etc)
     • Challenging Applications:
       • High Content screening,
       • Heavy image processing
       • Big image
   • Schemaless
     • Malcolm: some users will want a schema
       • Is it possible to reference other databases? (Not currently)
       • Donald: NoSQL may be the way to go here
   • We are also looking at ways to lower infrastructure prerequisites (HDFS, Hadoop, etc).
     • Malcolm: A lot of labs using Condor
       • Chris: true for computationally aware labs. We also want to deploy to those who aren't
       • Malcolm: Manchester has a nice setup with Condor worth checking out (dropbox analysis)
   • Distributed Processing
     • Currently using ICE Grid (which does not provide access to a cluster)
     • RAPID provides submission, monitoring, control 'for free'
   • Alternative Storage
     • Donald: Looking at ways to store data with documents, distributed storage/clusters
       • Malcolm: likes MonoDB
       • Malcolm: have a copy of SciDB
       • Malcolm: Might be worth talking to Microsoft research (interested in this problem)
       • Malcolm: building own machine to get more transfer bits as opposed to FLOPs
   • Couch DB (long term goal?)
     • Offers:
       • online/offline replication
       • inconsistency resolution
       • durabiltiy
       • in the field replication
       • clustering
       • annotation propagation
     • Malcolm: Standard problem is being I/O bound
   • Using RADID as an entry point to cluster (short term goal)
     • Use it for FLIM & Deconvolution
   • Better integration of RAPID into OMERO (mid term goal)
     • Malcolm: making use of "Saga" software (?)
   • Malcolm: ok to copy data to Edinburgh for testing? Yes.
   • Future Needs:
     • recovering/checkpoint mechanisms
       • Malcolm: Control database might be the best way to do this
       • Josh: Some overlap there between OMERO jobs and RAPID jobs
     • throttling (smart submission) to prevent killing the server
     • need parameter fine-tuning (unknown-unknown)
     • Malcolm: decide on which compute platforms we'll target
   • Conclusion (immediate todos):
     • Short term: need to get distribution working
     • Jos will join next meeting with David (Jean-Marie will organize one)
     • Donald will get Jos the script as it exists now
     • Donald will get Jos the user 'workflow'
       • Jean-Marie: Generated CSV files
     • Script will handle parallelization of images
       • Josh: interested in looking at those APIs / defining those boundaries
     • Gaol is to have this phase done by end of November

5. Other Business:

   • See forums: (Ivan for CMU), ome.tif (Brandon Scott), Icaoberg for Josh

Action Items

* Andrew: Move all matters arising from last week to Thursday