Personal tools
  • We're Hiring!

You are here: Home Community Minutes Meetings June 2007 Dundee Team Meeting

June 2007 Dundee Team Meeting

Items for Discussion and Resolution



Introduction-- why are we doing this

This meeting is to define activities for the OMERO and Usable Image teams over the remainder of 2007, targeting two major releases, Beta3 and Beta4. With the release of OMERO3-Beta2, we want to leverage the foundation we have built to provide a commonly used image data platform. Meeting will be all day June 29, in MSI BoardRoom. Aim to assemble at 9-- we have some room set-up to do, so we will try to start by 9:30.

Verbal feedback I have received suggests that our Feb 2007 Mtg was a bit to structured, and people liked the more free form Nov 2006 Mtg. So we'll try a more free form agenda and see how that goes. Note that we have a few more people, and alot more to get over....

Something to Note:

At our Feb 2007 Mtg, we cam up with a great list of to-do's, alot of which remain to be done. See especially the Major Functionality section. I have copied that here, and annotated what I know about them. Please update or change the presentation, as you see fit.

Red type indicates functionality missing in Beta2
Green type indicates functionality existing Beta2
This horrible color indicates is in progress

Note this is not to beat anyone up-- WE HAVE DONE ALOT IN Beta2!!!, but this is just to provide a starting point for discussion.



Major functionality for OMERO 2007


OME Data Model Updates

  • Transfer OME-XML Evolution to XML, database models (uml)
  • OME-XML, OME-TIFF validators
  • OMERO Data Model updates to support analytic types in OME Server (put off?)
  • SVN control

Import

  • Identify 5 formats from Bio-formats that are easiest and most important to support and get them in
  • PDI Updates
  • Resolution of importer/insight separation
  • Server-side import, including upload from client.

Data Export

  • Original Files
  • OME-TIFF (implies OME-XML)
  • Download of binary objects from DB (to support structured annotations, so still very crudely defined)

Data Migration

  • Backing up and restoring a database
  • Moving from one DB (e.g. hsqldb) to another (psql)
  • Moving from one vesion (Beta1) to another (RC1) !!
    • Vital to be able to change the DB (Add ProjectAnnotations, e.g.)

Rendering

  • allow input window out of range for images with same pixel type.
  • Compression for reduced bandwidth requirements
  • Projection (to be stored as new Pixels?)
  • .mov, .avi, .jpg, .png-- store in DB, deliver to client
  • support for defined ROIs of Pixel data (to support client measurements?)

Server

  • ICE, for better client support
  • Easier mapping of data model updates (from UML/XSD) to Hibernate, pojos, etc. (Formerly discussed as "EmfAsDsl" but now planned to be implemented via XSD-fu -Sep. 2008)
  • Lucene, Compass for indexing
  • Save, delete objects linked to PDI, CG/C/I
  • Model Validation (Currently a significant percentage (20-30%) of the model UML is NOT implemented in the server. These are constraints that don't take place at the SQL level and need code. This could possibly go hand-in-hand with Josiah's idea of various spec-levels. (Josh) )
  • Batch operations: supporting HiViewer multi-selection, to include annotations, cascading deletes, processing, etc Might have to reverse this concept, making a separate service that is batch manager, client of server. This needs some study.

Insight

  • Server-side related: Permissions strategy
  • Components-- move to new environment?
  • Given the functionality we want to add in 2007, is Java the right environment? (Obviously, this is the wrong question-- maybe the right question is, how do we support a plethora of clients tools under a single umbrella?)
  • Viewer
    • 3D viewing
    • Annotation of 5DROIs
  • HiViewer-- multi-selection and Actions/"Do Its" so that a selection and be sent for batch operations
  • HiViewer/DataManager-- rationalization of functions for finding, querying-- what is the portal for metadata query?
  • Major expansion of metadata viewing, editing, querying; must define how to show all, but layer so most important is easy to get to.
  • Analysis Functions
    • Intensity measurements in defined UIs
    • Save as object (probably CSV file) linked to Pixels
    • CellProfiler Plug-in

Community

  • OME Registry-- log for all "up" OMERO (and OME?) servers
  • Bug reporting for OMERO

Usable Image

  • Update of website and vision
  • Definition of metadata usage (CRITICAL!).
  • Output from PP workshop
  • ????

June 2007 Dundee Meeting Topics


The goal ofthis meeting is to define goals and milestone for the Beta3 releases of OMERO.server and its clients.

At this meeting, we will:
  • define Beta3 and beta4 and thus move closer to defining RC1
  • Ensure all tasks are ticketed and assigned
  • define future interactions between OMERO, Shoola, and UsableImage.


Draft Programme (please make comments etc as necessary)

TimeTopicLead(s)Comments
09:00Coffee, setup, gossip
09:30Summary of StatusJason, Catriona
09:45Server Summary and WishlistsChris & JoshInstall, WebAdmin, Deletes, Workflows, Docs
10:45Data ImportBrianDo we need to think beyond new file formats-- structured annotations...
11:45Insight Summary an WishlistsJean-Marie and DonaldRaising the bar for client interaction; remote application integration
13:00LunchAll
14:00-?Defining Beta3, Tickets on trac, Notes on Tiki, Docs on UsableImage




June 2007 Dundee Meeting Notes

  • In Attendance
    • Brian, Donald, Will, Josh, Jean-Marie, Jason, Chris, Catriona, Ola, Paula, Andrew, Xinyi, Scott



  • Notes
    • Jason intro
      • Beta 2 is out!
      • Presentations are going well and we're getting more responses and questions about OMERO!
      • A lot more talk about allowing our managed data going "live to the world" through OMERO!
      • We need to support more import formats!
      • On track for where we need to be for end of year
      • Work on visibility (see downloads; number of active servers)
      • Download counts: 217
    
callan@necromancer /var/log/apache2 $ grep 'omero-3.0-Beta1' cvs.openmicroscopy.org.uk-access_log > /tmp/downloads.out
callan@necromancer /var/log/apache2 $ wc -l /tmp/downloads.out
      217 /tmp/downloads.out

      • Tracking citations: http://www.scopus.com/scopus/home.url
      • For papers — "All images prepared in {photoshop|OMERO|...}"
      • Office situation: Nothing concrete just yet (being worked on) (general admin issues)
      • Hardware/software-meme across Scotland (supporting imaging and proteomics) with OMERO included!
    • Catriona
      • UI team is doing a biweekly meetings with Homburg now
    • Server
      • need better usability for sysadmins and developers
      • Beta2: solidification, performance issues
      • More loose ends in beta3 to tie off
      • Do we need to draw the line in the sand on 3.0? (No session mgmt??)
      • Discussion on how we define our versions.
      • A round of quality assurance (and defining our process for that) would do us good.
        • Chris: This is what we've been doing for all of OME
          • Push down activation energy (3.0-Beta2? 3.0?)
          • Paris: But I don't want to double my data (indexing) (3.0-Beta3? 3.1?)
        • Josh: But we need to start growing up
          • Like to share that process (define this?)
      • Get away from the naming differentiation.
        • What's minimal core that has to go into release?
        • And then let's work at maturing the process.
    • White papers (monthly?) (careful of original publishing on the web). See ZeroC Newsletters
      • Also, movies! (people really like that)
      • Each month a screencast? (on YouTube!)
      • More Howtos
      • Workflows
        • Touches on what most of we are working on
        • What can we get into the first work packet
        • Scripting with matlab and indexing gets a user just about everything they want
        • Not having to transfer the pixels back and forth is KEY!

    • Import
      • 7 different formats. More in the pipe to support Homburg
      • Get clear: where will Melissa be involved
        • Metadata support in bio-formats
        • Testing in OMERO rather than ImageJ (OMERO requirements and workflow supported)
        • This comes back to the maturity of our development process. (Defined, documented, tests,...)
      • Dicom seems to work
      • Feedback mechanism is working (add button "make trac")

    • Client
      • Huge usability improvements
      • Focused on data manager. Collaborative work.
      • Users more want to change permissions at the container level (not really at images)

  • Actions
    • Server readme file needs a small update for 2.1 next time!!
    • Go to Italian restaurant "OMERO" in Firenze: Trattoria OMERO
    • Trac site rework
      • Lets make sure we have a good viewable link on the main OMERO trac page to lead folks back to the main website
      • Design, logos, etc.
    • Need to address the admin support situation!
    • Need papers for rendering features (and other things that can be sighted often!) (mentioned in insight?)
    • Start producing white papers/videos
    • We need to state: What is an OMERO beta, what are the assurances we give for upgrades? (if any)
    • We need to mature the release process
    • Define process for after next package
    • Get Jason to call all his friends and get us file formats!!



  • Brainstorming Idea
    • Would be nice if we had "auto updates" of the client
    • Would be nice if we could target some of the individuals that have specific needs (ie: clients). "Support services" individuals?

  • Come back to topics
    • (Before we start down this list, decide what to cover today and what should be for separate dev meeting later)
    • Key for today: Where are we going to be at the end of the year?
    • proteomics data
    • Wireless access / compression
    • Office situation
    • Admin situation
    • Sys admin usability for the server
    • Managing the SVN server better!
    • When do we go into release candidates?!
    • Having a quality assurance round is absolutely needed! How do we do that?
    • Archiving feature: Would it be better if this was a more traditional "backup" feature?
    • Discuss maturing the release process (SVN, releases, etc).
    • Which parts of OME are we going to integrate next?
    • rethinking use of trac...
    • What happened to the "tagging" feature that was discussed for classifications?



  • Server Tasks
    • Hibernate, Spring, and JBoss updates
    • Decide what formats and how will the compression library work?
    • How much of the blitz functionality do we support? Do our clients use it?
    • Integrate Ola's LDAP stuff
    • Making sure Donald has what he needs for ROIs (Storage)
    • Develop screening model
    • Decide indexing (which pieces do we go after first)
    • Copy rendering definition settings between images
    • Set starts and ends in images!
    • Remove the restriction in the rendering engine (off by one's errors, arbitrary starts and ends)
    • Punted: Think about multiple rendering definitions per user per image (how do you manage your renderingdefs, defaults, etc.)
    • Punted: Synchronization between viewer settings and thumbnails
    • ISession
    • JobHandle — simple bundles which you only have to drop on.
    • DeleteTransaction
    • BatchOps
    • ILink?? and other hierarchies
    • registry of active servers (condor as an example)
    • webadmin exporting/importing users
    • webadmin configuration settings for ldap
    • webadmin bug reporter/help
    • ldap - integration of various user source
    • integration with unix password (automatic import, synchronization)
    • way to handle logging of errors (over webadmin?, syslog?)
    • webadmin configuration for rest of server
    • Need OmeroConfig / way to store configuration under /etc
    • Projections
    • Query builder (HQL isn't always the best)
    • Commandline getting images: omero login; omero png -id 1 -z 2 -t 5 -x 5 -o myimage.out # OMETiff/raw binary/
    • More accounts on valewalker (Chris)
    • macosx server dmg needs to be updateable
    • usability expert for server-side (sysadmin)
    • usability of languages against the server (dev)
    • Log all google queries to get a feeling for what "they" want to know
    • Josh request search columns from everyone
    • Josh request attribute opinions from everyone

  • Client Tasks
    • Be able to choose who can control/view my data (ACLs)
    • How are we going to handle classification (tagging) system? Needs rework
    • Annotations / Attachments (see "Attributes" reference below) (perhaps only storing a URL)
    • Threading (forum-like) of annotations from other users
    • Other starting points (for people with derived images, structured annotations) — in insight, but also browser
    • client substrate (ways for them to communicate)
    • some way to update metadata
    • Push ROI tool: where to manipulate the raw data
    • what is the min requirement for machine. Need to specify. Also caching
    • Deployment of insight and importer and their integration - sharing logs, libraries, and configuration (LHF)
    • Move forward HiViewer
    • comparing two difference sections, ... (browser)
    • Minor fixes for DataManager, or move functionality into HiViewer
    • "Real" Desktop Integration
    • Analysis and manipulating data server side (let's do things outside of ROI tool..."Meta-assays")
    • Define relationships between ROIs (parentage, history, ... UML-like)
    • Access to pixel data for doing analysis somewhere
    • Structuring ROI into something more than XML (incl. relationships)
    • Accessing ontologies. Integrate MIACA with protocol editor and GENE ONTOLOGY with the data model? Look at PHENOTE!
    • Expanding the number of hierarchies of the data manager
    • Keyword driven (Web 2.0) file system (Tagging!!)
    • Excel integration
    • Define workflow to add the metadata (web page)

  • Import Tasks
    • Set up OMERO for Melissa
    • Prioritize the formats she works on
    • Have her set up the importer and test all new formats
    • Mature her testing and development processes
    • Build a really good test image set
    • Command-line version
    • Separate gui and library
    • Every format --> OME-XML --> validate (full end-to-end)
    • Export to other tools like Aperture to Photoshop

  • Feedback Engine features
    • Fix it up a bit more (cleanup)
    • Add a feature to allow feedback items to be turned into trac tickets
    • Search functionality
    • Server feedback mechanism.

  • Admin Tasks
    • How do we organize our weekly work approach (UI and Dev team)
    • Admin usability: paula and scott??

  • Protocol Editor Tasks
    • Adding extra files/folders.
    • Development cycle for Protocol Editor

  • Usable Image
    • Redefining u.i. process for the longer goals (tagging,...)...richer, deeper bits of work. How to work with devs EOF
    • Tagging EOY
    • Role of HiViewer
    • Bits of development in insight for beta4, so that u.i. team and can start doing prototyping during beta3
      • guys here for design workshops rather than weekly reviews
      • they are now expert users, rather than "fresh meat" (re: homburg, paris)
      • remote users.
    • ProtocolEditor -> ontologies
    • Start rationalizing the collaboration across the project
      • Design workshop with the developers (not users), OmeroTesting alternative
    • Extraction and presentation of u.i. information (documenting workflows, re-evaluating scenarios, ...)
    • ROI Tools
    • Cross-platform tools
    • Paula doing history "measurable benefit"

  • Model
    • Removing (deprecating) our CGCI hierarchy.
    • Flag to Project as tag.
    • Getting changes into order (SPW)
    • Updated OME-XML example files and example code.
    • XML-Validator (with the specifications)
    • Workout difference between Regions, ROIs, and D.'s ROIs
    • Get rid of the existing OMERO ROIs
    • Pixel data external to the metadata (OME-XML files without pixels data)
    • CustomAttributes (GDIR) --> What else/all can we apply these two? What can they (the attributes) be?
    • "Introduction to the Data Model", other top level explanations (difficult to find things on web)
    • Linking to MIACA (re: SPW)
    • Further cleaning on the website
    • XSD version at http://www.ome-xml.org?
    • Clear, versioning of XSD (with working version for developers to get ahead)
    • How to add things to DB (is it a model?) — rating, downloaded link.
    • How to remove things: Unsupported model items (the extant rendering/RGB stuff)
    • Lucene/search
    • EmfAsDsl--code generation of the model and database
    • Code validation

  • DB
    • As easy as ruby on rails (db upgrades, webpage templates, ...)
    • Easy install (HSQLDB) -> easy macosx install for all users
    • Which dbs do we want to support
    • Decide what dbms do we want to support

  • Documentation
    • Would be nice to have an "Is OMERO for me?" check box sort of page
    • Trac page on server requirements.
    • Adding screenshots, workflows, personas, scenarios into subversion (shoola or server)


  • OMERO's Elevators Pitch
    • "At the end of the door, if we aren't helping scientists do their work...The whole thing we're trying to do is ... help them get disease and drug insight quicker." Will:"Publication". Cat:"How does what you're selling me, help me do that." Brian:"I write visualization for life sciences." Cat:"But why should I buy yours." Andrew:"Because it makes use of the OME Model." Brian:"There are 100 different formats in medical science. Our software tries to bring them all together." (Integration. Accessibility. But not better than anyone else's.) Collaboration...Doesn't cost anything...Allows scientists to work at their home or office, and not workstation (Portable). Freedom from your data. Biologists spend a lot of time acquiring images (doing experiment), and then a lot of time in front of screen...grunt work. (Automation) Real scientists helped to design it. Searchability of vast amounts of heterogeneous data. Collect metadata.



  • "What do we Do/What are our Goals"
    • Set the Standard - OME is becoming the standard format for storing microscopy imaging data
    • Collaboration - Allow scientists to share their data
    • Portability - Allow scientists to use their data anywhere they have net access
    • Automation - Allow users to automate and speed their image analysis
    • Accessibility - Allow users to import a lot of different data types
    • Interoperability - Allow users to interoperate with other science software (ex: Cell Profiler)
    • Archive - Allow users to store their data in one "secure" location
    • Manageability - Allows users to find, access, send, re-arrange, annotate their data. (includes search functionality)
    • "Enable discovery" -
    • "World leading provider of this software at any price" -
    • "Quickly becoming a standard"
    • Ease of Use - Allows the users to get his work done without a lot of fiddling. (Also easily installable. cF. with biorails.org)
    • Additional Comments:
      • Chris: There are lots of word processors. It's in the details. What makes me now use imagej, disk, and my labbook.
      • Catriona - Hard to believe the amount of time scientists have to spend messing around with software.
      • Will: What is the output of OME? Nice pictures from different wavelengths, what else? Would like to classify and build histograms, etc.
      • Donald: We allow you to do measurements, annotate what they are, these can be exported to excel, then can produce histograms etc.
      • Brian: We don't really think much about what people want to take out from the system - more what they are putting in.
      • Chris: Which users are we really trying to help? What technical knowledge will they require to use the system? Power user & the user who just wants to use the system without knowing too much details of how it works. They just want to get results out.
      • Brian: Data driven design - very successful. Chris: does it work for this environment?
      • Jason: support various applications in various institutions.



  • Who are we building this for?
    • The ones who do a lot of visual data management or the ones who just want to compare image A with image B? Different labs have varying levels of support.
    • If you try to sell to lab NoI.T., then you can still sale to LotsOfI.T.
    • Tagging (HiViewer showing more data), search (creation data, username, ...) get us pretty far.
    • And if a scripting person can get something out for imagej,matlab,excel...
    • Lay foundation for the more complex analysis use cases (like Aperture to Photoshop)
    • We would like to support "Quantitative Systems Biology" - users who need to use and analyze multiple images against each other.
    • Jason's lab has scientists who are interested in computing, other labs don't have these people, if the program doesn't work easily it won't be used. Usually not much technical support available, the system needs to be able to work out of the box.
    • Additional Comments:
      • Jean-marie : We also need to target what type of images we can support. Need to identify what is the most common (of the images we do not yet support). Now support more than 50% of file types.
      • What is a typical lab in terms of developers & expertise?
      • Chris: targeting 95% potential users but who might not be able to load OMERO on their server.
      • Jason: future funding is mainly from academic bodies, instead of commercial companies.
      • Brian: users need a certain expertise on servers - for stability.
      • Jason: make it clear about the server requirement on website.
      • Do we need to be integrated with excel? If so to what level?
      • Edge software sales pitch : "Our solutions are designed to reduce the cost and effort associated with software in these areas. They require less IT support and are adaptable enough to expand to meet the changing needs of your business".






  • Suggested Future meeting formats
    • Opening overview of the project
    • Brainstorming
      • Individual "To Do Lists"
    • Prioritizing the list
    • Scheduling the list to release points.


The OMERO sales tag: "Show me the data" : Full workflow for an image data management system.




  • Beta3 "Todo" List:

    • Concentrate on the HiViewer
    • Undo step by step
      • ROI
      • Thumbnail fixes
        • Display large numbers
        • Syncing
        • Thumbnail centre on screen
        • Lazy Loading?
    • Importing Features
      • Imported metadata validation
      • "Imported Today" Feature
      • UI/Engine split
      • Testing framework
      • Command line feature
    • Investigate tagging with users (perhaps during import dialog?)
      • smart tagging (review other systems - smart albums, clouds, tags, etc in other apps)
    • Server Features:
      • Rendering Engine stateful service update
      • Compression services
      • Copy/Paste rendering settings across images
      • Bug reporter / Helper for WebAdmin
      • Integration with external sources of users data (Ldap, files, etc.) - import/export
      • Registry (1 month) (TA)
      • House cleaning (upgrading JBoss, low handing goodies) (2 weeks) (TA)
      • API for external client (such as matlab) to get metadata/images in/out (3 weeks) (Donald will ticket)
      • Search functionality (2 months) — do we just index the files (TA in GatherReqs awaiting Will's findings)
      • Place for storage requirements for configuration (for example for ldapTemplate) (3 weeks) (TA)
      • Attributes/annotations (1 month) (TA)
      • Josh will add tickets
        • SessionMgmt (3 weeks)
        • Delete service


  • Beta4 "Todo" List:
    • JobHandle/Workflow (server-side analysis) (TA)




  • Abbreviations
    • LHF = low hanging fruit
    • EOY = End of year
    • TA = Tickets added
Document Actions