June 2008 Developers Meeting

Developers meeting held in Dundee

The goal of this meeting is to define goals and milestone for the Beta4 releases of [old-link http://trac.openmicroscopy.org.uk/omero/milestone/3.0-Beta4] OMERO.server and its [old-link http://trac.openmicroscopy.org.uk/shoola/milestone/3.0-Beta4] clients.

At this meeting, we will:

define Beta4 as well as any next betas and the RC series
ensure all tasks are ticketed and assigned

Draft Programme (please make comments etc as necessary)

Day 1 (June 20th)

Time	Topic	Lead(s)	Comments
09:00	Welcome Intro, Summary
09:30	Bio-Formats OMERO.importer	Melissa Brian	Status and evolution
10:30	File Formats	Curtis?	OME-HDF
11:30	Data Model	Andrew	Evolution, Image/Pixels
12:30	Lunch
13:30	OMERO.editor	Will	presentation+ Data Model from Users point of view
14:30	OME server	Ilya	status
15:30	LOCI Apps	Curtis	status
16:30	Discussion	??	extra time

Day 2 (June 21st)

Time	Topic	Lead(s)	Comments
09:00	Coffee, setup, gossip
09:00	Usable Image	Catriona	Beta3 status, evolution, methodology
10:00	OMERO.server	Chris Josh	status
11:00	OMERO.fs	Colin Chris?	demo
12:00	OMERO.scripts	Donald	status and evolution
13:00	Lunch
14:00	Messaging server	Lars	proposal
15:00	OMERO.insight	Jean-Marie	status evolution, integration
16:00	OMERO.web	Ola	status evolution

Day 3 (June 22nd)

Time	Topic	Lead(s)	Comments
10:00	Discussion	Jason
12:30	Lunch
14:30	Plans	Jason	tickets, website etc.

Notes

History/Intro
- Jason & Ilya at all
- First Dundee: 2004...
- Goal: get what we want to do for the next year or so
- catering: lunch, etc.
- have to talk fast
- looking at downloads, registry
  - New Chinese dot on registry
- Grants pending: 7 positions from Welcome Trust, Usablity for 6 positions and a PhD
  - Funding for Baltimore 2 FTEs until review next year
- Everyone healthy/happy?
  - Getting to Jason's tonight: taxi, etc.

Formats
- Brian
  - Q: Is an import a copy? A: Yes, but usually the history feature is used to come back (after bug report) and retry fails
  - Curtis: we're going to be using the ImportLibrary directly, too.
    - Have been some build issues.
    - Would like to see the newest version always compatible
    - Curtis: Try to not just change things every: "I'd like to change filters now"
    - Brian: Would be good to look at how we communicate
    - Curtis: Fine with changing OMEROMetadataStore so that it compiles with stubs
    - Chris: In the past, changing of methods is trickier than adding new methods
      - Legacy code which mismatches with the code-generated bits.
    - Need to setup automated testing of Bioformats jar in importer
  - Q: Advantage of server-side import? A: speed, just uploading file, getting files off the scopes
  - Importing at Pasteur: caching server (1TB), processed daily to long-term storage
    - dummy client on cache server to import
    - Q: option on OMERO.fs? A: Handle the upload
    - Jason: Hard-core reality. Got to get off the scope, initiate import and run in background
    - Brian: Would like to move import to service which starts a daemon
    - Has to be light-weight process: ftp file over, then control another service on the server to deal with OriginalFile
      - ImportDaemon (registers with OmeroGrid) and ImportService
    - Can't install anything on the scopes.
      - Single user environment
      - Might need OMERO.fs/SMB mount using no install (PUSH)
  - Curtis: new formats is not necessarily importer-dependent
    - No reason we can't expand everything to support new formats
    - Ilya: Have to be a bit careful here
      - Real lack of information management in science
      - Don't want to become that. Will unfocus everything.
      - "You guys deal with images. Why don't you handle ....?"
    - Jason: FCS - time-series data merged with image. But it's microscopy, so...
    - Ilya: If you are the only ones doing it well, ...
    - Curtis: Step-by-Step. Start with what we want.
  - Cross-language support
    - Curtis: We can extend JVMLink
    - Josh: What about Ice?
    - Curtis: Sure, what about deployment?
      - Now we don't need our scripting language: use beanshell
  - Chris: We never imagined people downloading and haxxoring importer from svn
    - Need to update for and support that
    - More flexible, support interfaces
- Melissa
  - Q: Performance not infrastructure based, right? A: No only formats (ND2)
  - Specification: will always want to be adding more (electrophys: AUDIO). There's a tension there.
    - Need audio attached to a video talking about the image
    - "soundtrack"
  - learning project uploading to OMERO
    - Nothing needed at the moment
  - OME-XML
    - Currently OMEROMetadataStore is write only. Need to implement the interface
  - Curtis has been thinking about GUI for converting between formats
    - Maybe not anymore. ImageJ has something now: VirtualStacks (check option in importer)
    - Had wanted to save RAM.
    - This may be just improving ImageJWriter
  - Dev documentation
    - Brian: same problem with the importer
    - Ilya: "Keeps the second-rate developers out".
  - Loci update
    - Jason: previous site there was a list "You can use this software with ..."
    - Curtis: Now under "Applications"
    - The visual — big, blue — was great
    - A big money line (screenshot)
    - Jean-Marie: More explicit under the metadata section in the format list
      - Good for building a client which supports the metadata
      - Chris/Donald: was using OMERO from matlab to dump, work, push
        Need a way to know what is supported
        A trac page where each format has its metadata-support listed
        Could also link the commit which supports new metadata ("r4096")
      - Curtis: a lot of work, perhaps new interface (mirror of MetadataStore)
        isThisSupported() - one per property
        default is false
        the code documents everything that is supported
        write a script which would generate HTML of what is also supported.
        Chris: generate Trac code.
        Chris: Could it be n a single data file. Code is a bit of pain. Good for understanding.
        Brian: people ask often "do you support field X on Y"
        Josh: possible to have coarser interfaces which short-cut checking all the properties
        Curtis: Can enumerate the properties supported, or list Image to say all of Image
        Donald: Would really like to see support: "Image", "Instrumentation", ...
        Andrew: Brian had asked for matrix for what populates which tables
  - Report from the field
    - lifetime imaging, high-content
    - "What we really need is support for Becker-Hickel (sp?) format?"
    - He knew: STD in bioformats (even though not dev), AND that support in bioformats had been turned off
    - "Why?" --> "Test-data..."
- Curtis
  - OME-XML
    - Writer
    - Expanding metadata store
    - Staying on same page. What omero supports and if we're converting?
    - MetadataStore as the glue workhorse
    - OME-XML which doesn't validate because of incomplete data _NOT_ in the proprietary file
      - What are we supposed to do?
      - Want to be able to fully populate everything that I have!
      - Our software is producing non-validating code
      - Not ok.
    - See [href="http://cvs.openmicroscopy.org.uk/tiki/tiki-index.php?page=ConferenceCall+2007-12-27&highlight=objective] http://cvs.openmicroscopy.org.uk/tiki/tiki-index.php?page=ConferenceCall+2007-12-27&highlight=objective
    - We need a plan, document, and implement
    - Brian: prioritize this, please.
    - Josh: read the thread for the propsed suggestions
    - Jean-Marie: We can compare an example from Will later to make this complete
  - OME-TIFF
    - Reworked the reader
    - Multi-file datasets: validator doesn't validate across files
    - Ilya: suggestion from NIST for some kind of naming convention of microscopy images
      - Especially in multi-file formats
      - Curtis: Can only ever be a recommendation
      - Chris: Do the microscopy producers care? Not in that realm.
      - Not in that realm because there are no guidelines
      - All instruments in some domains come with a certification for a guideline
    - Write up recommendation based on LOCI
    - But the current solution about embedding the file names is pretty nice.
    - Josh: Putting a flag in the file? Not really.
    - Add to golden start system
    - Josh: Possibility of someone accidentally/purposefully breaking the regex
  - OME-HDF
    - re: Matt Dougherty
    - Beyond the 5D model
      - On the pixels
      - Rasterizing on the Channel dimension, with info on how to cut it up
      - bioformats supports N dimensions, "subC" with rasterizing
  - Other
    - JVMLink -> Ice
    - Viewers/converters in ImageJ
    - Documenting what metadata is supported
    - Jason: something on the grant for someone working with formats
    - Curtis: several students wiscan, jvmlink, flow-cytometry, visbio would profit from a formats guy from grant
    - Jason: What does the plan need to look like coming out of the meeting?
      - Curtis: Who, what, when?
      - Monday as post-conf coding day!

conversation about communication
- possibility of using Adobe for whiteboarding, etc.
- unconference-style to get through all the points we want to discuss (hdf,...)

Data Model/Andrew
- ROI
  - Curtis: Have you referred to the notes on ROIs
  - Curtis: Would like input to compare imagej & visbio
  - Ilya: Is there no requirement for the object to be closed?
    - A: Not currently. There are closed shapes.
    - Ilya: There must be a way to determine if a fill is possible.
      - Draw a rectangle with 4 <lines/>
      - Donald: Advantages if you define it with <rect/>. Do queries on the types.
      - Andrew: The intent was to use the primitives provided by OpenGIS (http://www.opengeospatial.org/standards)
      - Ilya: Good to use some form of standard and tools that people understand
      - Donald: surfaces, ...
      - Ilya: are there anything not translatable?
      - Not a Z toolbar in google earth? But there is a time concept
      - time-series data? Yeah, but our binary data is a different order.
    - ROI structure here is completely unnormalized
      - Curtis: Would like to have some "share" concept.
      - Andrew: null z or t means over whole dimension
      - Curtis: fine.
      - Josh: could use an annotation if you really wanted to.
    - Ilya: Have to calculate the continuity
      - <inverse/>
      - SVG does allow for defining the orientation ("fill left-hand side", vectors)
    - Doesn't allow for two lines in z
      - Jean-Marie: use the derivation to say that they are an angle
      - Curtis: Just attach a 3d mesh
      - Donald: use "connector" to connect two ROIs
- Relaxing the model
  - Omitting Pixel Data
    - "open metadata environment"?
    - <metadatafile filename="..." hash="..."/>
    - Ilya: PixelsRef with LSID
    - stylistic issue
    - Josh: Let's use pass XML over an API, dump whole DB to a single file as backup
    - Brian: Different representation of pixels, compressed
    - Ilya: you need the pref
    - Curtis: Have the <MetadataFile file=""/> be anything that bioformats can read?
    - Chris: We originally didn't want to do the metadata companion files
    - Jason: we ate that poison pill when we did OME-TIFF
    - Josh: Gets worse when we do HDF
    - Companion file is not a starred profile
    - Capturing extra metadata on the scope machine
    - Single companion file for multi-binary files
    - Putting the carrots on the main page
    - doing the link at the provenance level (OriginalFile: STs)
      - Not in model, nor is provenance (only in XSD)
      - Consider it without server concept, pure exchange format
      - non-image extra files (audio, etc.) is fine
      - Curtis: Want <plane/> in <pixels/> so it can't be the bit that disappears
      - Just make <bindata/> optional.
      - DICOM flag: GE doesn't read SIEMENS doesn't read ...
      - temporary LSIDs pointing to /dev/null

Relaxing
- enumerations
  - Josh: like omero, using a reference
  - Chris: how to handle the resolution?
  - Ilya: modelling discussion. Enumeration or not.
  - Just define the process for defining a new enumeration.
- Curtis: I understand what you are saying that people will abuse "Other"
  - "Validation does not equal quality"
  - Multiple Quality Levels: validates but did you do a "No-No"
  - Will solve LOCI's problem.
  - Jean-Marie: Interesting to see what Will is doing, need to look at whole pipe-line
    - Need a formal way of describing, visualizing, ... Otherwise develop own way
    - cF. Compare XRay tomography's own OME schema
  - Chris: 30+ formats and none of them can fill out the objective
  - Ilya: Objective wasn't done arbitrarily
    - Was defined to show future manufacturers how to define objective
    - But we still need to deal with it
    - "7 of 20" but we need the definition. Once all 20 of 20, then you drop the optionals.
    - Sociology/social engineering
    - Ilya: A service we provide. Obvious that I need one number to go from provisional to full. Gold star!
- Cases
  - Objective,
- OME-Provisional, "Level -1"
- Who's going to do the evaluation? Put up the table with examples of each ranking.
  - Curtis: for each format someinfo
  - for every property (entity shortcuts): N/A, NO, PARTIAL, YES
  - format quality ratio: is it supported? low == not rich
  - each property:
  - weighting the properties, importance of properties, level of OME
  - four stars : import level, publication time, ???
3rd party
- Andrew: want a people to look at the definitions that we have and pick one
- Not just an <any/> yet
- hook a block one level lower
- pdf attached: what does the import/export workflow look like
- Ilya: have to be careful where the original format is important (pdf/mp3/...)

will/editor
- Ontology lookup: Curtis — going the right direction with that
- Ilya: results of timed experiment show up with PI on date for review
- future searching: Ilya — post-processing the returned protocols for hierarchical structure
- Curtis: Is there a way when they create a new block "Filter" using the OME model?
- Are allowed to link to another file but not to another element?
  - Some use cut-n-paste.
  - Josh: what happens if only one file gets copied? A: Only local link.
- Ilya: You have the MAGE block. Have you tried...
  - They have a nice description of a biological experiment
  - Problem: terminates in a microarray assay
  - But Treatment, protocols, incubations, ... new reagent.
  - Recursive: treatment, ... on new reagent
  - Data collection, instrument, Merck index entries
  - Reason: if you know enough about how you did the experiment, you can draw inferences (computationally)
  - Jason: Defining small parts of structure, and use those
  - Start typing "LightSource" template is applied, then the list gets computed, ...
    - MAGE is only useful if there is a redbook on the server
  - Summary
    - Easy to fill out template of : instruments, ...
    - Search facility for : "light source",...
  - Making this easier for fluid-handling robots like Ilya
  - sites where you have to fill out a protocol before you can use the scope
    - have to institue draconian measures because digital notebooks are tougher than paper
  - . first step is having things to cut-n-paste
  - . then point to other templates (perhaps data model)
  - data model: providing (file type) a list of filter sets.
  - Ilya: Has to be a carrot. Will: The only rule has been to make it easier to use
  - Andrew: Is the piece of paper a critical step?
    - Print and be remembered that it was printed
    - "Do you need to update anything?"
    - "I see you haven't finished your protocol"
    - mini-mobile (iphone) is a possible idea
    - have editor generate html which interacts with the server

ilya
- "solving an imaging problem one at a time is not a solution"
- Q: Any other tissue based (as opposed to hormonal levels, running through mazes) to differentiate mouse age/gender?
  - Not really
- how many HC screens? several. 2 dozen largest datasets (conservative)
- DAE no longer stores all the state (and re-create it after crash),
  - Instead Jobs create new jobs, then what dependencies are release (WAITs)
  - checkpoints!
  - less OME overhead
- Ability to remove a feature due to too little signal
  - Performance test on omitting features: no change
  - One of two parameters: how many features do you keep (10-90%) performance falls off on the ends
  - to increase execution, you'd have to remove the good features
  - but once you have the features computed, they're enormously valuable
  - Spend most time building classifiers and jiggering
- Donald: What's the classifier? Weighted-nearest neighbor (Euclidean distance)
  - Two kinds of thresholding
    - hard: lowest 1/3
    - soft:
  - What's the classification like when you don't throw away features? No change.
  - Q: Did preprocessing increase the classification?
    - Didn't help (segmenetation). Josh: but you may be already overly specialized on non-preprocessing
    - True, and may help performance.
  - Like Nearest neighbor for other qualities: easy to ask where images are in marginal probabilty space
  - Almost moved beyond classification, and now to image similarity (error rate isn't always the most informative)
  - Did you try to decrease the tile window?
    - Trying right now a computing window.
    - Do register knees via checking for a peak (a bit of preprocessing)
    - pharynx aligned on the scope
    - missing centromere screen: looking for dot(-absence) in goo, ...
  - Painters
    - You can vary training/testing (round-robin, 50 cycles of random splitting, cross validated)
    - No subsampling

jvmlink discussion
- Curtis will look into what needs to go into hashmap implementations
- The java.lang.Object return values will be ignored
- 4 interfaces to implement: IFormat{Reader/Writer}, Metadata{Store,Retrieve}
- Client will probably want to start server via exec('...') call.

Discussions at Jason's
- Google feedback
- @Removing different sizes of registry icons
- Storing protocol changes as diff
- Hierarchical searching in protocols
- Caching via BioFormats
- Macnification ...
- displaying metadata (table?)

Cat/Towards Beta4 : User Perspective
- Q: There's data about the improvement re: tagging. A: Yes. See "Focus Group '08".
- Without prompting: "Like the way that goes across the screen"
- usability data — possible directions
  - automated analysis (movies, tracking, ...)
  - @Need to improve messaging/alerts/conversations about data
  - improved workflow (also route to publication)
- big challenges
  - trust with data
    - How to we make the OMERO brand as/more trustable as/than Photoshop
  - confidence over what image "is" (change tech or change user)
    - Manage the "RTFM" level
- Q: What do you mean "Manage growth"?
  - A: Curtis was saying, building for ourselves is a good strategy
  - However, we're past that stage.
  - Josh: Shrinked-wrap software
  - Ilya: Most shrinked-wrap software is a hodge-podge
  - Catriona: true of word, but not Final Cut
  - Carlos: Adobe Premiere was small, but now is big (lame users <--> trained professionals)
  - Cat: hodge-podge is often because the product was started 20 years ago
    - FinalCut is new. Was used to produce movies before released.
  - Jason: No one's mentioned that we're trying to dvelop for scientific discovery
    - New methods are coming (HCS, ...) that we don't support
    - They're minorities, but it's a new start
  - Cat: No matter what we do, we're constricted by how we can get funding
  - Jason: Need to decide the strategic decision
    - Next step? Some of the functionality are things that no one has ever done before
    - Jean-Marie: train the developer to view everything as an entire pipeline, not "this is cool"
    - "I've changed my way of coding,...you can solve any problem...but it's how you create the energy from a user standpoint"
  - Ilya: It's important to realize that this group has more expertise than a user has foresight to drive further use
    - "Failures because they were not controlled, but allowed to be driven by the users"
  - Is the decision we trust overselves? Or do we target particular groups for wide-spread use (publications)?
  - Jean-Marie: problem appear because we are more confident of the tool we provide
    - Click on the link and put that on your machine
    - Previously we were only targeting a safety zone
    - Cat: Is Pasteur the testbed for this? Cautious, critical, ...
  - Chris: 1) all shrinked-wrapped software falls into trap; need cash to do this (driven by bottom line)
    - 2) getting into whole institutional level can be difficult/slow
      - Jason: Very reactionary approach to users. Barrier to adoption ("wait on feature 'x'")
      - How do we strategically (not tactiley) guage the response of take-up?
      - Cat: Need a process for taking a moment to look at relationships, directions, ...
      - Jason: Imperial is saying "we want to drive lifetime..."
      - Ilya: Should organize as a set of use cases (not people)
      - Cat: Practical — "They're close to a paper..."
      - Andrew: Need places that will do the best marketing. E.g. Mario
      - Cat: But there's no way to make those decisions, and fit into dev cycle
      - Jean-Marie: The reactive mode has been important to get us where we are
        Getting into a new domain is a slower reactive mode.
        Still hard for devs to extract yourself. Make extensible for later on.
        re: Washington - some things missing (not much) for whole-pipeline to go back to Photoshop
    - Chris: we have a very engineering (top-heavy) team. No sales, support, QA, ...
      - Jason: 2 anecdotes
        "You guys scare the pants off me. We have 4 developers"
        Company growing 40%. Marketing and applications.
        Chris: Like institutions they have no infrastructure to get things installed
        Cat: In the next grant, also staff for end-user websites, smoothing release process, etc.
- From feature design to concept design
  - "unknown known" --> "unknown unknowns"
  - getting the right info at the right time/place
  - how to work with the dev team. probably everyone needs to adapt.
  - handling suite at different level of maturity
  - Jean-Marie: decision on how to respond. Development might take 6 months, and so there are no black holes.
    - How do we communicate? When? Coming back in 2, 3 months.
    - Burnt out some of the users, and if they don't see something coming, no feedback where their information goes, we get no answers
    - Better job of advertising to user, not just on trac
      - Cat: presumes that we can say "here's the strategic decision"
      - Lot of things that are put on the table, but since there is no strategic view, hard to say "X"
      - Need something visible/clear window X comes before Y.
      - Jason: In the history of ethnography, is there anything about the users being spoiled?
        Fact is: we appreciate your feedback, but look at what you have.
        Chris: Not done anywhere.
        Cat: But it is, Nokia. Not user-centered design, but user-engaged design. "We won't do everything yo uwant"
        Chris: I don't have one application (open-source) that I can interact with. We can't achieve it.
        Cat: It's not that the users are spoiled. It's how we manage it. They don't need us; we need them.
        What we can control, is how we manage/control information. The more we feedback, and build into system
        Example: we learned that you have to tell the user ALL the things you have to upgrade at once
        Have to apply to other things. We are completely over-relying on certain relationships
        Choice: rely on ourselves; or keep using user/find way to work with them. But we need way to make that choice.
        Ilya: Project needs to become more strategic, and that needs to become visible
        There is a lot of information about the project, but it's all technical/tactical.
        Brian: It's because we all are developers.
        Ilya: But it's critical.
        Jason: Vision exists, but loose and not articulated in a well-defined way. Ilya: Mission statement?
        Chris Mission statement shows you what direction, but not what stops along the way.
        Cat: It's about managing the process. Then we can do it.
        Jean-Marie: Everything in two steps
        feedback.o.o.uk send something back to users when they report a problem.
        Will: A user has created a ticket. Because of anonymity, can I put their email address?
        Curtis: We do that. But the newest version is much better about hiding CCs
- Moving our relationship forward
  - Address key issues at the right level/time.
  - Jason: we'll find some things take one week
    - Where/who in this process is the responsibility to get the info back to the user? ("We'll get back in 2 years")
    - Cat: Need to make a decision about that. Ultimately the UI team has to take on more of the feedback loop.
    - What did Nokia do?
      - They have expert user groups who they deal with online.
      - Panel of ~5000 people in Scandinavia, who get software, then they go to online software/
      - Commitment: for every developer, they have 500 end users attached. Building right user base.
      - "Open Innovation Movement" (started at IBM). They know that they've exhausted users. Have to pay people.
      - Nokia handing out free phones in India (ethnographers). Chris: will they keep losing market share?
      - cF. Fujitsu teaching engineers how to talk to people
    - Ilya: In your domain? Instead of being a bit passive, but to _push_ the new things out to the users.
      - Cat: We can make a decision "We want to do X" then we tell users "No, you really want that"
      - Ilya: with WND-CHARM, over the past 2 years "I'm interested in analyzing your data"
        Classic response: if you can count nuclei better than X, "then what good are you?"
        How do you explain that to biologist? Can't give technical answer.
        Josh: Exactly like "here's a server architecture"
        Ilya: you have to take the reigns, and demonstrate that you've made something better
        Cat: strategic about taster sessions, to get one lab to realize everything they need to get to a single publication
        Will: One idea to help adoption, is to let people try out insight, before they install the server. Web account?
        Chris: Biggest problem is that it's not representative of how you can use it.
        Josh: Have to weigh what expectations they will recieve from the demo software that we give them? (Performance, bad demo data, ...)
        Ilya: Have to go in and solve someone's problems better than they can.
        Jason: 2 anecdotes
        Dumping images into powerpoint and can't do anything with the images. "Why not OMERO?" "What's an OMERO?"
- Josh: encoding process/learning effect
  - Chris: Don't make everything public
  - Ilya: But we aren't a company, ...

Josh's Presentation
- What are the advantages of Jboss over Ice (vise versa)
  - None, needs a good admin to keep running and is fundamentally focused on web apps
  - Its good if you have an application server and you just want to plop something in
  - We are getting away from Jboss (might keep it if they already have an application server). We haven't had anyone come to use saying they want to do that
  - We are getting away from JBoss (By Beta4 maybe? - depends on client perspective) In the long term, yes. Migration of clients will be a pain.
- Our server design allows data to be saved to the database, and then accessed by the process script engine
  - (so as a result, a script developer) - it is their responsibility to extract the data results from their analysis and make sense of it's presentation
- Needs to be a survey and feedback mechanism before we get into clustering - what culture are we supporting? Blast?
- Josh's cloud diagram doesn't articulate the interconnection between these different cloud regions
  - Jason: if we took what this cloud represents to the client today, the development of the solution would evolve
    - JM: Yes but we do need to narrow this cloud down and concentrate on some specific sub-areas of this diagram
    - Josh: Yes, we need to decide what resource constraints exists against this diagram

Chris&Colin / FS within our existing technology decisions
- setting up monitor on colleague's directory, no change in 2 weeks, ==> ... (trust/privacy issues)
- @Carlos sends Colin a link to Windows/Python FS-Events info
- Jean-Marie: looking at the whole pipeline, what happens when the file is archived?
  - Chris: Adding hooks/methods to the interface isArchived(), isCompressed(), ...
  - Extensibility/pluggability: add a python script which implements the isArchived() interface
  - FileSystem browsing/stat bits
  - Want a java.io.File implementation, to allow manual importing, ...
  - like copying to a USB
    - Have to integrated as well, or we'll do ourselves a disservice
  - Ilya: files in userspace. how are you going to deal with that
    - Pixels have been extended to have a URI
    - Qualify where the pixels is.
    - Server updates the URI when users move files.
    - Glyphs of "this file has changed"
  - Ilya: Who's doing the driving?
    - Who deletes. Moved an Image to another directory. Why.
  - Josh's wish list:
    - Immutable files (and how do we handle users' control over the files. a copy?)
    - Control structures for files: /josh/docs/foo.doc so that I can overwrite an existing file
  - Not replacing import model. To have your data secure, then use the import.
    - You give the user options
  - Relationship to caching
  - Possibly create an OMERO.ftp process where we control (clippy-style) what the user can do
  - Mapping FS file structure to PDI
    - Naive answer: whatever data management decisions in OMERO are only in OMERO, don't reflect on the FS
    - What do the user meanS?
    - Do I want all the annotation/tagging informed from what I've done
- Technical: Are you moving any data?
  - A: Will attempt to use bioformats directly to read the data
  - Ilya: User has you over a barrel when it comes to removing the file.
  - Brian: OMERO.fs to build an imaging cache. Our own archiving system.
  - Tech to store references to all files on various OMERO.fs servers (by hash sum)
  - Federation, finding the nearest copy (GEOME)
  - Ilya: User wants something out of the drop,
  - Jason: Maybe we have to control read/write
  - Playing hangman. Each click ...

Donald / Scripting
- Ilya: I don't see the tie between the a/b/inputs
- Ilya: Standard workflow - algoritm on a collection of images (or of regions)
  - Need the bit of code to handling the mapping of Image/ImageCollection
  - It's a key bit of the puzzle to not burden the algorithm developer with collections
  - Need iteratorOverCollection.py or convertDatasetToImage.py or roisFromScreen.py
  - Ilya: If you are writing bits in OMERO to let users collect images, ... it's death
- Ilya: When does matlab startup/shutdown? Can control it.
- Ilya: Maintenance of Matlab-Java bindings
  - A: scipy with mlabwrap seems to be on top of it
- Does it work in Octave? Should, but hasn't been tested
- The CellProfiler is not very objecty. A: Can convert it into an ROI
- Josh: IS THERE GOODEVAL?? A: No.
- ImageJ integration
  - What's it look like?
  - Possibly a MacroTemplate which python populates and writes to disk then "java -jar imagej.jar MacroTemplate"
  - >>OR<< Just have ImageJ as a client for full interaction.
  - Jason: What kinds of analysis tools do we want to develop? ImageJ plugins are "standard" in processing/image analysis
  - Are we doing next generation work.

Oliver / Messaging Server
- shmem, socket, pipe communication
- What are some examples of what you want to do? (what does it deliver)
  - HCS -> record images to OMERO
  - Post-screen analysis needs images from OMERO
  - Q: Further abstraction? A: Yes.
  - Concern: only 2 pieces under control — messaging server & omero
    - rest is closed-source. But, Arivis is signed on
    - doesn't scale with N collaborators
    - what really matters are the messages?
    - the message server has to define the messages/protocols
      - you have to buy in (cF. EtherCat)
  - Donald: would Ice not be a nice way to do it?
    - just an idea. This is exceptionally nice.
    - Brilliant to just have biologists control hardward from insight
  - LIM systems integration
    [+]
    - do everything from library-from-freezer to transfer-to-plate to put-on-scope to lost-tips
    - - all custom made things
    - - is there a standardization there?
  - big companies say: here's all everything you need, if you buy it all from us
    - certainly true in HCS
    - starting to break...
  - has to be so easy and complete
    - e.g. audio.
  - we're not the problem
    - some partners don't have time
    - some are afraid of changes, must be convinced.
  - need to find 3 components which can be used as an example
    - should be no problem.
    - is it possible to control their GUI? (VNC, scripts, ...)
  - a cool example, "Stage in wrong place"
    - unclear where experimenter protocol is started
    - browser is perhaps a pretty good starting point
    - "start pulsing device 12"
    - notices that table is wrong spot
    - "move table to"
    - "table is in position"
    - then starts pulsing
    - "pulsing"
  - analysis on several images
    - browser asks for analysis
    - analyzer asks for data
    - analyzed
    - return data
  - is it worth setting up dummy system?
    - can you run a 100,000 images?
    - like the demo from Chris and Colin of dropping an image in a directory?

insight / jean-marie
- interdepencey (cyclic): the tough stuff poses questions that we can't answer except holistically.
- Prototyping!
  - how to have multiple version which users can see and choose between
  - would preferences help?
  - Cat: difference from other prototyping, it's so personal that the user can't evaluate without their own data
    - have to build working prototype, no lo-fi or hi-fi
    - Jason: they want to work with their prototype
    - something that can't be handled with our old way of working
    - Eclipse as a model, with versioned plugins so users can compare
- HCS provides the deepest hierarchy (more than tagging) - Harry & multi-hierarchies
- finding new insights in data view traversing across hierarchies
- smart folders. Perhaps folders per protocol
- Ilya: Can you switch from one hierarchy to another keeping selection?
- Can always come back to the previous view. "Capture-Concept"
- Google-map/Screen-map/Large-image-map : not hard but a new concept

Ola / web
- "You have no tasks. Go add 10!"
- Ilya: Can you share with anyone? (re: botnet)
- sharing v. publication: one during your work one is after.
  - re: legal aspects with journal
  - part of sharing/publishing workflow is creating a new dataset, specifying the visualization
  - email contains a url? or url + session key
  - publication is wide-open
  - Josh:or as screen, so you can do formatting/positioning
    - Carlos: having positioning information would be great in clients
  - Jean-Marie: is the text wiki? how do other clients digest it?
    - A: HTML. But the share is on the application-side (during development time)
    - Storing as XHTML. Easiest.
- Ties into OmeroSecurity?
  - Yes. "Mounting a share" is the metaphor. Slower. And doesn't show up in my regular view.
- What kind of object in OMERO can be a "Task"?
  - Ilya: When did I take an Image?
  - When did I process it again? 3rd time, 4th time and what did I do those times?
  - Will: I want to pick up the tagging I was doing yesterday
  - Jean-Marie: definition of groups of actions
    - acquisition, tagging/manual data, analysis
  - Date in dataset or project name (experiment is annotated though somewhere)
  - Use TimestampAnnotation either globally or attached to the user
    - Needs a start/end, repeat, etc.
- Viewer
  - Hide-all? For LOTS of images.
  - Pager infrastructure? Not yet.
  - We need a dataset with 5000 images would help.
- Andrew: sunset clause? from when?
- Ilya: the real power is public access

HDF
- Position written into next grant
- Talk to M. Dougherty
- Definitely going to move forward

ImageJ
- Client. Yes
- Server-side? Perhaps discussion at the ImageJ meeting later this year.
- Email list after conversation with Johannes/Wayne
- first integration: use the data that's in the server for display
- second: then can store their ROIs as an attachment
- third: with strong need, we can look at converting to overlay
- fourth: with strong need, we can look at running in the background
- ideas:
  - storing macro as an attachment which produced the results
- 2-4 weeks of just getting plugins done.
  - fixing beta3
  - use OmeroJava? For now, use RMI

Search
- changing index schedule so that it doesn't lock
- "can you upload a website and then google right away?

Delete
- once you try to sole something simple like "Image" then
  - then what happens in SPW
  - which links do you break?
  - Ilya: deleting image from a well should be empty
  - propagate to the tag or not
  - Ola: Cleanup Wizard? "You have some unused items on your Desktop..."
  - Curtis: PDF-Dummy to link things (manually updating the ref count)
  - Brian: status deleted/failed/
    - Jean-Marie: Performance problem checking the flag on all the things
    - Chris: still slows down the overall transfer
    - Ilya: Just pushing the problem somewhere
  - DeleteService
    - Ilya: Based on deleting a MEX. But have to decide delete or unlink
    - Need a wizard which shows everything that you are possibly deleting
    - Curtis: Ways to intelligently flag stuff
    - Jean-Marie: Need the cleanup phase
    - Chris: Querying for orphans!
      - Problem: 100,000 orphans? Now What?
      - No structure to say what the PDFs mean, are/were linked to
  - "New Delete: here are the consequences."
  - "Old Delete: here are your orphans"
  - Cat: An image belongs to another user. Having an annotation, makes two owners, effectively.
    - Two owners want to delete: gone.
    - One owner: gets hidden/trashbin
    - Chris: I don't think it's practical because ownership is as simple as viewing an image.
    - Chris: Different priorities of annotations? Cat: All are making sense. Will: It's finding something.
    - Cat: Copying images when you attach.
    - Josh: Service needs to have the semantics "What will this delete?" "What will be orphaned?"
    - Jason: Don't want to force a value propsition
    - Ilya: Have to assert who the dominant owner is.
    - Ilya: Can publish a retraction, but can't get rid of the original
    - Will: If I attach something to a dataset, then I trust it won't change
    - Jason: conversation idea ... most people would ignore it because of the competitiveness
      - Marking something as a "shared" image to put it into a different state (community model)
      - Josh: Disagree
    - Have to support both models: protecting things and allowing delete
      - Accounting systems: audit and nothing gets deleted (Brian)
      - Chris: Naive approach because they have heavy pre-filtering
      - Ilya: Collaborative work is the decision to delete
      - Josh: Way for the event system/messaging to help primary owners to delete.
      - Chris: Need to help system to decide which links are overridable
        Josh: then then client can find out what's overridable
        Josh: But will still need the dialog box: "Primary wants to delete, do you agree?"
        Ilya: is there a permission trap, where you can't see something attached?
        Ilya: color code graph view of the things: Deleted/Could be Deleted/Things owned by others/Things blocking your delete !
        Need to know what they are getting into
        May be surprised when they go to another site
        Habituation!
- Todo
  - Start implementing static version
  - Use rules for possible GUI later
  - Need delete/unlink differentiation
  - Eventually message for confirmation
  - Use Sessions/Events to offer additional deletes
  - Permissions.trashContainer
  - Implemented Client size

Lunch discussion
- future: protocol, presentations,

Josh multiple pixels per image
- Ilya: Die!
- No one gets it, who cares?
- example: deconvolve. Dataset pointing to the deconvolved pixels, no way to see other pixels on the same image
- deconvolve should be another image? That's what we do now.
- Default pixels, what is used by default. That's what users think is *the* image
- Image link should have a link to the default pixel.
- Donald: I don't like multiple pixels, it's evil.
- Jason: in presentations, when mentioning multiple pixels, people look as if you said something stupid
- Event log is complicated to use too
- Image concept has 2 responsabilities
- images are just a view on the data
- You have something that's yellow... provenance model, call it what you will
- What are you tagging? Looks the same to users.
- Josh: information should be normalized, in the right place.
- Chris: one image is raw, one is projected, how to make statements about each image?
- how to say: "these 2 pixels are more alike than other 2" - is possible
- tool don't have the functionality to handle multiple related pixels-
- Donald: doesn't satisfy the way we work. We want to say how pixels are derived.
- Brian: It's a workflow problem, Image containers doesn't do a good job for that.
- Users do that anyway, they can add a new set of pixels. Now what?
- Josh: History is the new container
- Jason: must also support the algorithm case, not just UI
- Ilya: the yellow brick road: It changes a lot of fundamentals.
- Chris: Images have only name and description, nothing else.
- Andrew: what happens to info attached to image? ROIs?
- Chris: they should be in the pixels, not image.
- Josh: we lack linkage between dataset and pixels.
- Chris & Donald: Collapse Image and Pixels, removing multiple pixels completely.
- Next Major XML Release? (July) with light sources and xrays
  - No. Implement and see what it does what we want.
  - Write it up on the ome-xml trac, get suggestions
- From Ilya 3 requirements: (and all tools don't have this functionality)
  - 2 Pixels are more related than others (struc)
  - Container speaks about all the pixls (struc)
  - "image" / "pix" are convoluted over 400 years of microscopy

blitz libraries
- naming classes/libraries
- what methods are implemented
- coding style
- where's the committe?
  - similar to in server
  - no underscores, etc.
  - come to a conclusion
  - convention: load is top to bottom, find is bottom to top
  - store it as Client.ice under blitz/resources/omero/Something with doxygen in Javadoc

OME model adherence / properties
- Possible to come up with groups?
  - SPW, for example
- Do we discuss it after the fact?
- handling creatiaon defate as default

Big Images
- use cases: large plane images 22000x40000
- things are getting slow
- Andrew: a "big" image — does the server treat it differently
  - both sides need to adjust
  - provide an exception which says "BigImage"
  - new programming interface with sliding window
  - image pyramids
    - calculating multiple resolutions
    - pre-rendering scaling
    - subsampling. visualization impression is unsatisfactory (algorithm based)
- reasonable limit: .1 microns with slide 3 inches, 750,000 pixels a side
- future-proofing requires 1 Tb / plane
- compression
- does the metadata describe a tiling or a stitching?
  - use case: hospital basement full of slides that should all be digitized (surgical biopsy) Aperio.
  - one implementation are like geo tiffs. algorithm to do registration
  - optics don't let you perfectly align them
  - then they are split back apart
  - in that case there's less need for metadata to get the overlaps settled.
  - retiling is a common implementation
  - becomes an issue for bioformats. already partially have a tile api.
  - interface.whatAreTheTiles()
  - answer: we don't really need metadata changes
- there are both an optimal client tiling and an optional server tiling
- interface.setScaling(double) anything at 1 or above returns the whole image
- handle all of this on the server with exceptions
- Consider: 3D or non-XY tiles/blockings
- aperio
  - http://images2.aperio.com/view.apml?listview=0&returnurl=http://www.aperio.com/&returnest=0
  - this is life-and-death

Breakout discusions ====================================================================

Breakout list
- List of needed EventLogs - SESSION, ...
- Search/ontological
- Benchmark OMERO with WND-CHARM
  - Mime-type
  - parllel IO
  - tagging tags or using events
  - typing: patching v. fork
- HDF
- Image/Pixels
- OME-Profiles/Adherence/Nullability
- ImageJ/other integration/GLUE
- FullStack for next month
- Demo possibilities: vmware, live-dvd, ...
- production server for weblitz (nginx?)
- naming weblitz
- blitzconnector, gateway
- hierarchy/trash/delete/imported flag/...

Requirements discussions
- Image/Pixels discussion
- OME-HDF
- ROIs: supporting ImageJ rois, for example
- Relaxing the model
- converting csv/xls (re: Jun20/Will & printing)
- Server scripting (Java)
- standalone viewer
- ImageJ repository
- cluster support
- ImageJ client
- Discussion on documenting what properties are supported (Jun20/Melissa)
- Defining the weighting of all properties (Jun20/Andrew)
- Which scripting interface would users like to use?
- Stemming and Tokenization and Thesauri
- Jason's presentations stored in OMERO (presentation as publishing) ... audio? ... "AskJason.com".
- Look at Josh's cloud diagram and decide on some priorities?
- what's a task/event in the calendar? And how is that modelled? ICalAnnotation? (repalces Timestamp?)

Monday "Coding Day"
- JvmLink

Concrete todos
- Document or implement goodeval()
- encrypting password
- examine one story/feature from top to bottom. include as many people as possible to determine process.
  - simple workflow things: projections, AVI,
  - screen
  - sharing/import/export
  - search
- Rename OmeroJ->OmeroJava

Document Actions

Print this

Sections

Personal tools