Personal tools
  • We're Hiring!

You are here: Home Community Minutes Meetings June 2008 Developers Meeting

June 2008 Developers Meeting

Developers meeting held in Dundee

The goal of this meeting is to define goals and milestone for the Beta4 releases of [old-link http://trac.openmicroscopy.org.uk/omero/milestone/3.0-Beta4] OMERO.server and its [old-link http://trac.openmicroscopy.org.uk/shoola/milestone/3.0-Beta4] clients.

At this meeting, we will:
  • define Beta4 as well as any next betas and the RC series
  • ensure all tasks are ticketed and assigned

Suggested other topics (to be scheduled include):

  • OME-HDF
  • The Message Server proposed by Lars, Oliver and Aline
  • Extensions in OME-XML
    • Modelling and meta-modelling of OME-XML in our software (xsd-fu, Model Driven Architecture, etc.)
    • LED, X-Ray, ROI model, Metadata only files, OME-HDF, Relaxing the model, 3rd party data
  • Image/Pixels Data Model
  • WebAdmin priorities list
  • Used licenses (Mozilla, Apache, etc.)
  • Discussing naming and other models (e.g. xray) - ConferenceCall 2008-05-22
  • Are we missing anything in 'developer support'?
  • Are we missing anything in 'user suport'?
  • The Omero clients 'style bible' - how to create, what it should cover, etc.
  • Beta 3 - outstanding issues from user perspective
  • Notes from the field: opportunities and problems identified during the ethnographic investigations.
  • A summary of key findings from the Dundee and Pasteur scientist surveys.


Draft Programme (please make comments etc as necessary)

Day 1 (June 20th)

TimeTopicLead(s)Comments
09:00Welcome Intro, Summary
09:30Bio-Formats OMERO.importerMelissa Brian Status and evolution
10:30File Formats Curtis? OME-HDF
11:30Data Model Andrew Evolution, Image/Pixels
12:30Lunch
13:30OMERO.editor Will presentation+ Data Model from Users point of view
14:30OME server Ilya status
15:30LOCI Apps Curtis status
16:30Discussion ?? extra time



Day 2 (June 21st)

TimeTopicLead(s)Comments
09:00Coffee, setup, gossip
09:00Usable ImageCatriona Beta3 status, evolution, methodology
10:00OMERO.server Chris Josh status
11:00OMERO.fs Colin Chris? demo
12:00OMERO.scriptsDonald status and evolution
13:00Lunch
14:00Messaging server Lars proposal
15:00OMERO.insight Jean-Marie status evolution, integration
16:00OMERO.web Ola status evolution



Day 3 (June 22nd)

TimeTopicLead(s)Comments
10:00Discussion Jason
12:30Lunch
14:30Plans Jason tickets, website etc.




Notes



  • History/Intro
    • Jason & Ilya at all
    • First Dundee: 2004...
    • Goal: get what we want to do for the next year or so
    • catering: lunch, etc.
    • have to talk fast
    • looking at downloads, registry
      • New Chinese dot on registry
    • Grants pending: 7 positions from Welcome Trust, Usablity for 6 positions and a PhD
      • Funding for Baltimore 2 FTEs until review next year
    • Everyone healthy/happy?
      • Getting to Jason's tonight: taxi, etc.

  • Formats
    • Brian
      • Q: Is an import a copy? A: Yes, but usually the history feature is used to come back (after bug report) and retry fails
      • Curtis: we're going to be using the ImportLibrary directly, too.
        • Have been some build issues.
        • Would like to see the newest version always compatible
        • Curtis: Try to not just change things every: "I'd like to change filters now"
        • Brian: Would be good to look at how we communicate
        • Curtis: Fine with changing OMEROMetadataStore so that it compiles with stubs
        • Chris: In the past, changing of methods is trickier than adding new methods
          • Legacy code which mismatches with the code-generated bits.
        • Need to setup automated testing of Bioformats jar in importer
      • Q: Advantage of server-side import? A: speed, just uploading file, getting files off the scopes
      • Importing at Pasteur: caching server (1TB), processed daily to long-term storage
        • dummy client on cache server to import
        • Q: option on OMERO.fs? A: Handle the upload
        • Jason: Hard-core reality. Got to get off the scope, initiate import and run in background
        • Brian: Would like to move import to service which starts a daemon
        • Has to be light-weight process: ftp file over, then control another service on the server to deal with OriginalFile
          • ImportDaemon (registers with OmeroGrid) and ImportService
        • Can't install anything on the scopes.
          • Single user environment
          • Might need OMERO.fs/SMB mount using no install (PUSH)
      • Curtis: new formats is not necessarily importer-dependent
        • No reason we can't expand everything to support new formats
        • Ilya: Have to be a bit careful here
          • Real lack of information management in science
          • Don't want to become that. Will unfocus everything.
          • "You guys deal with images. Why don't you handle ....?"
        • Jason: FCS - time-series data merged with image. But it's microscopy, so...
        • Ilya: If you are the only ones doing it well, ...
        • Curtis: Step-by-Step. Start with what we want.
      • Cross-language support
        • Curtis: We can extend JVMLink
        • Josh: What about Ice?
        • Curtis: Sure, what about deployment?
          • Now we don't need our scripting language: use beanshell
      • Chris: We never imagined people downloading and haxxoring importer from svn
        • Need to update for and support that
        • More flexible, support interfaces
    • Melissa
      • Q: Performance not infrastructure based, right? A: No only formats (ND2)
      • Specification: will always want to be adding more (electrophys: AUDIO). There's a tension there.
        • Need audio attached to a video talking about the image
        • "soundtrack"
      • learning project uploading to OMERO
        • Nothing needed at the moment
      • OME-XML
        • Currently OMEROMetadataStore is write only. Need to implement the interface
      • Curtis has been thinking about GUI for converting between formats
        • Maybe not anymore. ImageJ has something now: VirtualStacks (check option in importer)
        • Had wanted to save RAM.
        • This may be just improving ImageJWriter
      • Dev documentation
        • Brian: same problem with the importer
        • Ilya: "Keeps the second-rate developers out".
      • Loci update
        • Jason: previous site there was a list "You can use this software with ..."
        • Curtis: Now under "Applications"
        • The visual — big, blue — was great
        • A big money line (screenshot)
        • Jean-Marie: More explicit under the metadata section in the format list
          • Good for building a client which supports the metadata
          • Chris/Donald: was using OMERO from matlab to dump, work, push
            • Need a way to know what is supported
            • A trac page where each format has its metadata-support listed
            • Could also link the commit which supports new metadata ("r4096")
          • Curtis: a lot of work, perhaps new interface (mirror of MetadataStore)
            • isThisSupported() - one per property
            • default is false
            • the code documents everything that is supported
            • write a script which would generate HTML of what is also supported.
            • Chris: generate Trac code.
            • Chris: Could it be n a single data file. Code is a bit of pain. Good for understanding.
            • Brian: people ask often "do you support field X on Y"
            • Josh: possible to have coarser interfaces which short-cut checking all the properties
              • Curtis: Can enumerate the properties supported, or list Image to say all of Image
            • Donald: Would really like to see support: "Image", "Instrumentation", ...
            • Andrew: Brian had asked for matrix for what populates which tables
      • Report from the field
        • lifetime imaging, high-content
        • "What we really need is support for Becker-Hickel (sp?) format?"
        • He knew: STD in bioformats (even though not dev), AND that support in bioformats had been turned off
        • "Why?" --> "Test-data..."
    • Curtis
      • OME-XML
        • Writer
        • Expanding metadata store
        • Staying on same page. What omero supports and if we're converting?
        • MetadataStore as the glue workhorse
        • OME-XML which doesn't validate because of incomplete data _NOT_ in the proprietary file
          • What are we supposed to do?
          • Want to be able to fully populate everything that I have!
          • Our software is producing non-validating code
          • Not ok.
        • See [href="http://cvs.openmicroscopy.org.uk/tiki/tiki-index.php?page=ConferenceCall+2007-12-27&highlight=objective] http://cvs.openmicroscopy.org.uk/tiki/tiki-index.php?page=ConferenceCall+2007-12-27&highlight=objective
        • We need a plan, document, and implement
        • Brian: prioritize this, please.
        • Josh: read the thread for the propsed suggestions
        • Jean-Marie: We can compare an example from Will later to make this complete
      • OME-TIFF
        • Reworked the reader
        • Multi-file datasets: validator doesn't validate across files
        • Ilya: suggestion from NIST for some kind of naming convention of microscopy images
          • Especially in multi-file formats
          • Curtis: Can only ever be a recommendation
          • Chris: Do the microscopy producers care? Not in that realm.
          • Not in that realm because there are no guidelines
          • All instruments in some domains come with a certification for a guideline
        • Write up recommendation based on LOCI
        • But the current solution about embedding the file names is pretty nice.
        • Josh: Putting a flag in the file? Not really.
        • Add to golden start system
        • Josh: Possibility of someone accidentally/purposefully breaking the regex
      • OME-HDF
      • Other
        • JVMLink -> Ice
        • Viewers/converters in ImageJ
        • Documenting what metadata is supported
        • Jason: something on the grant for someone working with formats
        • Curtis: several students wiscan, jvmlink, flow-cytometry, visbio would profit from a formats guy from grant
        • Jason: What does the plan need to look like coming out of the meeting?
          • Curtis: Who, what, when?
          • Monday as post-conf coding day!

  • conversation about communication
    • possibility of using Adobe for whiteboarding, etc.
    • unconference-style to get through all the points we want to discuss (hdf,...)

  • Data Model/Andrew
    • ROI
      • Curtis: Have you referred to the notes on ROIs
      • Curtis: Would like input to compare imagej & visbio
      • Ilya: Is there no requirement for the object to be closed?
        • A: Not currently. There are closed shapes.
        • Ilya: There must be a way to determine if a fill is possible.
          • Draw a rectangle with 4 <lines/>
          • Donald: Advantages if you define it with <rect/>. Do queries on the types.
          • Andrew: The intent was to use the primitives provided by OpenGIS (http://www.opengeospatial.org/standards)
          • Ilya: Good to use some form of standard and tools that people understand
          • Donald: surfaces, ...
          • Ilya: are there anything not translatable?
          • Not a Z toolbar in google earth? But there is a time concept
          • time-series data? Yeah, but our binary data is a different order.
        • ROI structure here is completely unnormalized
          • Curtis: Would like to have some "share" concept.
          • Andrew: null z or t means over whole dimension
          • Curtis: fine.
          • Josh: could use an annotation if you really wanted to.
        • Ilya: Have to calculate the continuity
          • <inverse/>
          • SVG does allow for defining the orientation ("fill left-hand side", vectors)
        • Doesn't allow for two lines in z
          • Jean-Marie: use the derivation to say that they are an angle
          • Curtis: Just attach a 3d mesh
          • Donald: use "connector" to connect two ROIs
    • Relaxing the model
      • Omitting Pixel Data
        • "open metadata environment"?
        • <metadatafile filename="..." hash="..."/>
        • Ilya: PixelsRef with LSID
        • stylistic issue
        • Josh: Let's use pass XML over an API, dump whole DB to a single file as backup
        • Brian: Different representation of pixels, compressed
        • Ilya: you need the pref
        • Curtis: Have the <MetadataFile file=""/> be anything that bioformats can read?
        • Chris: We originally didn't want to do the metadata companion files
        • Jason: we ate that poison pill when we did OME-TIFF
        • Josh: Gets worse when we do HDF
        • Companion file is not a starred profile
        • Capturing extra metadata on the scope machine
        • Single companion file for multi-binary files
        • Putting the carrots on the main page
        • doing the link at the provenance level (OriginalFile: STs)
          • Not in model, nor is provenance (only in XSD)
          • Consider it without server concept, pure exchange format
          • non-image extra files (audio, etc.) is fine
          • Curtis: Want <plane/> in <pixels/> so it can't be the bit that disappears
          • Just make <bindata/> optional.
          • DICOM flag: GE doesn't read SIEMENS doesn't read ...
          • temporary LSIDs pointing to /dev/null

      • Relaxing
        • enumerations
          • Josh: like omero, using a reference
          • Chris: how to handle the resolution?
          • Ilya: modelling discussion. Enumeration or not.
          • Just define the process for defining a new enumeration.
        • Curtis: I understand what you are saying that people will abuse "Other"
          • "Validation does not equal quality"
          • Multiple Quality Levels: validates but did you do a "No-No"
          • Will solve LOCI's problem.
          • Jean-Marie: Interesting to see what Will is doing, need to look at whole pipe-line
            • Need a formal way of describing, visualizing, ... Otherwise develop own way
            • cF. Compare XRay tomography's own OME schema
          • Chris: 30+ formats and none of them can fill out the objective
          • Ilya: Objective wasn't done arbitrarily
            • Was defined to show future manufacturers how to define objective
            • But we still need to deal with it
            • "7 of 20" but we need the definition. Once all 20 of 20, then you drop the optionals.
            • Sociology/social engineering
            • Ilya: A service we provide. Obvious that I need one number to go from provisional to full. Gold star!
        • Cases
          • Objective,
        • OME-Provisional, "Level -1"
        • Who's going to do the evaluation? Put up the table with examples of each ranking.
          • Curtis: for each format someinfo
          • for every property (entity shortcuts): N/A, NO, PARTIAL, YES
          • format quality ratio: is it supported? low == not rich
          • each property:
          • weighting the properties, importance of properties, level of OME
          • four stars : import level, publication time, ???
      • 3rd party
        • Andrew: want a people to look at the definitions that we have and pick one
        • Not just an <any/> yet
        • hook a block one level lower
        • pdf attached: what does the import/export workflow look like
        • Ilya: have to be careful where the original format is important (pdf/mp3/...)

  • will/editor
    • Ontology lookup: Curtis — going the right direction with that
    • Ilya: results of timed experiment show up with PI on date for review
    • future searching: Ilya — post-processing the returned protocols for hierarchical structure
    • Curtis: Is there a way when they create a new block "Filter" using the OME model?
    • Are allowed to link to another file but not to another element?
      • Some use cut-n-paste.
      • Josh: what happens if only one file gets copied? A: Only local link.
    • Ilya: You have the MAGE block. Have you tried...
      • They have a nice description of a biological experiment
      • Problem: terminates in a microarray assay
      • But Treatment, protocols, incubations, ... new reagent.
      • Recursive: treatment, ... on new reagent
      • Data collection, instrument, Merck index entries
      • Reason: if you know enough about how you did the experiment, you can draw inferences (computationally)
      • Jason: Defining small parts of structure, and use those
      • Start typing "LightSource" template is applied, then the list gets computed, ...
        • MAGE is only useful if there is a redbook on the server
      • Summary
        • Easy to fill out template of : instruments, ...
        • Search facility for : "light source",...
      • Making this easier for fluid-handling robots like Ilya
      • sites where you have to fill out a protocol before you can use the scope
        • have to institue draconian measures because digital notebooks are tougher than paper
      • . first step is having things to cut-n-paste
      • . then point to other templates (perhaps data model)
      • data model: providing (file type) a list of filter sets.
      • Ilya: Has to be a carrot. Will: The only rule has been to make it easier to use
      • Andrew: Is the piece of paper a critical step?
        • Print and be remembered that it was printed
        • "Do you need to update anything?"
        • "I see you haven't finished your protocol"
        • mini-mobile (iphone) is a possible idea
        • have editor generate html which interacts with the server

  • ilya
    • "solving an imaging problem one at a time is not a solution"
    • Q: Any other tissue based (as opposed to hormonal levels, running through mazes) to differentiate mouse age/gender?
      • Not really
    • how many HC screens? several. 2 dozen largest datasets (conservative)
    • DAE no longer stores all the state (and re-create it after crash),
      • Instead Jobs create new jobs, then what dependencies are release (WAITs)
      • checkpoints!
      • less OME overhead
    • Ability to remove a feature due to too little signal
      • Performance test on omitting features: no change
      • One of two parameters: how many features do you keep (10-90%) performance falls off on the ends
      • to increase execution, you'd have to remove the good features
      • but once you have the features computed, they're enormously valuable
      • Spend most time building classifiers and jiggering
    • Donald: What's the classifier? Weighted-nearest neighbor (Euclidean distance)
      • Two kinds of thresholding
        • hard: lowest 1/3
        • soft:
      • What's the classification like when you don't throw away features? No change.
      • Q: Did preprocessing increase the classification?
        • Didn't help (segmenetation). Josh: but you may be already overly specialized on non-preprocessing
        • True, and may help performance.
      • Like Nearest neighbor for other qualities: easy to ask where images are in marginal probabilty space
      • Almost moved beyond classification, and now to image similarity (error rate isn't always the most informative)
      • Did you try to decrease the tile window?
        • Trying right now a computing window.
        • Do register knees via checking for a peak (a bit of preprocessing)
        • pharynx aligned on the scope
        • missing centromere screen: looking for dot(-absence) in goo, ...
      • Painters
        • You can vary training/testing (round-robin, 50 cycles of random splitting, cross validated)
        • No subsampling

  • jvmlink discussion
    • Curtis will look into what needs to go into hashmap implementations
    • The java.lang.Object return values will be ignored
    • 4 interfaces to implement: IFormat{Reader/Writer}, Metadata{Store,Retrieve}
    • Client will probably want to start server via exec('...') call.

  • Discussions at Jason's
    • Google feedback
    • @Removing different sizes of registry icons
    • Storing protocol changes as diff
    • Hierarchical searching in protocols
    • Caching via BioFormats
    • Macnification ...
    • displaying metadata (table?)

  • Cat/Towards Beta4 : User Perspective
    • Q: There's data about the improvement re: tagging. A: Yes. See "Focus Group '08".
    • Without prompting: "Like the way that goes across the screen"
    • usability data — possible directions
      • automated analysis (movies, tracking, ...)
      • @Need to improve messaging/alerts/conversations about data
      • improved workflow (also route to publication)
    • big challenges
      • trust with data
        • How to we make the OMERO brand as/more trustable as/than Photoshop
      • confidence over what image "is" (change tech or change user)
        • Manage the "RTFM" level
    • Q: What do you mean "Manage growth"?
      • A: Curtis was saying, building for ourselves is a good strategy
      • However, we're past that stage.
      • Josh: Shrinked-wrap software
      • Ilya: Most shrinked-wrap software is a hodge-podge
      • Catriona: true of word, but not Final Cut
      • Carlos: Adobe Premiere was small, but now is big (lame users <--> trained professionals)
      • Cat: hodge-podge is often because the product was started 20 years ago
        • FinalCut is new. Was used to produce movies before released.
      • Jason: No one's mentioned that we're trying to dvelop for scientific discovery
        • New methods are coming (HCS, ...) that we don't support
        • They're minorities, but it's a new start
      • Cat: No matter what we do, we're constricted by how we can get funding
      • Jason: Need to decide the strategic decision
        • Next step? Some of the functionality are things that no one has ever done before
        • Jean-Marie: train the developer to view everything as an entire pipeline, not "this is cool"
        • "I've changed my way of coding,...you can solve any problem...but it's how you create the energy from a user standpoint"
      • Ilya: It's important to realize that this group has more expertise than a user has foresight to drive further use
        • "Failures because they were not controlled, but allowed to be driven by the users"
      • Is the decision we trust overselves? Or do we target particular groups for wide-spread use (publications)?
      • Jean-Marie: problem appear because we are more confident of the tool we provide
        • Click on the link and put that on your machine
        • Previously we were only targeting a safety zone
        • Cat: Is Pasteur the testbed for this? Cautious, critical, ...
      • Chris: 1) all shrinked-wrapped software falls into trap; need cash to do this (driven by bottom line)
        • 2) getting into whole institutional level can be difficult/slow
          • Jason: Very reactionary approach to users. Barrier to adoption ("wait on feature 'x'")
          • How do we strategically (not tactiley) guage the response of take-up?
          • Cat: Need a process for taking a moment to look at relationships, directions, ...
          • Jason: Imperial is saying "we want to drive lifetime..."
          • Ilya: Should organize as a set of use cases (not people)
          • Cat: Practical — "They're close to a paper..."
          • Andrew: Need places that will do the best marketing. E.g. Mario
          • Cat: But there's no way to make those decisions, and fit into dev cycle
          • Jean-Marie: The reactive mode has been important to get us where we are
            • Getting into a new domain is a slower reactive mode.
            • Still hard for devs to extract yourself. Make extensible for later on.
            • re: Washington - some things missing (not much) for whole-pipeline to go back to Photoshop
        • Chris: we have a very engineering (top-heavy) team. No sales, support, QA, ...
          • Jason: 2 anecdotes
            • "You guys scare the pants off me. We have 4 developers"
            • Company growing 40%. Marketing and applications.
              • Chris: Like institutions they have no infrastructure to get things installed
              • Cat: In the next grant, also staff for end-user websites, smoothing release process, etc.
    • From feature design to concept design
      • "unknown known" --> "unknown unknowns"
      • getting the right info at the right time/place
      • how to work with the dev team. probably everyone needs to adapt.
      • handling suite at different level of maturity
      • Jean-Marie: decision on how to respond. Development might take 6 months, and so there are no black holes.
        • How do we communicate? When? Coming back in 2, 3 months.
        • Burnt out some of the users, and if they don't see something coming, no feedback where their information goes, we get no answers
        • Better job of advertising to user, not just on trac
          • Cat: presumes that we can say "here's the strategic decision"
          • Lot of things that are put on the table, but since there is no strategic view, hard to say "X"
          • Need something visible/clear window X comes before Y.
          • Jason: In the history of ethnography, is there anything about the users being spoiled?
            • Fact is: we appreciate your feedback, but look at what you have.
            • Chris: Not done anywhere.
            • Cat: But it is, Nokia. Not user-centered design, but user-engaged design. "We won't do everything yo uwant"
            • Chris: I don't have one application (open-source) that I can interact with. We can't achieve it.
            • Cat: It's not that the users are spoiled. It's how we manage it. They don't need us; we need them.
              • What we can control, is how we manage/control information. The more we feedback, and build into system
              • Example: we learned that you have to tell the user ALL the things you have to upgrade at once
              • Have to apply to other things. We are completely over-relying on certain relationships
              • Choice: rely on ourselves; or keep using user/find way to work with them. But we need way to make that choice.
              • Ilya: Project needs to become more strategic, and that needs to become visible
                • There is a lot of information about the project, but it's all technical/tactical.
                • Brian: It's because we all are developers.
                • Ilya: But it's critical.
                • Jason: Vision exists, but loose and not articulated in a well-defined way. Ilya: Mission statement?
                • Chris Mission statement shows you what direction, but not what stops along the way.
                • Cat: It's about managing the process. Then we can do it.
            • Jean-Marie: Everything in two steps
              • feedback.o.o.uk send something back to users when they report a problem.
              • Will: A user has created a ticket. Because of anonymity, can I put their email address?
              • Curtis: We do that. But the newest version is much better about hiding CCs
    • Moving our relationship forward
      • Address key issues at the right level/time.
      • Jason: we'll find some things take one week
        • Where/who in this process is the responsibility to get the info back to the user? ("We'll get back in 2 years")
        • Cat: Need to make a decision about that. Ultimately the UI team has to take on more of the feedback loop.
        • What did Nokia do?
          • They have expert user groups who they deal with online.
          • Panel of ~5000 people in Scandinavia, who get software, then they go to online software/
          • Commitment: for every developer, they have 500 end users attached. Building right user base.
          • "Open Innovation Movement" (started at IBM). They know that they've exhausted users. Have to pay people.
          • Nokia handing out free phones in India (ethnographers). Chris: will they keep losing market share?
          • cF. Fujitsu teaching engineers how to talk to people
        • Ilya: In your domain? Instead of being a bit passive, but to _push_ the new things out to the users.
          • Cat: We can make a decision "We want to do X" then we tell users "No, you really want that"
          • Ilya: with WND-CHARM, over the past 2 years "I'm interested in analyzing your data"
            • Classic response: if you can count nuclei better than X, "then what good are you?"
            • How do you explain that to biologist? Can't give technical answer.
            • Josh: Exactly like "here's a server architecture"
            • Ilya: you have to take the reigns, and demonstrate that you've made something better
            • Cat: strategic about taster sessions, to get one lab to realize everything they need to get to a single publication
            • Will: One idea to help adoption, is to let people try out insight, before they install the server. Web account?
            • Chris: Biggest problem is that it's not representative of how you can use it.
            • Josh: Have to weigh what expectations they will recieve from the demo software that we give them? (Performance, bad demo data, ...)
            • Ilya: Have to go in and solve someone's problems better than they can.
            • Jason: 2 anecdotes
              • Dumping images into powerpoint and can't do anything with the images. "Why not OMERO?" "What's an OMERO?"
    • Josh: encoding process/learning effect
      • Chris: Don't make everything public
      • Ilya: But we aren't a company, ...

  • Josh's Presentation
    • What are the advantages of Jboss over Ice (vise versa)
      • None, needs a good admin to keep running and is fundamentally focused on web apps
      • Its good if you have an application server and you just want to plop something in
      • We are getting away from Jboss (might keep it if they already have an application server). We haven't had anyone come to use saying they want to do that
      • We are getting away from JBoss (By Beta4 maybe? - depends on client perspective) In the long term, yes. Migration of clients will be a pain.
    • Our server design allows data to be saved to the database, and then accessed by the process script engine
      • (so as a result, a script developer) - it is their responsibility to extract the data results from their analysis and make sense of it's presentation
    • Needs to be a survey and feedback mechanism before we get into clustering - what culture are we supporting? Blast?
    • Josh's cloud diagram doesn't articulate the interconnection between these different cloud regions
      • Jason: if we took what this cloud represents to the client today, the development of the solution would evolve
        • JM: Yes but we do need to narrow this cloud down and concentrate on some specific sub-areas of this diagram
        • Josh: Yes, we need to decide what resource constraints exists against this diagram

  • Chris&Colin / FS within our existing technology decisions
    • setting up monitor on colleague's directory, no change in 2 weeks, ==> ... (trust/privacy issues)
    • @Carlos sends Colin a link to Windows/Python FS-Events info
    • Jean-Marie: looking at the whole pipeline, what happens when the file is archived?
      • Chris: Adding hooks/methods to the interface isArchived(), isCompressed(), ...
      • Extensibility/pluggability: add a python script which implements the isArchived() interface
      • FileSystem browsing/stat bits
      • Want a java.io.File implementation, to allow manual importing, ...
      • like copying to a USB
        • Have to integrated as well, or we'll do ourselves a disservice
      • Ilya: files in userspace. how are you going to deal with that
        • Pixels have been extended to have a URI
        • Qualify where the pixels is.
        • Server updates the URI when users move files.
        • Glyphs of "this file has changed"
      • Ilya: Who's doing the driving?
        • Who deletes. Moved an Image to another directory. Why.
      • Josh's wish list:
        • Immutable files (and how do we handle users' control over the files. a copy?)
        • Control structures for files: /josh/docs/foo.doc so that I can overwrite an existing file
      • Not replacing import model. To have your data secure, then use the import.
        • You give the user options
      • Relationship to caching
      • Possibly create an OMERO.ftp process where we control (clippy-style) what the user can do
      • Mapping FS file structure to PDI
        • Naive answer: whatever data management decisions in OMERO are only in OMERO, don't reflect on the FS
        • What do the user meanS?
        • Do I want all the annotation/tagging informed from what I've done
    • Technical: Are you moving any data?
      • A: Will attempt to use bioformats directly to read the data
      • Ilya: User has you over a barrel when it comes to removing the file.
      • Brian: OMERO.fs to build an imaging cache. Our own archiving system.
      • Tech to store references to all files on various OMERO.fs servers (by hash sum)
      • Federation, finding the nearest copy (GEOME)
      • Ilya: User wants something out of the drop,
      • Jason: Maybe we have to control read/write
      • Playing hangman. Each click ...

  • Donald / Scripting
    • Ilya: I don't see the tie between the a/b/inputs
    • Ilya: Standard workflow - algoritm on a collection of images (or of regions)
      • Need the bit of code to handling the mapping of Image/ImageCollection
      • It's a key bit of the puzzle to not burden the algorithm developer with collections
      • Need iteratorOverCollection.py or convertDatasetToImage.py or roisFromScreen.py
      • Ilya: If you are writing bits in OMERO to let users collect images, ... it's death
    • Ilya: When does matlab startup/shutdown? Can control it.
    • Ilya: Maintenance of Matlab-Java bindings
      • A: scipy with mlabwrap seems to be on top of it
    • Does it work in Octave? Should, but hasn't been tested
    • The CellProfiler is not very objecty. A: Can convert it into an ROI
    • Josh: IS THERE GOODEVAL?? A: No.
    • ImageJ integration
      • What's it look like?
      • Possibly a MacroTemplate which python populates and writes to disk then "java -jar imagej.jar MacroTemplate"
      • >>OR<< Just have ImageJ as a client for full interaction.
      • Jason: What kinds of analysis tools do we want to develop? ImageJ plugins are "standard" in processing/image analysis
      • Are we doing next generation work.

  • Oliver / Messaging Server
    • shmem, socket, pipe communication
    • What are some examples of what you want to do? (what does it deliver)
      • HCS -> record images to OMERO
      • Post-screen analysis needs images from OMERO
      • Q: Further abstraction? A: Yes.
      • Concern: only 2 pieces under control — messaging server & omero
        • rest is closed-source. But, Arivis is signed on
        • doesn't scale with N collaborators
        • what really matters are the messages?
        • the message server has to define the messages/protocols
          • you have to buy in (cF. EtherCat)
      • Donald: would Ice not be a nice way to do it?
        • just an idea. This is exceptionally nice.
        • Brilliant to just have biologists control hardward from insight
      • LIM systems integration
        [+]
      • big companies say: here's all everything you need, if you buy it all from us
        • certainly true in HCS
        • starting to break...
      • has to be so easy and complete
        • e.g. audio.
      • we're not the problem
        • some partners don't have time
        • some are afraid of changes, must be convinced.
      • need to find 3 components which can be used as an example
        • should be no problem.
        • is it possible to control their GUI? (VNC, scripts, ...)
      • a cool example, "Stage in wrong place"
        • unclear where experimenter protocol is started
        • browser is perhaps a pretty good starting point
        • "start pulsing device 12"
        • notices that table is wrong spot
        • "move table to"
        • "table is in position"
        • then starts pulsing
        • "pulsing"
      • analysis on several images
        • browser asks for analysis
        • analyzer asks for data
        • analyzed
        • return data
      • is it worth setting up dummy system?
        • can you run a 100,000 images?
        • like the demo from Chris and Colin of dropping an image in a directory?

  • insight / jean-marie
    • interdepencey (cyclic): the tough stuff poses questions that we can't answer except holistically.
    • Prototyping!
      • how to have multiple version which users can see and choose between
      • would preferences help?
      • Cat: difference from other prototyping, it's so personal that the user can't evaluate without their own data
        • have to build working prototype, no lo-fi or hi-fi
        • Jason: they want to work with their prototype
        • something that can't be handled with our old way of working
        • Eclipse as a model, with versioned plugins so users can compare
    • HCS provides the deepest hierarchy (more than tagging) - Harry & multi-hierarchies
    • finding new insights in data view traversing across hierarchies
    • smart folders. Perhaps folders per protocol
    • Ilya: Can you switch from one hierarchy to another keeping selection?
    • Can always come back to the previous view. "Capture-Concept"
    • Google-map/Screen-map/Large-image-map : not hard but a new concept

  • Ola / web
    • "You have no tasks. Go add 10!"
    • Ilya: Can you share with anyone? (re: botnet)
    • sharing v. publication: one during your work one is after.
      • re: legal aspects with journal
      • part of sharing/publishing workflow is creating a new dataset, specifying the visualization
      • email contains a url? or url + session key
      • publication is wide-open
      • Josh:or as screen, so you can do formatting/positioning
        • Carlos: having positioning information would be great in clients
      • Jean-Marie: is the text wiki? how do other clients digest it?
        • A: HTML. But the share is on the application-side (during development time)
        • Storing as XHTML. Easiest.
    • Ties into OmeroSecurity?
      • Yes. "Mounting a share" is the metaphor. Slower. And doesn't show up in my regular view.
    • What kind of object in OMERO can be a "Task"?
      • Ilya: When did I take an Image?
      • When did I process it again? 3rd time, 4th time and what did I do those times?
      • Will: I want to pick up the tagging I was doing yesterday
      • Jean-Marie: definition of groups of actions
        • acquisition, tagging/manual data, analysis
      • Date in dataset or project name (experiment is annotated though somewhere)
      • Use TimestampAnnotation either globally or attached to the user
        • Needs a start/end, repeat, etc.
    • Viewer
      • Hide-all? For LOTS of images.
      • Pager infrastructure? Not yet.
      • We need a dataset with 5000 images would help.
    • Andrew: sunset clause? from when?
    • Ilya: the real power is public access

  • HDF
    • Position written into next grant
    • Talk to M. Dougherty
    • Definitely going to move forward

  • ImageJ
    • Client. Yes
    • Server-side? Perhaps discussion at the ImageJ meeting later this year.
    • Email list after conversation with Johannes/Wayne
    • first integration: use the data that's in the server for display
    • second: then can store their ROIs as an attachment
    • third: with strong need, we can look at converting to overlay
    • fourth: with strong need, we can look at running in the background
    • ideas:
      • storing macro as an attachment which produced the results
    • 2-4 weeks of just getting plugins done.
      • fixing beta3
      • use OmeroJava? For now, use RMI

  • Search
    • changing index schedule so that it doesn't lock
    • "can you upload a website and then google right away?

  • Delete
    • once you try to sole something simple like "Image" then
      • then what happens in SPW
      • which links do you break?
      • Ilya: deleting image from a well should be empty
      • propagate to the tag or not
      • Ola: Cleanup Wizard? "You have some unused items on your Desktop..."
      • Curtis: PDF-Dummy to link things (manually updating the ref count)
      • Brian: status deleted/failed/
        • Jean-Marie: Performance problem checking the flag on all the things
        • Chris: still slows down the overall transfer
        • Ilya: Just pushing the problem somewhere
      • DeleteService
        • Ilya: Based on deleting a MEX. But have to decide delete or unlink
        • Need a wizard which shows everything that you are possibly deleting
        • Curtis: Ways to intelligently flag stuff
        • Jean-Marie: Need the cleanup phase
        • Chris: Querying for orphans!
          • Problem: 100,000 orphans? Now What?
          • No structure to say what the PDFs mean, are/were linked to
      • "New Delete: here are the consequences."
      • "Old Delete: here are your orphans"
      • Cat: An image belongs to another user. Having an annotation, makes two owners, effectively.
        • Two owners want to delete: gone.
        • One owner: gets hidden/trashbin
        • Chris: I don't think it's practical because ownership is as simple as viewing an image.
        • Chris: Different priorities of annotations? Cat: All are making sense. Will: It's finding something.
        • Cat: Copying images when you attach.
        • Josh: Service needs to have the semantics "What will this delete?" "What will be orphaned?"
        • Jason: Don't want to force a value propsition
        • Ilya: Have to assert who the dominant owner is.
        • Ilya: Can publish a retraction, but can't get rid of the original
        • Will: If I attach something to a dataset, then I trust it won't change
        • Jason: conversation idea ... most people would ignore it because of the competitiveness
          • Marking something as a "shared" image to put it into a different state (community model)
          • Josh: Disagree
        • Have to support both models: protecting things and allowing delete
          • Accounting systems: audit and nothing gets deleted (Brian)
          • Chris: Naive approach because they have heavy pre-filtering
          • Ilya: Collaborative work is the decision to delete
          • Josh: Way for the event system/messaging to help primary owners to delete.
          • Chris: Need to help system to decide which links are overridable
            • Josh: then then client can find out what's overridable
            • Josh: But will still need the dialog box: "Primary wants to delete, do you agree?"
            • Ilya: is there a permission trap, where you can't see something attached?
            • Ilya: color code graph view of the things: Deleted/Could be Deleted/Things owned by others/Things blocking your delete !
            • Need to know what they are getting into
              • May be surprised when they go to another site
              • Habituation!
    • Todo
      • Start implementing static version
      • Use rules for possible GUI later
      • Need delete/unlink differentiation
      • Eventually message for confirmation
      • Use Sessions/Events to offer additional deletes
      • Permissions.trashContainer
      • Implemented Client size

  • Lunch discussion
    • future: protocol, presentations,

  • Josh multiple pixels per image
    • Ilya: Die!
    • No one gets it, who cares?
    • example: deconvolve. Dataset pointing to the deconvolved pixels, no way to see other pixels on the same image
    • deconvolve should be another image? That's what we do now.
    • Default pixels, what is used by default. That's what users think is *the* image
    • Image link should have a link to the default pixel.
    • Donald: I don't like multiple pixels, it's evil.
    • Jason: in presentations, when mentioning multiple pixels, people look as if you said something stupid
    • Event log is complicated to use too
    • Image concept has 2 responsabilities
    • images are just a view on the data
    • You have something that's yellow... provenance model, call it what you will
    • What are you tagging? Looks the same to users.
    • Josh: information should be normalized, in the right place.
    • Chris: one image is raw, one is projected, how to make statements about each image?
    • how to say: "these 2 pixels are more alike than other 2" - is possible
    • tool don't have the functionality to handle multiple related pixels-
    • Donald: doesn't satisfy the way we work. We want to say how pixels are derived.
    • Brian: It's a workflow problem, Image containers doesn't do a good job for that.
    • Users do that anyway, they can add a new set of pixels. Now what?
    • Josh: History is the new container
    • Jason: must also support the algorithm case, not just UI
    • Ilya: the yellow brick road: It changes a lot of fundamentals.
    • Chris: Images have only name and description, nothing else.
    • Andrew: what happens to info attached to image? ROIs?
    • Chris: they should be in the pixels, not image.
    • Josh: we lack linkage between dataset and pixels.
    • Chris & Donald: Collapse Image and Pixels, removing multiple pixels completely.
    • Next Major XML Release? (July) with light sources and xrays
      • No. Implement and see what it does what we want.
      • Write it up on the ome-xml trac, get suggestions
    • From Ilya 3 requirements: (and all tools don't have this functionality)
      • 2 Pixels are more related than others (struc)
      • Container speaks about all the pixls (struc)
      • "image" / "pix" are convoluted over 400 years of microscopy

  • blitz libraries
    • naming classes/libraries
    • what methods are implemented
    • coding style
    • where's the committe?
      • similar to in server
      • no underscores, etc.
      • come to a conclusion
      • convention: load is top to bottom, find is bottom to top
      • store it as Client.ice under blitz/resources/omero/Something with doxygen in Javadoc

  • OME model adherence / properties
    • Possible to come up with groups?
      • SPW, for example
    • Do we discuss it after the fact?
    • handling creatiaon defate as default

  • Big Images
    • use cases: large plane images 22000x40000
    • things are getting slow
    • Andrew: a "big" image — does the server treat it differently
      • both sides need to adjust
      • provide an exception which says "BigImage"
      • new programming interface with sliding window
      • image pyramids
        • calculating multiple resolutions
        • pre-rendering scaling
        • subsampling. visualization impression is unsatisfactory (algorithm based)
    • reasonable limit: .1 microns with slide 3 inches, 750,000 pixels a side
    • future-proofing requires 1 Tb / plane
    • compression
    • does the metadata describe a tiling or a stitching?
      • use case: hospital basement full of slides that should all be digitized (surgical biopsy) Aperio.
      • one implementation are like geo tiffs. algorithm to do registration
      • optics don't let you perfectly align them
      • then they are split back apart
      • in that case there's less need for metadata to get the overlaps settled.
      • retiling is a common implementation
      • becomes an issue for bioformats. already partially have a tile api.
      • interface.whatAreTheTiles()
      • answer: we don't really need metadata changes
    • there are both an optimal client tiling and an optional server tiling
    • interface.setScaling(double) anything at 1 or above returns the whole image
    • handle all of this on the server with exceptions
    • Consider: 3D or non-XY tiles/blockings
    • aperio

Breakout discusions ====================================================================

  • Breakout list
    • List of needed EventLogs - SESSION, ...
    • Search/ontological
    • Benchmark OMERO with WND-CHARM
      • Mime-type
      • parllel IO
      • tagging tags or using events
      • typing: patching v. fork
    • HDF
    • Image/Pixels
    • OME-Profiles/Adherence/Nullability
    • ImageJ/other integration/GLUE
    • FullStack for next month
    • Demo possibilities: vmware, live-dvd, ...
    • production server for weblitz (nginx?)
    • naming weblitz
    • blitzconnector, gateway
    • hierarchy/trash/delete/imported flag/...

  • Requirements discussions
    • Image/Pixels discussion
    • OME-HDF
    • ROIs: supporting ImageJ rois, for example
    • Relaxing the model
    • converting csv/xls (re: Jun20/Will & printing)
    • Server scripting (Java)
    • standalone viewer
    • ImageJ repository
    • cluster support
    • ImageJ client
    • Discussion on documenting what properties are supported (Jun20/Melissa)
    • Defining the weighting of all properties (Jun20/Andrew)
    • Which scripting interface would users like to use?
    • Stemming and Tokenization and Thesauri
    • Jason's presentations stored in OMERO (presentation as publishing) ... audio? ... "AskJason.com".
    • Look at Josh's cloud diagram and decide on some priorities?
    • what's a task/event in the calendar? And how is that modelled? ICalAnnotation? (repalces Timestamp?)

  • Monday "Coding Day"
    • JvmLink

  • Concrete todos
    • Document or implement goodeval()
    • encrypting password
    • examine one story/feature from top to bottom. include as many people as possible to determine process.
      • simple workflow things: projections, AVI,
      • screen
      • sharing/import/export
      • search
    • Rename OmeroJ->OmeroJava




Document Actions