June 2008 Developers Meeting
Developers meeting held in Dundee
The goal of this meeting is to define goals and milestone for the Beta4 releases of [old-link http://trac.openmicroscopy.org.uk/omero/milestone/3.0-Beta4] OMERO.server and its [old-link http://trac.openmicroscopy.org.uk/shoola/milestone/3.0-Beta4] clients.
At this meeting, we will:
Breakout discusions ====================================================================
At this meeting, we will:
- define Beta4 as well as any next betas and the RC series
- ensure all tasks are ticketed and assigned
Suggested other topics (to be scheduled include):
- OME-HDF
- The Message Server proposed by Lars, Oliver and Aline
- Extensions in OME-XML
- Modelling and meta-modelling of OME-XML in our software (xsd-fu, Model Driven Architecture, etc.)
- LED, X-Ray, ROI model, Metadata only files, OME-HDF, Relaxing the model, 3rd party data
- Image/Pixels Data Model
- WebAdmin priorities list
- Used licenses (Mozilla, Apache, etc.)
- Discussing naming and other models (e.g. xray) - ConferenceCall 2008-05-22
- Are we missing anything in 'developer support'?
- Are we missing anything in 'user suport'?
- The Omero clients 'style bible' - how to create, what it should cover, etc.
- Beta 3 - outstanding issues from user perspective
- Notes from the field: opportunities and problems identified during the ethnographic investigations.
- A summary of key findings from the Dundee and Pasteur scientist surveys.
Draft Programme (please make comments etc as necessary)
Day 1 (June 20th)
Time | Topic | Lead(s) | Comments |
09:00 | Welcome Intro, Summary | ||
09:30 | Bio-Formats OMERO.importer | Melissa Brian | Status and evolution |
10:30 | File Formats | Curtis? | OME-HDF |
11:30 | Data Model | Andrew | Evolution, Image/Pixels |
12:30 | Lunch | ||
13:30 | OMERO.editor | Will | presentation+ Data Model from Users point of view |
14:30 | OME server | Ilya | status |
15:30 | LOCI Apps | Curtis | status |
16:30 | Discussion | ?? | extra time |
Day 2 (June 21st)
Time | Topic | Lead(s) | Comments |
09:00 | Coffee, setup, gossip | ||
09:00 | Usable Image | Catriona | Beta3 status, evolution, methodology |
10:00 | OMERO.server | Chris Josh | status |
11:00 | OMERO.fs | Colin Chris? | demo |
12:00 | OMERO.scripts | Donald | status and evolution |
13:00 | Lunch | ||
14:00 | Messaging server | Lars | proposal |
15:00 | OMERO.insight | Jean-Marie | status evolution, integration |
16:00 | OMERO.web | Ola | status evolution |
Day 3 (June 22nd)
Time | Topic | Lead(s) | Comments |
10:00 | Discussion | Jason | |
12:30 | Lunch | ||
14:30 | Plans | Jason | tickets, website etc. |
Notes
- History/Intro
- Jason & Ilya at all
- First Dundee: 2004...
- Goal: get what we want to do for the next year or so
- catering: lunch, etc.
- have to talk fast
- looking at downloads, registry
- New Chinese dot on registry
- Grants pending: 7 positions from Welcome Trust, Usablity for 6 positions and a PhD
- Funding for Baltimore 2 FTEs until review next year
- Everyone healthy/happy?
- Getting to Jason's tonight: taxi, etc.
- Formats
- Brian
- Q: Is an import a copy? A: Yes, but usually the history feature is used to come back (after bug report) and retry fails
- Curtis: we're going to be using the ImportLibrary directly, too.
- Have been some build issues.
- Would like to see the newest version always compatible
- Curtis: Try to not just change things every: "I'd like to change filters now"
- Brian: Would be good to look at how we communicate
- Curtis: Fine with changing OMEROMetadataStore so that it compiles with stubs
- Chris: In the past, changing of methods is trickier than adding new methods
- Legacy code which mismatches with the code-generated bits.
- Need to setup automated testing of Bioformats jar in importer
- Q: Advantage of server-side import? A: speed, just uploading file, getting files off the scopes
- Importing at Pasteur: caching server (1TB), processed daily to long-term storage
- dummy client on cache server to import
- Q: option on OMERO.fs? A: Handle the upload
- Jason: Hard-core reality. Got to get off the scope, initiate import and run in background
- Brian: Would like to move import to service which starts a daemon
- Has to be light-weight process: ftp file over, then control another service on the server to deal with OriginalFile
- ImportDaemon (registers with OmeroGrid) and ImportService
- Can't install anything on the scopes.
- Single user environment
- Might need OMERO.fs/SMB mount using no install (PUSH)
- Curtis: new formats is not necessarily importer-dependent
- No reason we can't expand everything to support new formats
- Ilya: Have to be a bit careful here
- Real lack of information management in science
- Don't want to become that. Will unfocus everything.
- "You guys deal with images. Why don't you handle ....?"
- Jason: FCS - time-series data merged with image. But it's microscopy, so...
- Ilya: If you are the only ones doing it well, ...
- Curtis: Step-by-Step. Start with what we want.
- Cross-language support
- Curtis: We can extend JVMLink
- Josh: What about Ice?
- Curtis: Sure, what about deployment?
- Now we don't need our scripting language: use beanshell
- Chris: We never imagined people downloading and haxxoring importer from svn
- Need to update for and support that
- More flexible, support interfaces
- Melissa
- Q: Performance not infrastructure based, right? A: No only formats (ND2)
- Specification: will always want to be adding more (electrophys: AUDIO). There's a tension there.
- Need audio attached to a video talking about the image
- "soundtrack"
- learning project uploading to OMERO
- Nothing needed at the moment
- OME-XML
- Currently OMEROMetadataStore is write only. Need to implement the interface
- Curtis has been thinking about GUI for converting between formats
- Maybe not anymore. ImageJ has something now: VirtualStacks (check option in importer)
- Had wanted to save RAM.
- This may be just improving ImageJWriter
- Dev documentation
- Brian: same problem with the importer
- Ilya: "Keeps the second-rate developers out".
- Loci update
- Jason: previous site there was a list "You can use this software with ..."
- Curtis: Now under "Applications"
- The visual — big, blue — was great
- A big money line (screenshot)
- Jean-Marie: More explicit under the metadata section in the format list
- Good for building a client which supports the metadata
- Chris/Donald: was using OMERO from matlab to dump, work, push
- Need a way to know what is supported
- A trac page where each format has its metadata-support listed
- Could also link the commit which supports new metadata ("r4096")
- Curtis: a lot of work, perhaps new interface (mirror of MetadataStore)
- isThisSupported() - one per property
- default is false
- the code documents everything that is supported
- write a script which would generate HTML of what is also supported.
- Chris: generate Trac code.
- Chris: Could it be n a single data file. Code is a bit of pain. Good for understanding.
- Brian: people ask often "do you support field X on Y"
- Josh: possible to have coarser interfaces which short-cut checking all the properties
- Curtis: Can enumerate the properties supported, or list Image to say all of Image
- Donald: Would really like to see support: "Image", "Instrumentation", ...
- Andrew: Brian had asked for matrix for what populates which tables
- Report from the field
- lifetime imaging, high-content
- "What we really need is support for Becker-Hickel (sp?) format?"
- He knew: STD in bioformats (even though not dev), AND that support in bioformats had been turned off
- "Why?" --> "Test-data..."
- Curtis
- OME-XML
- Writer
- Expanding metadata store
- Staying on same page. What omero supports and if we're converting?
- MetadataStore as the glue workhorse
- OME-XML which doesn't validate because of incomplete data _NOT_ in the proprietary file
- What are we supposed to do?
- Want to be able to fully populate everything that I have!
- Our software is producing non-validating code
- Not ok.
- See [href="http://cvs.openmicroscopy.org.uk/tiki/tiki-index.php?page=ConferenceCall+2007-12-27&highlight=objective] http://cvs.openmicroscopy.org.uk/tiki/tiki-index.php?page=ConferenceCall+2007-12-27&highlight=objective
- We need a plan, document, and implement
- Brian: prioritize this, please.
- Josh: read the thread for the propsed suggestions
- Jean-Marie: We can compare an example from Will later to make this complete
- OME-TIFF
- Reworked the reader
- Multi-file datasets: validator doesn't validate across files
- Ilya: suggestion from NIST for some kind of naming convention of microscopy images
- Especially in multi-file formats
- Curtis: Can only ever be a recommendation
- Chris: Do the microscopy producers care? Not in that realm.
- Not in that realm because there are no guidelines
- All instruments in some domains come with a certification for a guideline
- Write up recommendation based on LOCI
- But the current solution about embedding the file names is pretty nice.
- Josh: Putting a flag in the file? Not really.
- Add to golden start system
- Josh: Possibility of someone accidentally/purposefully breaking the regex
- OME-HDF
- re: Matt Dougherty
- http://www.medsbio.org/meetings/BNL_May08_imgCIF_Workshop_Report_files/Dougherty_files/imageCore_V01-RFC1.pdf
- http://www.medsbio.org/meetings/BNL_May08_imgCIF_Workshop_Report_files/Dougherty_files/MD_MEDSBIO_22May08.pdf
- http://www.medsbio.org/meetings/BNL_May08_imgCIF_Workshop_files/Dougherty_Abstract.pdf
- Beyond the 5D model
- On the pixels
- Rasterizing on the Channel dimension, with info on how to cut it up
- bioformats supports N dimensions, "subC" with rasterizing
- re: Matt Dougherty
- Other
- JVMLink -> Ice
- Viewers/converters in ImageJ
- Documenting what metadata is supported
- Jason: something on the grant for someone working with formats
- Curtis: several students wiscan, jvmlink, flow-cytometry, visbio would profit from a formats guy from grant
- Jason: What does the plan need to look like coming out of the meeting?
- Curtis: Who, what, when?
- Monday as post-conf coding day!
- OME-XML
- Brian
- conversation about communication
- possibility of using Adobe for whiteboarding, etc.
- unconference-style to get through all the points we want to discuss (hdf,...)
- Data Model/Andrew
- ROI
- Curtis: Have you referred to the notes on ROIs
- Curtis: Would like input to compare imagej & visbio
- Ilya: Is there no requirement for the object to be closed?
- A: Not currently. There are closed shapes.
- Ilya: There must be a way to determine if a fill is possible.
- Draw a rectangle with 4 <lines/>
- Donald: Advantages if you define it with <rect/>. Do queries on the types.
- Andrew: The intent was to use the primitives provided by OpenGIS (http://www.opengeospatial.org/standards)
- Ilya: Good to use some form of standard and tools that people understand
- Donald: surfaces, ...
- Ilya: are there anything not translatable?
- Not a Z toolbar in google earth? But there is a time concept
- time-series data? Yeah, but our binary data is a different order.
- ROI structure here is completely unnormalized
- Curtis: Would like to have some "share" concept.
- Andrew: null z or t means over whole dimension
- Curtis: fine.
- Josh: could use an annotation if you really wanted to.
- Ilya: Have to calculate the continuity
- <inverse/>
- SVG does allow for defining the orientation ("fill left-hand side", vectors)
- Doesn't allow for two lines in z
- Jean-Marie: use the derivation to say that they are an angle
- Curtis: Just attach a 3d mesh
- Donald: use "connector" to connect two ROIs
- Relaxing the model
- Omitting Pixel Data
- "open metadata environment"?
- <metadatafile filename="..." hash="..."/>
- Ilya: PixelsRef with LSID
- stylistic issue
- Josh: Let's use pass XML over an API, dump whole DB to a single file as backup
- Brian: Different representation of pixels, compressed
- Ilya: you need the pref
- Curtis: Have the <MetadataFile file=""/> be anything that bioformats can read?
- Chris: We originally didn't want to do the metadata companion files
- Jason: we ate that poison pill when we did OME-TIFF
- Josh: Gets worse when we do HDF
- Companion file is not a starred profile
- Capturing extra metadata on the scope machine
- Single companion file for multi-binary files
- Putting the carrots on the main page
- doing the link at the provenance level (OriginalFile: STs)
- Not in model, nor is provenance (only in XSD)
- Consider it without server concept, pure exchange format
- non-image extra files (audio, etc.) is fine
- Curtis: Want <plane/> in <pixels/> so it can't be the bit that disappears
- Just make <bindata/> optional.
- DICOM flag: GE doesn't read SIEMENS doesn't read ...
- temporary LSIDs pointing to /dev/null
- Omitting Pixel Data
- ROI
- Relaxing
- enumerations
- Josh: like omero, using a reference
- Chris: how to handle the resolution?
- Ilya: modelling discussion. Enumeration or not.
- Just define the process for defining a new enumeration.
- Curtis: I understand what you are saying that people will abuse "Other"
- "Validation does not equal quality"
- Multiple Quality Levels: validates but did you do a "No-No"
- Will solve LOCI's problem.
- Jean-Marie: Interesting to see what Will is doing, need to look at whole pipe-line
- Need a formal way of describing, visualizing, ... Otherwise develop own way
- cF. Compare XRay tomography's own OME schema
- Chris: 30+ formats and none of them can fill out the objective
- Ilya: Objective wasn't done arbitrarily
- Was defined to show future manufacturers how to define objective
- But we still need to deal with it
- "7 of 20" but we need the definition. Once all 20 of 20, then you drop the optionals.
- Sociology/social engineering
- Ilya: A service we provide. Obvious that I need one number to go from provisional to full. Gold star!
- Cases
- Objective,
- OME-Provisional, "Level -1"
- Who's going to do the evaluation? Put up the table with examples of each ranking.
- Curtis: for each format someinfo
- for every property (entity shortcuts): N/A, NO, PARTIAL, YES
- format quality ratio: is it supported? low == not rich
- each property:
- weighting the properties, importance of properties, level of OME
- four stars : import level, publication time, ???
- enumerations
- 3rd party
- Andrew: want a people to look at the definitions that we have and pick one
- Not just an <any/> yet
- hook a block one level lower
- pdf attached: what does the import/export workflow look like
- Ilya: have to be careful where the original format is important (pdf/mp3/...)
- will/editor
- Ontology lookup: Curtis — going the right direction with that
- Ilya: results of timed experiment show up with PI on date for review
- future searching: Ilya — post-processing the returned protocols for hierarchical structure
- Curtis: Is there a way when they create a new block "Filter" using the OME model?
- Are allowed to link to another file but not to another element?
- Some use cut-n-paste.
- Josh: what happens if only one file gets copied? A: Only local link.
- Ilya: You have the MAGE block. Have you tried...
- They have a nice description of a biological experiment
- Problem: terminates in a microarray assay
- But Treatment, protocols, incubations, ... new reagent.
- Recursive: treatment, ... on new reagent
- Data collection, instrument, Merck index entries
- Reason: if you know enough about how you did the experiment, you can draw inferences (computationally)
- Jason: Defining small parts of structure, and use those
- Start typing "LightSource" template is applied, then the list gets computed, ...
- MAGE is only useful if there is a redbook on the server
- Summary
- Easy to fill out template of : instruments, ...
- Search facility for : "light source",...
- Making this easier for fluid-handling robots like Ilya
- sites where you have to fill out a protocol before you can use the scope
- have to institue draconian measures because digital notebooks are tougher than paper
- . first step is having things to cut-n-paste
- . then point to other templates (perhaps data model)
- data model: providing (file type) a list of filter sets.
- Ilya: Has to be a carrot. Will: The only rule has been to make it easier to use
- Andrew: Is the piece of paper a critical step?
- Print and be remembered that it was printed
- "Do you need to update anything?"
- "I see you haven't finished your protocol"
- mini-mobile (iphone) is a possible idea
- have editor generate html which interacts with the server
- ilya
- "solving an imaging problem one at a time is not a solution"
- Q: Any other tissue based (as opposed to hormonal levels, running through mazes) to differentiate mouse age/gender?
- Not really
- how many HC screens? several. 2 dozen largest datasets (conservative)
- DAE no longer stores all the state (and re-create it after crash),
- Instead Jobs create new jobs, then what dependencies are release (WAITs)
- checkpoints!
- less OME overhead
- Ability to remove a feature due to too little signal
- Performance test on omitting features: no change
- One of two parameters: how many features do you keep (10-90%) performance falls off on the ends
- to increase execution, you'd have to remove the good features
- but once you have the features computed, they're enormously valuable
- Spend most time building classifiers and jiggering
- Donald: What's the classifier? Weighted-nearest neighbor (Euclidean distance)
- Two kinds of thresholding
- hard: lowest 1/3
- soft:
- What's the classification like when you don't throw away features? No change.
- Q: Did preprocessing increase the classification?
- Didn't help (segmenetation). Josh: but you may be already overly specialized on non-preprocessing
- True, and may help performance.
- Like Nearest neighbor for other qualities: easy to ask where images are in marginal probabilty space
- Almost moved beyond classification, and now to image similarity (error rate isn't always the most informative)
- Did you try to decrease the tile window?
- Trying right now a computing window.
- Do register knees via checking for a peak (a bit of preprocessing)
- pharynx aligned on the scope
- missing centromere screen: looking for dot(-absence) in goo, ...
- Painters
- You can vary training/testing (round-robin, 50 cycles of random splitting, cross validated)
- No subsampling
- Two kinds of thresholding
- jvmlink discussion
- Curtis will look into what needs to go into hashmap implementations
- The java.lang.Object return values will be ignored
- 4 interfaces to implement: IFormat{Reader/Writer}, Metadata{Store,Retrieve}
- Client will probably want to start server via exec('...') call.
- Discussions at Jason's
- Google feedback
- @Removing different sizes of registry icons
- Storing protocol changes as diff
- Hierarchical searching in protocols
- Caching via BioFormats
- Macnification ...
- displaying metadata (table?)
- Cat/Towards Beta4 : User Perspective
- Q: There's data about the improvement re: tagging. A: Yes. See "Focus Group '08".
- Without prompting: "Like the way that goes across the screen"
- usability data — possible directions
- automated analysis (movies, tracking, ...)
- @Need to improve messaging/alerts/conversations about data
- improved workflow (also route to publication)
- big challenges
- trust with data
- How to we make the OMERO brand as/more trustable as/than Photoshop
- confidence over what image "is" (change tech or change user)
- Manage the "RTFM" level
- trust with data
- Q: What do you mean "Manage growth"?
- A: Curtis was saying, building for ourselves is a good strategy
- However, we're past that stage.
- Josh: Shrinked-wrap software
- Ilya: Most shrinked-wrap software is a hodge-podge
- Catriona: true of word, but not Final Cut
- Carlos: Adobe Premiere was small, but now is big (lame users <--> trained professionals)
- Cat: hodge-podge is often because the product was started 20 years ago
- FinalCut is new. Was used to produce movies before released.
- Jason: No one's mentioned that we're trying to dvelop for scientific discovery
- New methods are coming (HCS, ...) that we don't support
- They're minorities, but it's a new start
- Cat: No matter what we do, we're constricted by how we can get funding
- Jason: Need to decide the strategic decision
- Next step? Some of the functionality are things that no one has ever done before
- Jean-Marie: train the developer to view everything as an entire pipeline, not "this is cool"
- "I've changed my way of coding,...you can solve any problem...but it's how you create the energy from a user standpoint"
- Ilya: It's important to realize that this group has more expertise than a user has foresight to drive further use
- "Failures because they were not controlled, but allowed to be driven by the users"
- Is the decision we trust overselves? Or do we target particular groups for wide-spread use (publications)?
- Jean-Marie: problem appear because we are more confident of the tool we provide
- Click on the link and put that on your machine
- Previously we were only targeting a safety zone
- Cat: Is Pasteur the testbed for this? Cautious, critical, ...
- Chris: 1) all shrinked-wrapped software falls into trap; need cash to do this (driven by bottom line)
- 2) getting into whole institutional level can be difficult/slow
- Jason: Very reactionary approach to users. Barrier to adoption ("wait on feature 'x'")
- How do we strategically (not tactiley) guage the response of take-up?
- Cat: Need a process for taking a moment to look at relationships, directions, ...
- Jason: Imperial is saying "we want to drive lifetime..."
- Ilya: Should organize as a set of use cases (not people)
- Cat: Practical — "They're close to a paper..."
- Andrew: Need places that will do the best marketing. E.g. Mario
- Cat: But there's no way to make those decisions, and fit into dev cycle
- Jean-Marie: The reactive mode has been important to get us where we are
- Getting into a new domain is a slower reactive mode.
- Still hard for devs to extract yourself. Make extensible for later on.
- re: Washington - some things missing (not much) for whole-pipeline to go back to Photoshop
- Chris: we have a very engineering (top-heavy) team. No sales, support, QA, ...
- Jason: 2 anecdotes
- "You guys scare the pants off me. We have 4 developers"
- Company growing 40%. Marketing and applications.
- Chris: Like institutions they have no infrastructure to get things installed
- Cat: In the next grant, also staff for end-user websites, smoothing release process, etc.
- Jason: 2 anecdotes
- 2) getting into whole institutional level can be difficult/slow
- From feature design to concept design
- "unknown known" --> "unknown unknowns"
- getting the right info at the right time/place
- how to work with the dev team. probably everyone needs to adapt.
- handling suite at different level of maturity
- Jean-Marie: decision on how to respond. Development might take 6 months, and so there are no black holes.
- How do we communicate? When? Coming back in 2, 3 months.
- Burnt out some of the users, and if they don't see something coming, no feedback where their information goes, we get no answers
- Better job of advertising to user, not just on trac
- Cat: presumes that we can say "here's the strategic decision"
- Lot of things that are put on the table, but since there is no strategic view, hard to say "X"
- Need something visible/clear window X comes before Y.
- Jason: In the history of ethnography, is there anything about the users being spoiled?
- Fact is: we appreciate your feedback, but look at what you have.
- Chris: Not done anywhere.
- Cat: But it is, Nokia. Not user-centered design, but user-engaged design. "We won't do everything yo uwant"
- Chris: I don't have one application (open-source) that I can interact with. We can't achieve it.
- Cat: It's not that the users are spoiled. It's how we manage it. They don't need us; we need them.
- What we can control, is how we manage/control information. The more we feedback, and build into system
- Example: we learned that you have to tell the user ALL the things you have to upgrade at once
- Have to apply to other things. We are completely over-relying on certain relationships
- Choice: rely on ourselves; or keep using user/find way to work with them. But we need way to make that choice.
- Ilya: Project needs to become more strategic, and that needs to become visible
- There is a lot of information about the project, but it's all technical/tactical.
- Brian: It's because we all are developers.
- Ilya: But it's critical.
- Jason: Vision exists, but loose and not articulated in a well-defined way. Ilya: Mission statement?
- Chris Mission statement shows you what direction, but not what stops along the way.
- Cat: It's about managing the process. Then we can do it.
- Jean-Marie: Everything in two steps
- feedback.o.o.uk send something back to users when they report a problem.
- Will: A user has created a ticket. Because of anonymity, can I put their email address?
- Curtis: We do that. But the newest version is much better about hiding CCs
- Moving our relationship forward
- Address key issues at the right level/time.
- Jason: we'll find some things take one week
- Where/who in this process is the responsibility to get the info back to the user? ("We'll get back in 2 years")
- Cat: Need to make a decision about that. Ultimately the UI team has to take on more of the feedback loop.
- What did Nokia do?
- They have expert user groups who they deal with online.
- Panel of ~5000 people in Scandinavia, who get software, then they go to online software/
- Commitment: for every developer, they have 500 end users attached. Building right user base.
- "Open Innovation Movement" (started at IBM). They know that they've exhausted users. Have to pay people.
- Nokia handing out free phones in India (ethnographers). Chris: will they keep losing market share?
- cF. Fujitsu teaching engineers how to talk to people
- Ilya: In your domain? Instead of being a bit passive, but to _push_ the new things out to the users.
- Cat: We can make a decision "We want to do X" then we tell users "No, you really want that"
- Ilya: with WND-CHARM, over the past 2 years "I'm interested in analyzing your data"
- Classic response: if you can count nuclei better than X, "then what good are you?"
- How do you explain that to biologist? Can't give technical answer.
- Josh: Exactly like "here's a server architecture"
- Ilya: you have to take the reigns, and demonstrate that you've made something better
- Cat: strategic about taster sessions, to get one lab to realize everything they need to get to a single publication
- Will: One idea to help adoption, is to let people try out insight, before they install the server. Web account?
- Chris: Biggest problem is that it's not representative of how you can use it.
- Josh: Have to weigh what expectations they will recieve from the demo software that we give them? (Performance, bad demo data, ...)
- Ilya: Have to go in and solve someone's problems better than they can.
- Jason: 2 anecdotes
- Dumping images into powerpoint and can't do anything with the images. "Why not OMERO?" "What's an OMERO?"
- Josh: encoding process/learning effect
- Chris: Don't make everything public
- Ilya: But we aren't a company, ...
- Josh's Presentation
- What are the advantages of Jboss over Ice (vise versa)
- None, needs a good admin to keep running and is fundamentally focused on web apps
- Its good if you have an application server and you just want to plop something in
- We are getting away from Jboss (might keep it if they already have an application server). We haven't had anyone come to use saying they want to do that
- We are getting away from JBoss (By Beta4 maybe? - depends on client perspective) In the long term, yes. Migration of clients will be a pain.
- Our server design allows data to be saved to the database, and then accessed by the process script engine
- (so as a result, a script developer) - it is their responsibility to extract the data results from their analysis and make sense of it's presentation
- Needs to be a survey and feedback mechanism before we get into clustering - what culture are we supporting? Blast?
- Josh's cloud diagram doesn't articulate the interconnection between these different cloud regions
- Jason: if we took what this cloud represents to the client today, the development of the solution would evolve
- JM: Yes but we do need to narrow this cloud down and concentrate on some specific sub-areas of this diagram
- Josh: Yes, we need to decide what resource constraints exists against this diagram
- Jason: if we took what this cloud represents to the client today, the development of the solution would evolve
- What are the advantages of Jboss over Ice (vise versa)
- Chris&Colin / FS within our existing technology decisions
- setting up monitor on colleague's directory, no change in 2 weeks, ==> ... (trust/privacy issues)
- @Carlos sends Colin a link to Windows/Python FS-Events info
- Jean-Marie: looking at the whole pipeline, what happens when the file is archived?
- Chris: Adding hooks/methods to the interface isArchived(), isCompressed(), ...
- Extensibility/pluggability: add a python script which implements the isArchived() interface
- FileSystem browsing/stat bits
- Want a java.io.File implementation, to allow manual importing, ...
- like copying to a USB
- Have to integrated as well, or we'll do ourselves a disservice
- Ilya: files in userspace. how are you going to deal with that
- Pixels have been extended to have a URI
- Qualify where the pixels is.
- Server updates the URI when users move files.
- Glyphs of "this file has changed"
- Ilya: Who's doing the driving?
- Who deletes. Moved an Image to another directory. Why.
- Josh's wish list:
- Immutable files (and how do we handle users' control over the files. a copy?)
- Control structures for files: /josh/docs/foo.doc so that I can overwrite an existing file
- Not replacing import model. To have your data secure, then use the import.
- You give the user options
- Relationship to caching
- Possibly create an OMERO.ftp process where we control (clippy-style) what the user can do
- Mapping FS file structure to PDI
- Naive answer: whatever data management decisions in OMERO are only in OMERO, don't reflect on the FS
- What do the user meanS?
- Do I want all the annotation/tagging informed from what I've done
- Technical: Are you moving any data?
- A: Will attempt to use bioformats directly to read the data
- Ilya: User has you over a barrel when it comes to removing the file.
- Brian: OMERO.fs to build an imaging cache. Our own archiving system.
- Tech to store references to all files on various OMERO.fs servers (by hash sum)
- Federation, finding the nearest copy (GEOME)
- Ilya: User wants something out of the drop,
- Jason: Maybe we have to control read/write
- Playing hangman. Each click ...
- Donald / Scripting
- Ilya: I don't see the tie between the a/b/inputs
- Ilya: Standard workflow - algoritm on a collection of images (or of regions)
- Need the bit of code to handling the mapping of Image/ImageCollection
- It's a key bit of the puzzle to not burden the algorithm developer with collections
- Need iteratorOverCollection.py or convertDatasetToImage.py or roisFromScreen.py
- Ilya: If you are writing bits in OMERO to let users collect images, ... it's death
- Ilya: When does matlab startup/shutdown? Can control it.
- Ilya: Maintenance of Matlab-Java bindings
- A: scipy with mlabwrap seems to be on top of it
- Does it work in Octave? Should, but hasn't been tested
- The CellProfiler is not very objecty. A: Can convert it into an ROI
- Josh: IS THERE GOODEVAL?? A: No.
- ImageJ integration
- What's it look like?
- Possibly a MacroTemplate which python populates and writes to disk then "java -jar imagej.jar MacroTemplate"
- >>OR<< Just have ImageJ as a client for full interaction.
- Jason: What kinds of analysis tools do we want to develop? ImageJ plugins are "standard" in processing/image analysis
- Are we doing next generation work.
- Oliver / Messaging Server
- shmem, socket, pipe communication
- What are some examples of what you want to do? (what does it deliver)
- HCS -> record images to OMERO
- Post-screen analysis needs images from OMERO
- Q: Further abstraction? A: Yes.
- Concern: only 2 pieces under control — messaging server & omero
- rest is closed-source. But, Arivis is signed on
- doesn't scale with N collaborators
- what really matters are the messages?
- the message server has to define the messages/protocols
- you have to buy in (cF. EtherCat)
- Donald: would Ice not be a nice way to do it?
- just an idea. This is exceptionally nice.
- Brilliant to just have biologists control hardward from insight
- LIM systems integration
[+] - big companies say: here's all everything you need, if you buy it all from us
- certainly true in HCS
- starting to break...
- has to be so easy and complete
- e.g. audio.
- we're not the problem
- some partners don't have time
- some are afraid of changes, must be convinced.
- need to find 3 components which can be used as an example
- should be no problem.
- is it possible to control their GUI? (VNC, scripts, ...)
- a cool example, "Stage in wrong place"
- unclear where experimenter protocol is started
- browser is perhaps a pretty good starting point
- "start pulsing device 12"
- notices that table is wrong spot
- "move table to"
- "table is in position"
- then starts pulsing
- "pulsing"
- analysis on several images
- browser asks for analysis
- analyzer asks for data
- analyzed
- return data
- is it worth setting up dummy system?
- can you run a 100,000 images?
- like the demo from Chris and Colin of dropping an image in a directory?
- insight / jean-marie
- interdepencey (cyclic): the tough stuff poses questions that we can't answer except holistically.
- Prototyping!
- how to have multiple version which users can see and choose between
- would preferences help?
- Cat: difference from other prototyping, it's so personal that the user can't evaluate without their own data
- have to build working prototype, no lo-fi or hi-fi
- Jason: they want to work with their prototype
- something that can't be handled with our old way of working
- Eclipse as a model, with versioned plugins so users can compare
- HCS provides the deepest hierarchy (more than tagging) - Harry & multi-hierarchies
- finding new insights in data view traversing across hierarchies
- smart folders. Perhaps folders per protocol
- Ilya: Can you switch from one hierarchy to another keeping selection?
- Can always come back to the previous view. "Capture-Concept"
- Google-map/Screen-map/Large-image-map : not hard but a new concept
- Ola / web
- "You have no tasks. Go add 10!"
- Ilya: Can you share with anyone? (re: botnet)
- sharing v. publication: one during your work one is after.
- re: legal aspects with journal
- part of sharing/publishing workflow is creating a new dataset, specifying the visualization
- email contains a url? or url + session key
- publication is wide-open
- Josh:or as screen, so you can do formatting/positioning
- Carlos: having positioning information would be great in clients
- Jean-Marie: is the text wiki? how do other clients digest it?
- A: HTML. But the share is on the application-side (during development time)
- Storing as XHTML. Easiest.
- Ties into OmeroSecurity?
- Yes. "Mounting a share" is the metaphor. Slower. And doesn't show up in my regular view.
- What kind of object in OMERO can be a "Task"?
- Ilya: When did I take an Image?
- When did I process it again? 3rd time, 4th time and what did I do those times?
- Will: I want to pick up the tagging I was doing yesterday
- Jean-Marie: definition of groups of actions
- acquisition, tagging/manual data, analysis
- Date in dataset or project name (experiment is annotated though somewhere)
- Use TimestampAnnotation either globally or attached to the user
- Needs a start/end, repeat, etc.
- Viewer
- Hide-all? For LOTS of images.
- Pager infrastructure? Not yet.
- We need a dataset with 5000 images would help.
- Andrew: sunset clause? from when?
- Ilya: the real power is public access
- HDF
- Position written into next grant
- Talk to M. Dougherty
- Definitely going to move forward
- ImageJ
- Client. Yes
- Server-side? Perhaps discussion at the ImageJ meeting later this year.
- Email list after conversation with Johannes/Wayne
- first integration: use the data that's in the server for display
- second: then can store their ROIs as an attachment
- third: with strong need, we can look at converting to overlay
- fourth: with strong need, we can look at running in the background
- ideas:
- storing macro as an attachment which produced the results
- 2-4 weeks of just getting plugins done.
- fixing beta3
- use OmeroJava? For now, use RMI
- Search
- changing index schedule so that it doesn't lock
- "can you upload a website and then google right away?
- Delete
- once you try to sole something simple like "Image" then
- then what happens in SPW
- which links do you break?
- Ilya: deleting image from a well should be empty
- propagate to the tag or not
- Ola: Cleanup Wizard? "You have some unused items on your Desktop..."
- Curtis: PDF-Dummy to link things (manually updating the ref count)
- Brian: status deleted/failed/
- Jean-Marie: Performance problem checking the flag on all the things
- Chris: still slows down the overall transfer
- Ilya: Just pushing the problem somewhere
- DeleteService
- Ilya: Based on deleting a MEX. But have to decide delete or unlink
- Need a wizard which shows everything that you are possibly deleting
- Curtis: Ways to intelligently flag stuff
- Jean-Marie: Need the cleanup phase
- Chris: Querying for orphans!
- Problem: 100,000 orphans? Now What?
- No structure to say what the PDFs mean, are/were linked to
- "New Delete: here are the consequences."
- "Old Delete: here are your orphans"
- Cat: An image belongs to another user. Having an annotation, makes two owners, effectively.
- Two owners want to delete: gone.
- One owner: gets hidden/trashbin
- Chris: I don't think it's practical because ownership is as simple as viewing an image.
- Chris: Different priorities of annotations? Cat: All are making sense. Will: It's finding something.
- Cat: Copying images when you attach.
- Josh: Service needs to have the semantics "What will this delete?" "What will be orphaned?"
- Jason: Don't want to force a value propsition
- Ilya: Have to assert who the dominant owner is.
- Ilya: Can publish a retraction, but can't get rid of the original
- Will: If I attach something to a dataset, then I trust it won't change
- Jason: conversation idea ... most people would ignore it because of the competitiveness
- Marking something as a "shared" image to put it into a different state (community model)
- Josh: Disagree
- Have to support both models: protecting things and allowing delete
- Accounting systems: audit and nothing gets deleted (Brian)
- Chris: Naive approach because they have heavy pre-filtering
- Ilya: Collaborative work is the decision to delete
- Josh: Way for the event system/messaging to help primary owners to delete.
- Chris: Need to help system to decide which links are overridable
- Josh: then then client can find out what's overridable
- Josh: But will still need the dialog box: "Primary wants to delete, do you agree?"
- Ilya: is there a permission trap, where you can't see something attached?
- Ilya: color code graph view of the things: Deleted/Could be Deleted/Things owned by others/Things blocking your delete !
- Need to know what they are getting into
- May be surprised when they go to another site
- Habituation!
- Todo
- Start implementing static version
- Use rules for possible GUI later
- Need delete/unlink differentiation
- Eventually message for confirmation
- Use Sessions/Events to offer additional deletes
- Permissions.trashContainer
- Implemented Client size
- once you try to sole something simple like "Image" then
- Lunch discussion
- future: protocol, presentations,
- Josh multiple pixels per image
- Ilya: Die!
- No one gets it, who cares?
- example: deconvolve. Dataset pointing to the deconvolved pixels, no way to see other pixels on the same image
- deconvolve should be another image? That's what we do now.
- Default pixels, what is used by default. That's what users think is *the* image
- Image link should have a link to the default pixel.
- Donald: I don't like multiple pixels, it's evil.
- Jason: in presentations, when mentioning multiple pixels, people look as if you said something stupid
- Event log is complicated to use too
- Image concept has 2 responsabilities
- images are just a view on the data
- You have something that's yellow... provenance model, call it what you will
- What are you tagging? Looks the same to users.
- Josh: information should be normalized, in the right place.
- Chris: one image is raw, one is projected, how to make statements about each image?
- how to say: "these 2 pixels are more alike than other 2" - is possible
- tool don't have the functionality to handle multiple related pixels-
- Donald: doesn't satisfy the way we work. We want to say how pixels are derived.
- Brian: It's a workflow problem, Image containers doesn't do a good job for that.
- Users do that anyway, they can add a new set of pixels. Now what?
- Josh: History is the new container
- Jason: must also support the algorithm case, not just UI
- Ilya: the yellow brick road: It changes a lot of fundamentals.
- Chris: Images have only name and description, nothing else.
- Andrew: what happens to info attached to image? ROIs?
- Chris: they should be in the pixels, not image.
- Josh: we lack linkage between dataset and pixels.
- Chris & Donald: Collapse Image and Pixels, removing multiple pixels completely.
- Next Major XML Release? (July) with light sources and xrays
- No. Implement and see what it does what we want.
- Write it up on the ome-xml trac, get suggestions
- From Ilya 3 requirements: (and all tools don't have this functionality)
- 2 Pixels are more related than others (struc)
- Container speaks about all the pixls (struc)
- "image" / "pix" are convoluted over 400 years of microscopy
- blitz libraries
- naming classes/libraries
- what methods are implemented
- coding style
- where's the committe?
- similar to in server
- no underscores, etc.
- come to a conclusion
- convention: load is top to bottom, find is bottom to top
- store it as Client.ice under blitz/resources/omero/Something with doxygen in Javadoc
- OME model adherence / properties
- Possible to come up with groups?
- SPW, for example
- Do we discuss it after the fact?
- handling creatiaon defate as default
- Possible to come up with groups?
- Big Images
- use cases: large plane images 22000x40000
- things are getting slow
- Andrew: a "big" image — does the server treat it differently
- both sides need to adjust
- provide an exception which says "BigImage"
- new programming interface with sliding window
- image pyramids
- calculating multiple resolutions
- pre-rendering scaling
- subsampling. visualization impression is unsatisfactory (algorithm based)
- reasonable limit: .1 microns with slide 3 inches, 750,000 pixels a side
- future-proofing requires 1 Tb / plane
- compression
- does the metadata describe a tiling or a stitching?
- use case: hospital basement full of slides that should all be digitized (surgical biopsy) Aperio.
- one implementation are like geo tiffs. algorithm to do registration
- optics don't let you perfectly align them
- then they are split back apart
- in that case there's less need for metadata to get the overlaps settled.
- retiling is a common implementation
- becomes an issue for bioformats. already partially have a tile api.
- interface.whatAreTheTiles()
- answer: we don't really need metadata changes
- there are both an optimal client tiling and an optional server tiling
- interface.setScaling(double) anything at 1 or above returns the whole image
- handle all of this on the server with exceptions
- Consider: 3D or non-XY tiles/blockings
- aperio
- http://images2.aperio.com/view.apml?listview=0&returnurl=http://www.aperio.com/&returnest=0
- this is life-and-death
Breakout discusions ====================================================================
- Breakout list
- List of needed EventLogs - SESSION, ...
- Search/ontological
- Benchmark OMERO with WND-CHARM
- Mime-type
- parllel IO
- tagging tags or using events
- typing: patching v. fork
- HDF
- Image/Pixels
- OME-Profiles/Adherence/Nullability
- ImageJ/other integration/GLUE
- FullStack for next month
- Demo possibilities: vmware, live-dvd, ...
- production server for weblitz (nginx?)
- naming weblitz
- blitzconnector, gateway
- hierarchy/trash/delete/imported flag/...
- Requirements discussions
- Image/Pixels discussion
- OME-HDF
- ROIs: supporting ImageJ rois, for example
- Relaxing the model
- converting csv/xls (re: Jun20/Will & printing)
- Server scripting (Java)
- standalone viewer
- ImageJ repository
- cluster support
- ImageJ client
- Discussion on documenting what properties are supported (Jun20/Melissa)
- Defining the weighting of all properties (Jun20/Andrew)
- Which scripting interface would users like to use?
- Stemming and Tokenization and Thesauri
- Jason's presentations stored in OMERO (presentation as publishing) ... audio? ... "AskJason.com".
- Look at Josh's cloud diagram and decide on some priorities?
- what's a task/event in the calendar? And how is that modelled? ICalAnnotation? (repalces Timestamp?)
- Monday "Coding Day"
- JvmLink
- Concrete todos
- Document or implement goodeval()
- encrypting password
- examine one story/feature from top to bottom. include as many people as possible to determine process.
- simple workflow things: projections, AVI,
- screen
- sharing/import/export
- search
- Rename OmeroJ->OmeroJava