2010-04-06 Tuesday Meeting
In Attendance: Brian, Donald, Will, Jean-Marie, Chris, Scott, Ola, Carlos, Melissa, Curtis
Agenda
('Official' agenda is absent this week due to vacations)
-
Matters Arising (<10 mins)
- Points from last week
- Points from Thursday
- Bio-Formats Talk (Melissa/Chris)
Meeting Notes
Permissions
- Behind on coding due to change in RO, RW, RL logic
- Should be doing a QA at the end of this week
FLIM
- Donald has created a couple of tickets based on interaction discussions
- Trying to make it more user friendly
- Keywords on ROIs, etc.
-
Problem: sorted in temporary manner
- But when do we do the measurement tool rewriting
- For next milestone we will need to commit to fixing it.
- More than measurement tool, logic is also useful for generating figure
-
Continuing with adding FLIM on current measurement tool infrastructure
- FLIM needs rest of this iteration at least
- tabling alternate storage
Filtering model
- Decide in next iteration whether or not to do it
- Chris: so far no modifications made to core model
BioFormats
- "cleanup" branch in OMERO trunk for 10 days
-
NioFileHandle
- One of biggest changes
- reason: get rid of filehandle caching mechanism
- new strategies for dealing with file handle issues
- BufferedIO : previously all seek driven (seek & read on each call)
- BufferedIO now has a good set of tests against it
-
Possibility to crash VM (large LSM files)
- Chris is currently working to fix this
- Stream-to-Stream IO is now easier
-
BF now uses SLF4J & log4J and is now highly configurable
- Also integrated with importer
-
Services/Dependency Injection
- Not currently using an existing framework. May in the future
- Tried to remove as much user of ReflectedUniverse as possible
- Exception handling for missing dependences reported properly now
- Examples at https://skyking.microscopy.wisc.edu/trac/java/wiki/BioFormatsService
-
Features
-
Levels
- metadata-validator is now testing levels
- more flexible API for specifying what's filled
- ALL and MINIMUM at the moment (usable by FS?)
- Currently can't do MINIMUM metadata import then ALL in the same instance
-
Concurrency
- Put a lot of effort into the whole infrastructure
- Currently reviewing readers
-
OME Data Model
- Not using instances of the model
- A service sits in front of all the OME-XML jars/dom/etc.
- Will be moving to lighter-weight framework for those objects
- Should we start using those objects?
- @Curtis: would like to discuss that more later
-
XML code generation
- goal for 4.2: have bioformats support the latest 2 schemas
-
Levels
-
New formats
- BD Pathway (HCS), a new Gel, SVI HDF (tricky), ...
- ~40 readers (code and samples) (15 done so far)
- slide formats are post 4.2
-
two new domains: scanning probe (SPM) & scanning electron (SEM).
- not a lot of metadata but possibly overlap with EM
- @Melissa and Will will discuss
- Jason: does anyone seem to know what metadata required? * Melissa doing direct port of existing code (not verified against real metadata) * Who should we write to? The ImageJ list? * Will: Asking the guy who wrote ImageSXM * Melissa: could also ask on microscopy list
-
Format writers API
- saving to smaller tiles (scans with 10G planes etc.)
-
exporting to multiple files planned
- Josh: Exporting out to zip collection would be nice
- @Melissa estimate work effort for discussion next week
- Curtis: @could lastInSeries booleans go away
-
XsdFu
- Still have some templates to do but mostly the same as before
-
planning on adding new types (PercentFraction, positiveInteger, nonNegativeInteger,dateTime,anyURI)
- Andrew is normalizing this list so some may still be depreciated
- Josh: should allow for StructuredAnnotations export? Yes.
- Only the latest data model will be available to BF.
- Readers could start using these internal model objects
- Allows us to be able to store state in memory without parsing files for performance
-
Curtis: excited
- Agree that Core metadata is superfluous
- Uncertain about "more concrete use" and getting rid of the abstractions
- How will we handle importing/reading older schema versions? * Chris: always convert to the latest version with stylesheet.
-
Jean-Marie: not fully thread-safe?
- Chris: thread-safe in that each reader is thread-safe
- But can't call methods on a reader concurrently
- Need to have one reader per thread (can't share one reader)
- Nothing procludes making them synchronized
-
Other levels: ACQUIRED (from scope) and ADDED (annotations) metadata
- Helps with FS and export
- CustomAttributes can show up as a block of XML
-
Validation
- Jean-Marie: we have tests and the metadata-validator
- ...but Will's work on manual validating was valuable
- ...do we generate a template
- Chris: metadata-validator doesn't version the metadata * "I lost 10% of the metadata" * With code-gen, we'll be able to do that. * Can also turn on plane checksumming * Then can produce the OME-XML and store it on disk * But someone has to go in an compare * Jean-Marie: is there anything smarter we can do?
- Curtis: What validator is this using? * Chris: the importer-based one which includes the BF tests plus some extra ones * Curtis: could this be a part of OME-XML.jar? * Chris: with code-gen would like to do this in XML * Then possibly (currently uses OMERO importer infrastructure)
-
Discussion
-
Jason: "Compliant" everything about an image
- How hard would it be to create that?
- Chris: Melissa has done a good job of compartmentalizing metadata
- Should be in a good place for adding them, but we don't want 6
- Jason: asking about one specific one
- Melissa: yes, we just need to agree on what COMPLIANT means
- Jason: Would be good to add rules that are in our paper.
- Jean-Marie: "ALL" means something different for EM
- Jason: May need to do something like "EM_COMPLIANT", etc.
- Chris: cF. HCS, "MINIMAL" doesn't do anything for HCS
-
API
- Jason: have we notified the lists?
- Yes, last time was beginning of March about trunk destabilization
- Chris: After discussion with Curtis, trying to be more careful to not change API
- OME-XML changes will destabilize again largely
-
FS
- Key: reimporting metadata
- Indexing binary plane to find the offsets
- Chris: Should now be able to hook everything up except for short reader initializations
- But minimal set support should help now.
- cF. initialize opera plate in ~1-2 secs.
-
Metadata update
- Will: what do we do if we realize we've been importing/updating incorrectly?
- We don't know for any attribute what version of bioformats was used.
- @Perhaps in global metadata we could add bioformats version
- @Or possibly a bioformats revision annotation
- Chris: would also like to be able to say authoritatively which attributes in which version
-
Multiple-readers
- Curtis: Would have to initialize multiple readers
- TIFFs require linked lists. Pixel-based init can get messy.
- Chris: will be adding performance tests for setId() to M-V to conretely say
-
Curtis - several things
- with tighter coupling, can't write old formats
- keeping up with schema releases (new jars with each XSD)
- "...with tighter coupling wouldn't have more than 1 way of representing..." * Curtis thinks we still need OME-XML jar * Chris: Currently all loosly coupled, may need to review that. * Jason: might be a good idea to ask the community * Curtis: eventually a C++ model? Yes. Would be useful for LOCI.
- Custom Attributes will now be translated into an XML annotation and added * Will be preserved but won't be creatable
-
Jason: "Compliant" everything about an image
- https://skyking.microscopy.wisc.edu/trac/java/wiki/BioFormatsService
Action Items
- Josh/Chris: Make the settings changes for permissions ASAP
- Melissa/Will: Discuss EM/SPM/SEM metadata
- Melissa: Estimate work effort for 4.2 next week
- Chris/Melissa/Curtis: discuss the use of OME Model objects
- Chris/Melissa: add bioformats version to imported properties