Personal tools
  • We're Hiring!

You are here: Home Community Minutes Conference Calls 2010 2010-04-06 Tuesday Meeting

2010-04-06 Tuesday Meeting

In Attendance: Brian, Donald, Will, Jean-Marie, Chris, Scott, Ola, Carlos, Melissa, Curtis

Agenda

('Official' agenda is absent this week due to vacations)

  1. Matters Arising (<10 mins)
    • Points from last week
    • Points from Thursday
  2. Bio-Formats Talk (Melissa/Chris)

Meeting Notes

  • Permissions

    • Behind on coding due to change in RO, RW, RL logic
    • Should be doing a QA at the end of this week
  • FLIM

    • Donald has created a couple of tickets based on interaction discussions
    • Trying to make it more user friendly
    • Keywords on ROIs, etc.
    • Problem: sorted in temporary manner
      • But when do we do the measurement tool rewriting
      • For next milestone we will need to commit to fixing it.
      • More than measurement tool, logic is also useful for generating figure
    • Continuing with adding FLIM on current measurement tool infrastructure
      • FLIM needs rest of this iteration at least
      • tabling alternate storage
  • Filtering model

    • Decide in next iteration whether or not to do it
    • Chris: so far no modifications made to core model
  • BioFormats

    • "cleanup" branch in OMERO trunk for 10 days
    • NioFileHandle
      • One of biggest changes
      • reason: get rid of filehandle caching mechanism
      • new strategies for dealing with file handle issues
      • BufferedIO : previously all seek driven (seek & read on each call)
      • BufferedIO now has a good set of tests against it
      • Possibility to crash VM (large LSM files)
        • Chris is currently working to fix this
      • Stream-to-Stream IO is now easier
    • BF now uses SLF4J & log4J and is now highly configurable
      • Also integrated with importer
    • Services/Dependency Injection
      • Not currently using an existing framework. May in the future
      • Tried to remove as much user of ReflectedUniverse as possible
      • Exception handling for missing dependences reported properly now
      • Examples at https://skyking.microscopy.wisc.edu/trac/java/wiki/BioFormatsService
    • Features
      • Levels
        • metadata-validator is now testing levels
        • more flexible API for specifying what's filled
        • ALL and MINIMUM at the moment (usable by FS?)
        • Currently can't do MINIMUM metadata import then ALL in the same instance
      • Concurrency
        • Put a lot of effort into the whole infrastructure
        • Currently reviewing readers
      • OME Data Model
        • Not using instances of the model
        • A service sits in front of all the OME-XML jars/dom/etc.
        • Will be moving to lighter-weight framework for those objects
        • Should we start using those objects?
        • @Curtis: would like to discuss that more later
      • XML code generation
        • goal for 4.2: have bioformats support the latest 2 schemas
    • New formats
      • BD Pathway (HCS), a new Gel, SVI HDF (tricky), ...
      • ~40 readers (code and samples) (15 done so far)
      • slide formats are post 4.2
      • two new domains: scanning probe (SPM) & scanning electron (SEM).
        • not a lot of metadata but possibly overlap with EM
        • @Melissa and Will will discuss
        • Jason: does anyone seem to know what metadata required? * Melissa doing direct port of existing code (not verified against real metadata) * Who should we write to? The ImageJ list? * Will: Asking the guy who wrote ImageSXM * Melissa: could also ask on microscopy list
    • Format writers API
      • saving to smaller tiles (scans with 10G planes etc.)
      • exporting to multiple files planned
        • Josh: Exporting out to zip collection would be nice
        • @Melissa estimate work effort for discussion next week
      • Curtis: @could lastInSeries booleans go away
    • XsdFu
      • Still have some templates to do but mostly the same as before
      • planning on adding new types (PercentFraction, positiveInteger, nonNegativeInteger,dateTime,anyURI)
        • Andrew is normalizing this list so some may still be depreciated
      • Josh: should allow for StructuredAnnotations export? Yes.
      • Only the latest data model will be available to BF.
      • Readers could start using these internal model objects
      • Allows us to be able to store state in memory without parsing files for performance
      • Curtis: excited
        • Agree that Core metadata is superfluous
        • Uncertain about "more concrete use" and getting rid of the abstractions
        • How will we handle importing/reading older schema versions? * Chris: always convert to the latest version with stylesheet.
      • Jean-Marie: not fully thread-safe?
        • Chris: thread-safe in that each reader is thread-safe
        • But can't call methods on a reader concurrently
        • Need to have one reader per thread (can't share one reader)
        • Nothing procludes making them synchronized
      • Other levels: ACQUIRED (from scope) and ADDED (annotations) metadata
        • Helps with FS and export
        • CustomAttributes can show up as a block of XML
      • Validation
        • Jean-Marie: we have tests and the metadata-validator
        • ...but Will's work on manual validating was valuable
        • ...do we generate a template
        • Chris: metadata-validator doesn't version the metadata * "I lost 10% of the metadata" * With code-gen, we'll be able to do that. * Can also turn on plane checksumming * Then can produce the OME-XML and store it on disk * But someone has to go in an compare * Jean-Marie: is there anything smarter we can do?
        • Curtis: What validator is this using? * Chris: the importer-based one which includes the BF tests plus some extra ones * Curtis: could this be a part of OME-XML.jar? * Chris: with code-gen would like to do this in XML * Then possibly (currently uses OMERO importer infrastructure)
    • Discussion
      • Jason: "Compliant" everything about an image
        • How hard would it be to create that?
        • Chris: Melissa has done a good job of compartmentalizing metadata
        • Should be in a good place for adding them, but we don't want 6
        • Jason: asking about one specific one
        • Melissa: yes, we just need to agree on what COMPLIANT means
        • Jason: Would be good to add rules that are in our paper.
        • Jean-Marie: "ALL" means something different for EM
        • Jason: May need to do something like "EM_COMPLIANT", etc.
        • Chris: cF. HCS, "MINIMAL" doesn't do anything for HCS
      • API
        • Jason: have we notified the lists?
        • Yes, last time was beginning of March about trunk destabilization
        • Chris: After discussion with Curtis, trying to be more careful to not change API
        • OME-XML changes will destabilize again largely
      • FS
        • Key: reimporting metadata
        • Indexing binary plane to find the offsets
        • Chris: Should now be able to hook everything up except for short reader initializations
        • But minimal set support should help now.
        • cF. initialize opera plate in ~1-2 secs.
      • Metadata update
        • Will: what do we do if we realize we've been importing/updating incorrectly?
        • We don't know for any attribute what version of bioformats was used.
        • @Perhaps in global metadata we could add bioformats version
        • @Or possibly a bioformats revision annotation
        • Chris: would also like to be able to say authoritatively which attributes in which version
      • Multiple-readers
        • Curtis: Would have to initialize multiple readers
        • TIFFs require linked lists. Pixel-based init can get messy.
        • Chris: will be adding performance tests for setId() to M-V to conretely say
      • Curtis - several things
        • with tighter coupling, can't write old formats
        • keeping up with schema releases (new jars with each XSD)
        • "...with tighter coupling wouldn't have more than 1 way of representing..." * Curtis thinks we still need OME-XML jar * Chris: Currently all loosly coupled, may need to review that. * Jason: might be a good idea to ask the community * Curtis: eventually a C++ model? Yes. Would be useful for LOCI.
        • Custom Attributes will now be translated into an XML annotation and added * Will be preserved but won't be creatable
    • https://skyking.microscopy.wisc.edu/trac/java/wiki/BioFormatsService

Action Items

  • Josh/Chris: Make the settings changes for permissions ASAP
  • Melissa/Will: Discuss EM/SPM/SEM metadata
  • Melissa: Estimate work effort for 4.2 next week
  • Chris/Melissa/Curtis: discuss the use of OME Model objects
  • Chris/Melissa: add bioformats version to imported properties
Document Actions