Personal tools
  • We're Hiring!

You are here: Home Community Minutes Conference Calls 2016 2016-04-05 Tuesday Team Meeting

2016-04-05 Tuesday Team Meeting

Dundee: Petr, Kenny, Will, Dominik, Simon, Roger, June, Mark, Balaji, Remote: Josh, Andrea, Gus, Sebastien, Simone, Ian, Colin, Melissa, David, Emil, Kelli, Jason, Chris, Andreas,

Agenda - 2:30pm Start

  1. Accepting minutes from last meeting

    • Accepted
  2. Project Timelines (2-3 minutes each)

    1. Spaces

      1. Mainline (J-m)
        • BF - 5.1.9 PR freeze yesterday
        • Testing PRs today
        • All PRs should be merged by tomorrow evening
        • RC by Thursday or Friday if no hiccups
        • No OMERO update this week
      2. Model (Sebastien)
        • Micromanager bug report for OME-TIFF reader
        • Needed new set of OME samples - PR to open against formats
        • Yesterday Regions meeting 5.3.0
        • Cleaning up client work
        • Looking at scaling issues E.g. 1000s of ROIs per image
        • Handling large number of ROIs in server and client UI
        • Jason: Demo?
        • Seb: yes, next milestone completes planned model changes. Need to set date.
      3. Metadata (Josh)
        • Progress on charts WNDcharm
        • Simon working on mini IDR - re-import a plate from each screen
        • Ola working on Robot test with Redis backend
        • Decide on use of Redis for EBI
        • IDR hardware still having periodic outages
    2. Other releases/upgrades:

      1. Figure

      2. FLIMfit/OPT (Ian) a. FLIMfit 4.11 ready for release following BF 5.1.9 b. Ongoing problems with LaVision .msr file format. - Getting lots of these now, mostly from 1 site in Germany - Workaround is to generate OME-TIFFs, but these can be old schema or invalid - Trying to get users to contact OME team - FLIMfit also supports some formats not supported by BF - Sebastien: Suggest LaVision to use recent OME schema with Modulo Annotation to store FLIM data - Ian: current solution is working but not sure how reliable. c. In discussion with LaVision about OME-TIFFS from the same system (attempting to move from e-mail). d. NB Several FLIM file-formats now supported in FLIMfit but not in Bio-Formats!

      3. ImageJ

      4. Learning

      5. Sysadmin (Kenny)

        • Nightshade had hardware issues last week.
        • Brought back online after safety checks (postgres etc)
        • Checking DB hashes with files
        • Thanks for help with SQL queries etc.
        • Planning blog post about process
        • No problems found so far, still checking thumbnails etc.
        • Need to discuss / quotes for update of hardwear
        • Josh: new trello cards - how to prevent repeat of these problems
        • Balaji: releasing any useful scripts would also be nice
      6. OME Files C++ update

        • Roger: 1st release later today
        • Few teething troubles since splitting components
        • Plan a 0.1.1 release with few fixes
        • Josh: OME files Java timeline is outstanding
        • Discuss as part of 5.3.0
        • Jason: Congrats to Roger & David on release
    3. Glencoe Update (Chris) - Discussions on omero-marshal usage wrt web API

    4. OME Meeting Update (Jason)

      • Board on meeting is partially filled out
      • Need more volunteers to help AV etc for lightning sessions
      • Working on dinner arrangements for day 1
      • Workshops for day 2 needs discussion
      • Can use existing template with updates, E.g. OME files.
      • June: 29 signed up so far 8-10 externals
      • Jason: need to get programme out to boost interest
  3. 'ome-smuggler' presentation by Andrea Falconi (Montpellier)

    • Working on this last few months
    • Java web-based work queue - also hooked up to run in Insight
    • Use case: want to avoid upload of files AFTER acquisition - reduced microscope availability and user billed for data transfer
    • Idea: run import as background process - unsupervised, needs to be reliable and recover from failure.
    • Nice if it could be used as back-pressure mechanism to reduce load on OMERO. Regular import can fail with large numbers of files.
    • Other maintenance tasks - running scripts etc.
    • POST ome/import {email: foo, file: my/img}
    • Added to Java based import queue - not imported immediately
    • Returns url /ome/import/a1b2
    • GET /health to get state of queue
    • Then can get update of import GET /ome/import/a1b2
    • Returns “running 10%”, “success 100%” etc.
    • Tasks are messages on abstract async channel (think CSP) implemented using a message queue
    • All messaging is async
    • Several backend configs possible, e.g. routing to multiple OMERO servers, load balancing etc
    • Scalability: queues can be split across multiple machines.
    • Resilience: Persistent queues can withstand a server crash.
    • Failures are notified to user and sysadmin
    • Queuing work can be used as mechanism to reduce server load when needed
    • Large imports / deletes can be split into smaller batches and queued
    • Use Spring, Undertow (web server), HornetQ
    • Github repo - currently being moved to official MRI repo
    • Status: alpha. Code stable and tested, pilot deployed on April 2016
    • 12k lines of code, equal amount of tests.
    • Chris: “easy question:”: how do you prevent users importing as other users, handling passwords etc?
    • Andrea: We don’t store username / password anywhere. Client passes along valid session key. Session kept alive by pinging (as in import library).
    • Chris: what if user logs out / destroys session?
    • Andrea: we create a new session. Need to speak to Josh to improve understanding of sessions
    • Chris: handling of file path differences between client and server
    • Andrea: assume client and http server ‘agent’ have access to same file system. Will install agent on each workstation.
    • Jason: seems you have well-defined use-case / setup / steps for users to follow etc.
    • Andrea: yes - limitation of access to same file-system
    • Andrea: use messaging tech to integrate different pieces of software, ImageJ, ICY etc instead of using plugins.
    • Also could be used for federated repositories
    • Jason: thanks for presentation and beautiful slides
    • Jason: release timeline?
    • Andreas: pilot deployment. Roll-out further after upgrading to OMERO 5.2 end of April
    • Need to train users - steps to follow, need more feedback, e.g. webpage status.
    • Jason: deploying on commercial systems?
    • Andrea: yes, most are Windows machines, some Linux.
    • Can run background process with low priority to avoid breaking acquisition etc.
Document Actions