2016-04-05 Tuesday Team Meeting
Dundee: Petr, Kenny, Will, Dominik, Simon, Roger, June, Mark, Balaji, Remote: Josh, Andrea, Gus, Sebastien, Simone, Ian, Colin, Melissa, David, Emil, Kelli, Jason, Chris, Andreas,
Agenda - 2:30pm Start
Accepting minutes from last meeting
- Accepted
Project Timelines (2-3 minutes each)
Spaces
-
Mainline (J-m)
- BF - 5.1.9 PR freeze yesterday
- Testing PRs today
- All PRs should be merged by tomorrow evening
- RC by Thursday or Friday if no hiccups
- No OMERO update this week
-
Model (Sebastien)
- Micromanager bug report for OME-TIFF reader
- Needed new set of OME samples - PR to open against formats
- Yesterday Regions meeting 5.3.0
- Cleaning up client work
- Looking at scaling issues E.g. 1000s of ROIs per image
- Handling large number of ROIs in server and client UI
- Jason: Demo?
- Seb: yes, next milestone completes planned model changes. Need to set date.
-
Metadata (Josh)
- Progress on charts WNDcharm
- Simon working on mini IDR - re-import a plate from each screen
- Ola working on Robot test with Redis backend
- Decide on use of Redis for EBI
- IDR hardware still having periodic outages
-
Mainline (J-m)
Other releases/upgrades:
Figure
FLIMfit/OPT (Ian) a. FLIMfit 4.11 ready for release following BF 5.1.9 b. Ongoing problems with LaVision .msr file format. - Getting lots of these now, mostly from 1 site in Germany - Workaround is to generate OME-TIFFs, but these can be old schema or invalid - Trying to get users to contact OME team - FLIMfit also supports some formats not supported by BF - Sebastien: Suggest LaVision to use recent OME schema with Modulo Annotation to store FLIM data - Ian: current solution is working but not sure how reliable. c. In discussion with LaVision about OME-TIFFS from the same system (attempting to move from e-mail). d. NB Several FLIM file-formats now supported in FLIMfit but not in Bio-Formats!
ImageJ
Learning
Sysadmin (Kenny)
- Nightshade had hardware issues last week.
- Brought back online after safety checks (postgres etc)
- Checking DB hashes with files
- Thanks for help with SQL queries etc.
- Planning blog post about process
- No problems found so far, still checking thumbnails etc.
- Need to discuss / quotes for update of hardwear
- Josh: new trello cards - how to prevent repeat of these problems
- Balaji: releasing any useful scripts would also be nice
OME Files C++ update
- Roger: 1st release later today
- Few teething troubles since splitting components
- Plan a 0.1.1 release with few fixes
- Josh: OME files Java timeline is outstanding
- Discuss as part of 5.3.0
- Jason: Congrats to Roger & David on release
Glencoe Update (Chris) - Discussions on omero-marshal usage wrt web API
OME Meeting Update (Jason)
- Board on meeting is partially filled out
- Need more volunteers to help AV etc for lightning sessions
- Working on dinner arrangements for day 1
- Workshops for day 2 needs discussion
- Can use existing template with updates, E.g. OME files.
- June: 29 signed up so far 8-10 externals
- Jason: need to get programme out to boost interest
'ome-smuggler' presentation by Andrea Falconi (Montpellier)
- Working on this last few months
- Java web-based work queue - also hooked up to run in Insight
- Use case: want to avoid upload of files AFTER acquisition - reduced microscope availability and user billed for data transfer
- Idea: run import as background process - unsupervised, needs to be reliable and recover from failure.
- Nice if it could be used as back-pressure mechanism to reduce load on OMERO. Regular import can fail with large numbers of files.
- Other maintenance tasks - running scripts etc.
- POST ome/import {email: foo, file: my/img}
- Added to Java based import queue - not imported immediately
- Returns url /ome/import/a1b2
- GET /health to get state of queue
- Then can get update of import GET /ome/import/a1b2
- Returns “running 10%”, “success 100%” etc.
- Tasks are messages on abstract async channel (think CSP) implemented using a message queue
- All messaging is async
- Several backend configs possible, e.g. routing to multiple OMERO servers, load balancing etc
- Scalability: queues can be split across multiple machines.
- Resilience: Persistent queues can withstand a server crash.
- Failures are notified to user and sysadmin
- Queuing work can be used as mechanism to reduce server load when needed
- Large imports / deletes can be split into smaller batches and queued
- Use Spring, Undertow (web server), HornetQ
- Github repo - currently being moved to official MRI repo
- Status: alpha. Code stable and tested, pilot deployed on April 2016
- 12k lines of code, equal amount of tests.
- Chris: “easy question:”: how do you prevent users importing as other users, handling passwords etc?
- Andrea: We don’t store username / password anywhere. Client passes along valid session key. Session kept alive by pinging (as in import library).
- Chris: what if user logs out / destroys session?
- Andrea: we create a new session. Need to speak to Josh to improve understanding of sessions
- Chris: handling of file path differences between client and server
- Andrea: assume client and http server ‘agent’ have access to same file system. Will install agent on each workstation.
- Jason: seems you have well-defined use-case / setup / steps for users to follow etc.
- Andrea: yes - limitation of access to same file-system
- Andrea: use messaging tech to integrate different pieces of software, ImageJ, ICY etc instead of using plugins.
- Also could be used for federated repositories
- Jason: thanks for presentation and beautiful slides
- Jason: release timeline?
- Andreas: pilot deployment. Roll-out further after upgrading to OMERO 5.2 end of April
- Need to train users - steps to follow, need more feedback, e.g. webpage status.
- Jason: deploying on commercial systems?
- Andrea: yes, most are Windows machines, some Linux.
- Can run background process with low priority to avoid breaking acquisition etc.