2013-11-05 Tuesday Team Meeting
Remember: Agenda must be complete (with estimated times) on the day before the meeting. Any additions after that must go at the bottom (AOCB)
Agenda - 2:30pm Start
Dundee: Kenny, Chris, Jean-Marie, Blazej, Andrew, June, Emil, Gus, Petr, Helen, Ola, Simon, Roger, Will, Mark Remote: Colin, Sebastien, Sebastien, Josh, Josh, Mark H, Douglas, Melissa, Carlos, Stick
Accepting minutes from last meeting
- Accepted
Welcome June Matthew (new OME Admin in Dundee) and Joshua Ballanco (Glencoe)
- June: worked at university for 10 years
- Joshua: lives in Turkey. PhD in evolutionary biology, then dev..
Testing update/project status
Glencoe update - 14:37 UK
- Nothing beyond last week
Consortium update - 14:37 UK
- Nada
Upcoming meetings/conferences/outreach visits
-
GRE symposium - November 22nd
- Present various OMERO tools on poster. U-track & FLIMfit
- 2 weeks, biobank presentation on 19th Tuesday meeting.
-
GRE symposium - November 22nd
JCB DataViewer presentation - Chris Allan (15 mins plus discussion) - 14:41 UK
- manuscript: 201303145
- challenges of working on top of open source
- Thanks to Glencoe team - big now. Most tasks distributed
- OMERO 4.4.x, Custom Importer, BF fixes back to open source
- Blitz gateway API is critical
- JCB application is full standalone plugin
- webpublic infrastructure used for public
- Screens use OMERO.tables and search.
- No Insight / webclient usage.
- JCB data maps onto standard OMERO data types & annotations
- Lots of OMERO tech used to generate SPW pages, charts etc
- Numbers for latest screen are big - 1.3 TB. Many metadata objects. E.g. 482416 Channels
-
Process: 35 hours to import --no_thumbnail is critical
- populate_metadata - 20 mins
- Annotate Screen
- Adjust custom index bridge - reindex 3.5 hours
- Set image metadata. Channel names, rendering settings etc. 150ms per image = 11 hours!
-
Lessons:
- All components are deployed separately.
- Limited number of workers: 8
- Don’t thumbnail on import
- Offline indexing (complete DB replica)
- Large annotation counts suck. JOIN across 100,000s of objects
- Avoid omero.gateway for performance critical sections - Use custom queries, IQuery.projection()
- Template optimisation. “with” keyword vital in some areas. DRY!
- More advanced Django worker and nginx setups. zlib compression, more workers & longer lived. Gunicorn
- Separate OMERO.web deploy highly beneficial - 2-3 times a week.
- webgateway_cache needs a lot of work (in progress)
- CLI critical for large datasets, hql, delete, chgrp etc
- Downstream of dev_4_4. Merge points need to be clearly defined. Parallel builds
- OMERO.web plugin config - needs discussion
- Need to consume JARs via dependency resolution
-
Points to discuss
- HCS tools used cross consortium - large datasets - testing
- Tension between “gateways” and performance
- Large datasets can break standard clients
- Future of rendering settings and rendering engine API
- Future of S-P-W-WS-I model. Need to load lots of data to get plate dimensions etc
- Backporting - web plugins, worker config, caching, lib/repository and target/ layout, Ivy modifications
Questions
- Ola - list of performance changes backported? Chris - need to review
- JM - Re-use Channels across plate? Chris - could do that now (data reduction)
- Still have LogicalChannel? (removed from OME model a while back)
- Andrew - How to fix rendering settings? Chris - fewer rendering settings - 1 per image / plate. Andrew - rdef needs to be in OME model, for export etc. Chris - JCB don’t want 1 rdef per user. Will - can’t show custom rdef on public data
- Will - lots of effort to provide data, but users can’t really use the data (just browsing thumbs etc) Chris - Graph is most useful - browsing plates/thumbs is more of a gimmick
- Roger - Load plate of thumbnails as one image? Chris - Looked at this but image is big, slow (don’t see thumbs loading) Client-side plotting etc browser support improving - but still 25% IE8!
Any other business (<5 mins)
-
Build email discussion (Chris/Josh)
- omero-team currently set for a number of jobs. Getting too many e-mails on failing builds.
- Need more flexible e-mail lists for builds.
- Propose new alias "ci-status". Who would join?
-
Build email discussion (Chris/Josh)
Youtube for demo videos (of not released functionality) - Discuss next time!