Personal tools
  • We're Hiring!

You are here: Home Community Minutes Conference Calls 2013 2013-11-05 Tuesday Team Meeting

2013-11-05 Tuesday Team Meeting

Remember: Agenda must be complete (with estimated times) on the day before the meeting. Any additions after that must go at the bottom (AOCB)

Agenda - 2:30pm Start

Dundee: Kenny, Chris, Jean-Marie, Blazej, Andrew, June, Emil, Gus, Petr, Helen, Ola, Simon, Roger, Will, Mark Remote: Colin, Sebastien, Sebastien, Josh, Josh, Mark H, Douglas, Melissa, Carlos, Stick

  1. Accepting minutes from last meeting

    • Accepted
  2. Welcome June Matthew (new OME Admin in Dundee) and Joshua Ballanco (Glencoe)

    • June: worked at university for 10 years
    • Joshua: lives in Turkey. PhD in evolutionary biology, then dev..
  3. Testing update/project status

    1. QA tool see report. We are going to use Squash
    2. FS non FS testing see Plan
    3. Squash not ready for testing just yet - needs time to get set up. Gradual migration.
    4. Take 4.4. DB and upgrade to 5.0 each day.
    5. Start small - increase data amount to cover as much as possible.
  4. Glencoe update - 14:37 UK

    • Nothing beyond last week
  5. Consortium update - 14:37 UK

    • Nada
  6. Upcoming meetings/conferences/outreach visits

    • GRE symposium - November 22nd
      • Present various OMERO tools on poster. U-track & FLIMfit
    • 2 weeks, biobank presentation on 19th Tuesday meeting.
  7. JCB DataViewer presentation - Chris Allan (15 mins plus discussion) - 14:41 UK

    • manuscript: 201303145
    • challenges of working on top of open source
    • Thanks to Glencoe team - big now. Most tasks distributed
    • OMERO 4.4.x, Custom Importer, BF fixes back to open source
    • Blitz gateway API is critical
    • JCB application is full standalone plugin
    • webpublic infrastructure used for public
    • Screens use OMERO.tables and search.
    • No Insight / webclient usage.
    • JCB data maps onto standard OMERO data types & annotations
    • Lots of OMERO tech used to generate SPW pages, charts etc
    • Numbers for latest screen are big - 1.3 TB. Many metadata objects. E.g. 482416 Channels
    • Process: 35 hours to import --no_thumbnail is critical
      • populate_metadata - 20 mins
      • Annotate Screen
      • Adjust custom index bridge - reindex 3.5 hours
      • Set image metadata. Channel names, rendering settings etc. 150ms per image = 11 hours!
    • Lessons:
      • All components are deployed separately.
      • Limited number of workers: 8
      • Don’t thumbnail on import
      • Offline indexing (complete DB replica)
      • Large annotation counts suck. JOIN across 100,000s of objects
      • Avoid omero.gateway for performance critical sections - Use custom queries, IQuery.projection()
      • Template optimisation. “with” keyword vital in some areas. DRY!
      • More advanced Django worker and nginx setups. zlib compression, more workers & longer lived. Gunicorn
      • Separate OMERO.web deploy highly beneficial - 2-3 times a week.
      • webgateway_cache needs a lot of work (in progress)
      • CLI critical for large datasets, hql, delete, chgrp etc
      • Downstream of dev_4_4. Merge points need to be clearly defined. Parallel builds
      • OMERO.web plugin config - needs discussion
      • Need to consume JARs via dependency resolution
    • Points to discuss
      • HCS tools used cross consortium - large datasets - testing
      • Tension between “gateways” and performance
      • Large datasets can break standard clients
      • Future of rendering settings and rendering engine API
      • Future of S-P-W-WS-I model. Need to load lots of data to get plate dimensions etc
      • Backporting - web plugins, worker config, caching, lib/repository and target/ layout, Ivy modifications
  8. Questions

    • Ola - list of performance changes backported? Chris - need to review
    • JM - Re-use Channels across plate? Chris - could do that now (data reduction)
    • Still have LogicalChannel? (removed from OME model a while back)
    • Andrew - How to fix rendering settings? Chris - fewer rendering settings - 1 per image / plate. Andrew - rdef needs to be in OME model, for export etc. Chris - JCB don’t want 1 rdef per user. Will - can’t show custom rdef on public data
    • Will - lots of effort to provide data, but users can’t really use the data (just browsing thumbs etc) Chris - Graph is most useful - browsing plates/thumbs is more of a gimmick
    • Roger - Load plate of thumbnails as one image? Chris - Looked at this but image is big, slow (don’t see thumbs loading) Client-side plotting etc browser support improving - but still 25% IE8!
  9. Any other business (<5 mins)

    • Build email discussion (Chris/Josh)
      • omero-team currently set for a number of jobs. Getting too many e-mails on failing builds.
      • Need more flexible e-mail lists for builds.
      • Propose new alias "ci-status". Who would join?
  10. Youtube for demo videos (of not released functionality) - Discuss next time!

Document Actions