2014-09-09 Tuesday Team Meeting
Dundee: Kenny, Gus, Will, Petr, Emil, Ola, Blazej, Helen, Dominik, Balaji, Andrew, Graeme, June, Remote: Josh M, Ian, Melissa, Colin, Kelli, Seb, Wilma, Yuriy, Josh B, Curtis, Mark, Douglas
Dundee: Kenny, Gus, Will, Petr, Emil, Ola, Blazej, Helen, Dominik, Balaji, Andrew, Graeme, June, Jean-Marie, Simone, Roger Remote: Josh M, Ian, Melissa, Colin, Kelli, Seb, Wilma, Yuriy, Josh B, Curtis, Mark, Douglas
Notes
Project Status (15 minutes max)
Mainline (5 minutes)
- Still working from proposed scheduled (nitpick)
-
Change based on stability of
develop
schema - Upcoming releases (5.0.5 & 5.1.0-m1)
- Backport requests on the “Priorities List” board
-
Brief discussion about update sites
- Jars are downloaded from d.o.org and then served from ij servers
- Curtis: like the develop plan; users want “stable” or “latest”
Glencoe Update (5 minutes) - 14:44 UK
-
JB still working on ORM layer and updating jars (see PR 3000)
- travis builds are green; but server doesn’t start.
- indexer & pixeldata are complaining that they don’t have a session.
- more eyes welcome.
- Stick & Emil working on cellprofiler
-
JB still working on ORM layer and updating jars (see PR 3000)
Consortium Update (5 minutes) - 14:47
- Ian: thanks for the release? re: GUI - hope not reproducing the wheel?
- Seb S.: nothing in particular
- Mark/Curtis: update by the end of the week; work on scripting parameters
-
Douglas: jquery update! Eyes welcome. Lots of little things are possibly broken.
- Will to look later today
Simone - 'Integrating Hadoop into OMERO, Update 2014-09-09' (20-25 minutes plus 15 mins questions max)
-
IO backend work to have HDFS backend for path.py
- Bigger picture: FS abstraction layer for OMERO
-
Scalable backend for numerical data
- Currently looking at SciDB & Apache Spark (basis for ADAM)
-
Feature calculation with WND-CHRM
- distributed version with pydoop script
- in new repository pydoop-features
- Simon added number crunching part in his fork
- working on bundle which installs everything on the fly on the cluster
-
HDFS In/Out
- Simon has exported numpy arrays via an OMERO script
- array extraction directly from HDFS would require direct Bio-Formats access
- storing back in to OMERO is a further problem
- path.py update PR under review
- fixed Pydoop unicode issues
- work is scattered over a number of repos/branches
- CRS4 developing (incl. a new dev) a mostly pure Python rewriting of Pydoop's backend (performance-critical (de)serialization section will probably stay in C/C++)
- want everything merged back into develop as soon as possible
-
could move to pathlib; only in Python3 mainline though there is a backport
- there is an FS abstraction layer in Hadoop itself.
-
Josh: primarily concerned with the synchronization between Python & Java
- Simone: can some version start expecting URIs? fork for HPC?
-
IO backend work to have HDFS backend for path.py
AOB (5 mins max - technical discussions should be highlighted to relevant people and rescheduled)
-
J-m: workshops (c++ last Wednesday)
- idea to move to Tuesdays; optional in case not of interest
- keep project review beforehand short (~20 minutes)
- upcoming: code gen (Josh while in Dundee)
-
J-m: workshops (c++ last Wednesday)