2013-03-12 Tuesday Meeting
Dundee: Kenny, Elwood, Will, Gus, Helen, Joyce, Mark, Andrew, Colin, Blazej, Roger, Jean-Marie, Petr, Chris, Simon, Jason Remote: Jesse, Jos, Andreas, Carlos, Stick, Curtis, Ian, Ivan, Josh, Kevin, Liza, Mark Hiner, Melissa, Sebastien
Accepting minutes from last meeting
- Done.
Paris 2013 - Speakers & Workshop Brainstorming Ideas (Joyce 10 mins) 14:35 UK
- Joyce: Dates confirmed 24-26 June, confirmed auditorium (space upstairs)
- Joyce: Outreach for suggestions & feedback on workshops, speakers, format, contact via email with ideas etc.
-
Joyce: Suggestions etc by end of week please.
- Kevin: Stowers maybe, their usage deployment etc.
- Kevin: 10/15min ‘how we use OME’
- Joyce: Potential audience, who we can target/contact directly apart from forums, our lists etc.
Jos Koetsier presentation - Rapid Omero - A message based job submission system (45 mins)
- Started with Donald a few years ago.
- Use case: Microscopes and techniques are getting powerful, in OMERO we used to be able to perform cpu intensive tasks to dedicated hardware. Goal of Rapid OMERO.
- Build a job submission system within OMERO itself, became quite difficult. Taking on a new approach, 1st attempt is image pyramid generation on dedicated hardware.
- Decouple job submission from OMERO, move out this part to a separate component. Should be able to run high-cpu job on local machine.
- User can log out and job does not disapear, state can be preserved and resumed.
- Robust: Don’t want to lose jobs and get results back in time.
- Omero submits job to a message queue instead of directly to the dedicated server, and then the high-perf system can take the message from the queue and start processing it.
- AQMP, multiple languages supported, java/python etc..
- Wire protocol: Format of messages is standardised
- AQMP: Conumer can write to multiple queues, consumer can read from 1->many queues.
- Pub-Sub. Producer sends message to exchange, exchange directs messages to queues and consumers read from their queue.
- Robustness issues, component failures, problems can occur and you would need ot have thought about these failings.
- Message queue failure, RabbitMQ saves each message in to a separate database, starting back up the queue machine will restore the state from the database.
- Consumer removes message from queue, while processing the consumer fails. Message is lost, AMQP defines a process for reporting (ack after processing). Queue server detects failure as no ack.
- Producer dies, not a lot happens.
- For the system we define, we want to give reply back to producer what happened to the message, AMQP has a protocol for this. Producer.reply_to is an excusilve queue, only producer can listen to it. Exculsive queue is transitory. A series of replies can be sent.
- Different queues for job submission and file transfer to transfer the file to the compute node.
- Advantage of doing this is that the queueing system can be placed elsewhere or same machine. Consumers can be elsewhere too.
- Message of AMQP is not defined, content is what ever you want. JSON encoded message.
- Simple integration code to send messages and process replies.
- Integrate in to OMERO client
- Live Demo!
-
Questions?
-
Simon: configure ation required for a differnt cluster?
- YAML file, you’d have to know which cluster and system you have, rewrite the YAML file.
- Jos: Used saga project. Various system types supported. PBS, SGE, possibly now Condor
-
Jason: File transfer - access the data through the API?
- Using OMERO API directly would tightly couple the system. Just file transfered.
- Andrew: no way to not transfer to the cluster?
-
Malcolm: removing things from the queue?
- Jos: more difficult. Assumed that someone else is to handle it.
- Chris/Jason: possible issues of congestion.
- Jos: management interface from rabbitmq, but low-level. Might want to hide the names of queues, etc.
- Chris: ideally messages should be in the queue for milliseconds. It’s not a scheduler.
- Malcolm: possibility of looking at job properties and send work to proper queue
-
Jason: cluster deployment, pains
- This requires rabbitmq, etc. Is the burden reasonable?
- Jos: could imagine rabbitmq deployed with OMERO. just sending things to cluster.
- Josh: Cluster should be able to be ignorant of everything OMERO, transfering bio-formats, bfconvert etc.
- Jos: transfer the file & jars as well as scripts
-
Roger: define dependencies between queues?
- Jos: “Workflows” are difficult.
- Similar to the issue with file transfers before the job.
- Can’t upload 3 things to a queue and assume they’re run in order.
- Decided to handle ordering in OMERO itself.
- Malcolm: could submit KNIME jobs.
-
Andrew: requesting further information from job? (cF. KNIME)
- Malcolm: cluster admins are hesitant to allow communication during processing.
-
Simon: take code and get started tomorrow?
- Jos: Need to work on deployment. (and upload more stuff)
- Another day or two.
-
Simon: configure ation required for a differnt cluster?
Bio-Formats redirects (Kevin et al.) - 15:38 UK
- Jason & Kevin to discuss Monday night.
Glencoe Update - 15:39 UK
- New developer. Niko joined starting Monday. 1st thing to look at message queues for search.
Any other business (<5 mins) - 15:41 UK
-
Permissions testing next week - see Testing setup
- Petr: Testing next week using dedicated server
- Performed on new server established by Kenny/Blazej
- Data prepared by the script. Precious!
- Testing each day in standup notes will include everyone
- Jean-Marie: RWRW is turned on. We don’t know the impact since we haven’t tested with it on before.
-
Bioformats Spec/Xsd-Fu PR on Develop - Merge spec into bioformats discussion
- Andrew: Current state is that it’s not perfect, but can clean up with further PRs.
- Josh: Has been merged in for the last couple of days, not broken anything.
- Josh: Standup tomorrow last chance for veto otherwise merged in.
-
Openmicroscopy Spec Cleanup PR on Dev_4_4
- Andrew: Standup tomorrow last chance for veto otherwise merged in.
-
QA / requesting files
- Josh: File qa spreadsheet
-
Jean-Marie: when people asked for a file on forum, update the spreadsheet to link to qa when it comes through.
- Close once received and updated.
- Possible to patch QA2 to track this. (Also Java info, etc.)
- Jason: QA2 as summer project.
- Andrew: student? Chris: previous projects have been painful
- Simon: Dundee Tech Meet (26th Mar) - seminar about what we do to engage students? (hard to hear)
- Jason: issue of vacation.
-
Permissions testing next week - see Testing setup
Done: 15:56 UK