Personal tools
  • We're Hiring!

You are here: Home → Community → Minutes → Conference Calls → 2013 → 2013-03-12 Tuesday Meeting
September 2022 »
September
MoTuWeThFrSaSu
1234
567891011
12131415161718
19202122232425
2627282930
 

2013-03-12 Tuesday Meeting

Dundee: Kenny, Elwood, Will, Gus, Helen, Joyce, Mark, Andrew, Colin, Blazej, Roger, Jean-Marie, Petr, Chris, Simon, Jason Remote: Jesse, Jos, Andreas, Carlos, Stick, Curtis, Ian, Ivan, Josh, Kevin, Liza, Mark Hiner, Melissa, Sebastien

  1. Accepting minutes from last meeting

    • Done.

  2. Paris 2013 - Speakers & Workshop Brainstorming Ideas (Joyce 10 mins) 14:35 UK

    • Joyce: Dates confirmed 24-26 June, confirmed auditorium (space upstairs)
    • Joyce: Outreach for suggestions & feedback on workshops, speakers, format, contact via email with ideas etc.
    • Joyce: Suggestions etc by end of week please.
      • Kevin: Stowers maybe, their usage deployment etc.
      • Kevin: 10/15min ‘how we use OME’
    • Joyce: Potential audience, who we can target/contact directly apart from forums, our lists etc.

  3. Jos Koetsier presentation - Rapid Omero - A message based job submission system (45 mins)

    • Started with Donald a few years ago.
    • Use case: Microscopes and techniques are getting powerful, in OMERO we used to be able to perform cpu intensive tasks to dedicated hardware. Goal of Rapid OMERO.
    • Build a job submission system within OMERO itself, became quite difficult. Taking on a new approach, 1st attempt is image pyramid generation on dedicated hardware.
    • Decouple job submission from OMERO, move out this part to a separate component. Should be able to run high-cpu job on local machine.
    • User can log out and job does not disapear, state can be preserved and resumed.
    • Robust: Don’t want to lose jobs and get results back in time.
    • Omero submits job to a message queue instead of directly to the dedicated server, and then the high-perf system can take the message from the queue and start processing it.
    • AQMP, multiple languages supported, java/python etc..
    • Wire protocol: Format of messages is standardised
    • AQMP: Conumer can write to multiple queues, consumer can read from 1->many queues.
    • Pub-Sub. Producer sends message to exchange, exchange directs messages to queues and consumers read from their queue.
    • Robustness issues, component failures, problems can occur and you would need ot have thought about these failings.
    • Message queue failure, RabbitMQ saves each message in to a separate database, starting back up the queue machine will restore the state from the database.
    • Consumer removes message from queue, while processing the consumer fails. Message is lost, AMQP defines a process for reporting (ack after processing). Queue server detects failure as no ack.
    • Producer dies, not a lot happens.
    • For the system we define, we want to give reply back to producer what happened to the message, AMQP has a protocol for this. Producer.reply_to is an excusilve queue, only producer can listen to it. Exculsive queue is transitory. A series of replies can be sent.
    • Different queues for job submission and file transfer to transfer the file to the compute node.
    • Advantage of doing this is that the queueing system can be placed elsewhere or same machine. Consumers can be elsewhere too.
    • Message of AMQP is not defined, content is what ever you want. JSON encoded message.
    • Simple integration code to send messages and process replies.
    • Integrate in to OMERO client
    • Live Demo!
    • Questions?
      • Simon: configure ation required for a differnt cluster?
        • YAML file, you’d have to know which cluster and system you have, rewrite the YAML file.
        • Jos: Used saga project. Various system types supported. PBS, SGE, possibly now Condor
      • Jason: File transfer - access the data through the API?
        • Using OMERO API directly would tightly couple the system. Just file transfered.
        • Andrew: no way to not transfer to the cluster?
      • Malcolm: removing things from the queue?
        • Jos: more difficult. Assumed that someone else is to handle it.
        • Chris/Jason: possible issues of congestion.
        • Jos: management interface from rabbitmq, but low-level. Might want to hide the names of queues, etc.
      • Chris: ideally messages should be in the queue for milliseconds. It’s not a scheduler.
      • Malcolm: possibility of looking at job properties and send work to proper queue
      • Jason: cluster deployment, pains
        • This requires rabbitmq, etc. Is the burden reasonable?
        • Jos: could imagine rabbitmq deployed with OMERO. just sending things to cluster.
        • Josh: Cluster should be able to be ignorant of everything OMERO, transfering bio-formats, bfconvert etc.
        • Jos: transfer the file & jars as well as scripts
      • Roger: define dependencies between queues?
        • Jos: “Workflows” are difficult.
        • Similar to the issue with file transfers before the job.
        • Can’t upload 3 things to a queue and assume they’re run in order.
        • Decided to handle ordering in OMERO itself.
        • Malcolm: could submit KNIME jobs.
      • Andrew: requesting further information from job? (cF. KNIME)
        • Malcolm: cluster admins are hesitant to allow communication during processing.
      • Simon: take code and get started tomorrow?
        • Jos: Need to work on deployment. (and upload more stuff)
        • Another day or two.

  4. Bio-Formats redirects (Kevin et al.) - 15:38 UK

    • Jason & Kevin to discuss Monday night.

  5. Glencoe Update - 15:39 UK

    • New developer. Niko joined starting Monday. 1st thing to look at message queues for search.

  6. Any other business (<5 mins) - 15:41 UK

    • Permissions testing next week - see Testing setup
      • Petr: Testing next week using dedicated server
      • Performed on new server established by Kenny/Blazej
      • Data prepared by the script. Precious!
      • Testing each day in standup notes will include everyone
      • Jean-Marie: RWRW is turned on. We don’t know the impact since we haven’t tested with it on before.
    • Bioformats Spec/Xsd-Fu PR on Develop - Merge spec into bioformats discussion
      • Andrew: Current state is that it’s not perfect, but can clean up with further PRs.
      • Josh: Has been merged in for the last couple of days, not broken anything.
      • Josh: Standup tomorrow last chance for veto otherwise merged in.
    • Openmicroscopy Spec Cleanup PR on Dev_4_4
      • Andrew: Standup tomorrow last chance for veto otherwise merged in.
    • QA / requesting files
      • Josh: File qa spreadsheet
      • Jean-Marie: when people asked for a file on forum, update the spreadsheet to link to qa when it comes through.
        • Close once received and updated.
        • Possible to patch QA2 to track this. (Also Java info, etc.)
        • Jason: QA2 as summer project.
        • Andrew: student? Chris: previous projects have been painful
        • Simon: Dundee Tech Meet (26th Mar) - seminar about what we do to engage students? (hard to hear)
        • Jason: issue of vacation.

  7. Done: 15:56 UK

Document Actions