Creating workflows that execute external code bases that are under development

Abstract

There is an increasing interest in the use of scientific workflows as a way to automate data management and model execution without requiring deep computing technical knowledge. Scientific workflows allow users to re-use previously developed code in multiple languages while providing repeatability and flexibility. Additional complexity is introduced when scientific workflows use code that is still in development, especially when the code development is not linked to workflow development. In this paper, within the context of a particular workflow engine, we look at existing tools for code re-use and techniques to manage the complexity of working with rapidly changing code. Hydrologist's Workbench (HWB) is a suite of tools, activities and recommendations built to support Microsoft's Project Trident, a scientific workflow engine. HWB contains several tools that assist in turning existing code into Trident Activities - atomic, composable, executable modules - via a process known as code “wrapping”. The existing tools within HWB are designed to wrap code that is reasonably stable and unchanging and therefore these tools are insufficient for code that is subject to changes over time. We have investigated techniques to minimise the effort required to turn code under active development into Activities and workflows. The techniques include: an agile methodology for workflow and code co-development; treating the code as a dataset itself; using an agreed interface; dynamically generating then executing scripted code; a simplified, template-based Activity generation tool and manual transliteration. Issues around testing, versioning, integration and communication are also discussed. Show more