Examples

This section shows some usage examples on how to create a project, add some scripts with dependencies, and run a parent script that creates a pipeline of processes.

Make sure you have everything setup before moving down, see Installation.

Example Scripts

There are some pre-built example scripts in the examples folder under the root of the workflow-manager repository - /path/to/workflow-manager/examples

• create_and_run_project.py

  Create and run a example project.

  import os
  import pymongo
  import shutil
  
  import workflow_manager as wm
  
  if __name__ == '__main__':
      project_name = 'test_project'
      root = './tmp/test_project'
  
      myclient = pymongo.MongoClient('localhost', 27017)
      if project_name in myclient.list_database_names():
          print('DROP %s database' % project_name)
          myclient.drop_database(project_name)
      if os.path.exists(root):
          print('DELETE %s directory' % root)
          shutil.rmtree(root)
      os.makedirs(root)
  
      P = wm.create_project(project_name, root_dir=root)
      P.import_script('scripts/pretend_import.py')
      P.import_script('scripts/pretend_segment.py')
      P.import_script('scripts/pretend_fit.py')
      P.import_script('scripts/pretend_mechanics1.py')
      P.import_script('scripts/pretend_mechanics2.py')
      P.import_script('scripts/pretend_send.py')
  
      P = wm.Project(project_name)
      script = P.script('pretend_import')
      script_input_arguments = {'path': 'data/pretend_data.txt', 'send_dir': os.path.join(P.root_dir, 'results')}
      # script_input_arguments = {'path': 'data/example_dicom.dcm'}
      # script_input_arguments = {'path': 'data'}
      script.run(script_input_arguments)
  
      wm.project.start_process_monitor(project_name, minutes_alive=1, sleep_time=3, total_cores=8)
  

• create_project.py

  Create a WM project in /path/to/workflow-manager/examples/tmp/test_project/ and import some mock scripts into the project.

  import os
  import pymongo
  import shutil
  
  import workflow_manager as wm
  
  if __name__ == '__main__':
      project_name = 'test_project'
      project_root = './tmp/test_project'
  
      myclient = pymongo.MongoClient('localhost', 27017)
      if project_name in myclient.list_database_names():
          print('DROP %s database' % project_name)
          myclient.drop_database(project_name)
      if os.path.exists(project_root):
          print('DELETE %s directory' % project_root)
          shutil.rmtree(project_root)
      os.makedirs(project_root)
  
      P = wm.create_project(project_name, root_dir=project_root)
  
      P.import_script('scripts/pretend_import.py')
      P.import_script('scripts/pretend_segment.py')
      P.import_script('scripts/pretend_fit.py')
      P.import_script('scripts/pretend_mechanics1.py')
      P.import_script('scripts/pretend_mechanics2.py')
      P.import_script('scripts/pretend_send.py')
  

• run_project.py

  Put the scripts of the example project into a queue and run the project. This will also start a process monitor for the project. If it finds pending jobs it runs the job and monitors it for success or failure and updates the database.

  import os
  
  import workflow_manager as wm
  
  if __name__ == '__main__':
      project_name= "test_project"
  
      P = wm.Project(project_name)
      script = P.script('pretend_import')
      script_input_arguments = {'path': 'data/pretend_data.txt', 'send_dir': os.path.join(P.root_dir, 'results')}
      script.run(script_input_arguments)
  
      wm.project.start_process_monitor(project_name, minutes_alive=99, sleep_time=3, total_cores=8)
  

• monitor_project.py

  Monitor the example project status by listing the imported scripts and their status - success or failure.

  import workflow_manager as wm
  
  if __name__ == '__main__':
      project_name = 'test_project'
      P = wm.Project(project_name)
  
      print("Scripts:")
      P.list_scripts()
      print("Processes and their status:")
      P.list_processes()
  

• delete_project.py

  Delete the example project

  import os
  import pymongo
  import shutil
  
  if __name__ == '__main__':
      project_name = 'test_project'
      root = './tmp/test_project'
  
      myclient = pymongo.MongoClient('localhost', 27017)
      if project_name in myclient.list_database_names():
          print('DROP %s database' % project_name)
          myclient.drop_database(project_name)
      if os.path.exists(root):
          print('DELETE %s directory' % root)
          shutil.rmtree(root)
  

Other usage examples

• Create a WM project

  import workflow_manager
  P = workflow_manager.create_project('my_project', './path/to/project_datastore')
  

• Add scripts to the project

  P = workflow_manager.create_project(project_name, root_dir=root)
  P.import_script('/path/to/scripts_1.py')
  P.import_script('/path/to/scripts_2.py.py')
  P.import_script('/path/to/scripts_3.py.py')
  

  You can can set script dependency by

  script = P.script('child_script')
  script.add_dependency('parent_script')
  

  or by specifying depends_on = ['parent_script'] at the beginning of the child script

  Both will run the child script after the completion of the parent script.

• Create processes from imported scripts and put them into a project queue. The code below will generate processes for the pretend_import script and its child scripts.

  script = P.script('pretend_import')
  script_input_arguments = {'path': 'data/pretend_data.txt'}
  script.run(script_input_arguments)
  

• Start process monitor

  Start a process monitor to actually run and monitor the processes in the project queue.

  workflow_manager.project.start_process_monitor(project_name, minutes_alive=3, sleep_time=3, total_cores=8)
  

• To look at the processes of a given project, type

  P = workflow_manager.Project('project_name')
  P.list_processes()
  

• To look at a given processes log, type

  P = workflow_manager.Project('project_name')
  P.process(process_id).log()