eqsql.worker_pool module

class eqsql.worker_pool.LocalPool(name, proc, cfg_file)

Bases: object

Encapsulates a locally running EQSQL worker pool. LocalPool instances should only be created using start_local_pool().

Parameters:

  • name (str) – the name of the worker pool

  • proc (Popen) – the POpen instance used to launch the pool.

  • cfg_file – the configuration file used to launch this pool

cancel(timeout=10)

Cancels this worker pool.

Parameters:

timeout – the attempt to cancel will timeout after this duration.

status()

Gets the status (active, completed, or canceled) of this LocalPool.

Returns:

The current JobStatus of this LocalPool with JobState attribute set appropriately - JobState.CANCELED, JobState.ACTIVE, or JobState.COMPLETE.

Return type:

JobStatus

Examples

>>> pool.status().state
JobState.ACTIVE
class eqsql.worker_pool.ScheduledPool(name, job_id, scheduler, gcx, cfg_file)

Bases: object

cancel(timeout=60)

Cancels this worker pool.

Parameters:

timeout – the attempt to cancel will timeout after this duration.

status(timeout=60)

Gets the status (active, completed, or canceled) of this LocalPool.

Returns:

The current JobStatus of this LocalPool with JobState attribute set appropriately - JobState.CANCELED, JobState.ACTIVE, or JobState.COMPLETE.

Examples

>>> pool.status().state
JobState.ACTIVE
eqsql.worker_pool.cfg_file_to_dict(cfg_file)

Reads a bash cfg file into a dictionary

Parameters:

cfg_file (str)

Return type:

Dict

eqsql.worker_pool.cfg_tofile(cfg_params)
Parameters:

cfg_params (Dict)

Return type:

str

eqsql.worker_pool.format_pool_exp_id(exp_id, name)
Parameters:

  • exp_id (str)

  • name (str)

eqsql.worker_pool.start_local_pool(name, launch_script, exp_id, cfg_params)

Starts a local worker pool and returns the LocalPool instance encapsulating that worker pool.

Parameters:

  • name (str) – the name of the worker pool

  • launch_script (str | bytes | PathLike) – the path to an executable script used to start the worker pool

  • exp_id (str) – the experiment id for the experiment that is running the pool

  • cfg_params (Dict) – the parameters used to start the pool

Returns:

A LocalPool instance.

Return type:

LocalPool

eqsql.worker_pool.start_scheduled_pool(name, launch_script, exp_id, cfg_params, scheduler, gcx)

Starts a worker pool on a scheduled resource, i.e., a resource that queues jobs using some sort of scheduler, e.g., slurm.

Parameters:

  • name (str) – the name of the worker pool

  • launch_script (str | bytes | PathLike) – the path to an executable script used to start the worker pool

  • exp_id (str) – the experiment id for the experiment that is running the pool

  • cfg_params (Dict) – the parameters used to start the pool

  • scheduler (str) – scheduled resource schedule type - slurm, etc.

  • gcx (Executor) – a globus compute Executor instance. If this is None, then the assumption is that the call to this function is running on the resource where the pool is to be launched. If this is not None, then the executor will be used to launch the pool remotely.

Returns:

An instance of a ScheduledPool object.