Exscript.queue module

The heart of Exscript.

class Exscript.queue.Queue(domain='', verbose=1, mode='threading', max_threads=1, host_driver=None, exc_cb=None, stdout=<open file '<stdout>', mode 'w'>, stderr=<open file '<stderr>', mode 'w'>)

Bases: future.types.newobject.newobject

Manages hosts/tasks, accounts, connections, and threads.

__init__(domain='', verbose=1, mode='threading', max_threads=1, host_driver=None, exc_cb=None, stdout=<open file '<stdout>', mode 'w'>, stderr=<open file '<stderr>', mode 'w'>)

Constructor. All arguments should be passed as keyword arguments. Depending on the verbosity level, the following types of output are written to stdout/stderr (or to whatever else is passed in the stdout/stderr arguments):

• S = status bar
• L = live conversation
• D = debug messages
• E = errors
• ! = errors with tracebacks
• F = fatal errors with tracebacks

The output types are mapped depending on the verbosity as follows:

• verbose = -1: stdout = None, stderr = F
• verbose = 0: stdout = None, stderr = EF
• verbose = 1, max_threads = 1: stdout = L, stderr = EF
• verbose = 1, max_threads = n: stdout = S, stderr = EF
• verbose >= 2, max_threads = 1: stdout = DL, stderr = !F
• verbose >= 2, max_threads = n: stdout = DS, stderr = !F

Parameters:

• domain (str) – The default domain of the contacted hosts.
• verbose (int) – The verbosity level.
• mode (str) – ‘multiprocessing’ or ‘threading’
• max_threads (int) – The maximum number of concurrent threads.
• host_driver (str) – driver name like “ios” for manual override
• exc_cb (func(jobname, exc_info)) – callback function to call on exceptions
• stdout (file) – The output channel, defaults to sys.stdout.
• stderr (file) – The error channel, defaults to sys.stderr.

add_account(account)

Adds the given account to the default account pool that Exscript uses to log into all hosts that have no specific Account attached.

Parameters:account (Account) – The account that is added.

add_account_pool(pool, match=None)

Adds a new account pool. If the given match argument is None, the pool the default pool. Otherwise, the match argument is a callback function that is invoked to decide whether or not the given pool should be used for a host.

When Exscript logs into a host, the account is chosen in the following order:

# Exscript checks whether an account was attached to the Host object using Host.set_account()), and uses that.

# If the Host has no account attached, Exscript walks through all pools that were passed to Queue.add_account_pool(). For each pool, it passes the Host to the function in the given match argument. If the return value is True, the account pool is used to acquire an account. (Accounts within each pool are taken in a round-robin fashion.)

# If no matching account pool is found, an account is taken from the default account pool.

# Finally, if all that fails and the default account pool contains no accounts, an error is raised.

Example usage:

def do_nothing(conn):
    conn.autoinit()

def use_this_pool(host):
    return host.get_name().startswith('foo')

default_pool = AccountPool()
default_pool.add_account(Account('default-user', 'password'))

other_pool = AccountPool()
other_pool.add_account(Account('user', 'password'))

queue = Queue()
queue.add_account_pool(default_pool)
queue.add_account_pool(other_pool, use_this_pool)

host = Host('localhost')
queue.run(host, do_nothing)

In the example code, the host has no account attached. As a result, the queue checks whether use_this_pool() returns True. Because the hostname does not start with ‘foo’, the function returns False, and Exscript takes the ‘default-user’ account from the default pool.

Parameters:

• pool (AccountPool) – The account pool that is added.
• match (callable) – A callback to check if the pool should be used.

destroy(force=False)

Like shutdown(), but also removes all accounts, hosts, etc., and does not restart the queue. In other words, the queue can no longer be used after calling this method.

Parameters:force (bool) – Whether to wait until all jobs were processed.

enqueue(function, name=None, attempts=1)

Places the given function in the queue and calls it as soon as a thread is available. To pass additional arguments to the callback, use Python’s functools.partial().

Parameters:

• function (function) – The function to execute.
• name (string) – A name for the task.
• attempts (int) – The number of attempts on failure.

Return type:

object

Returns:

An object representing the task.

force_run(hosts, function, attempts=1)

Like priority_run(), but starts the task immediately even if that max_threads is exceeded.

Parameters:

• hosts (string|list(string)|Host|list(Host)) – A hostname or Host object, or a list of them.
• function (function) – The function to execute.
• attempts (int) – The number of attempts on failure.

Return type:

object

Returns:

An object representing the task.

get_max_threads()

Returns the maximum number of concurrent threads.

Return type:int Returns:The maximum number of connections.

get_progress()

Returns the progress in percent.

Return type:float Returns:The progress in percent.

is_completed()

Returns True if the task is completed, False otherwise. In other words, this methods returns True if the queue is empty.

Return type:bool Returns:Whether all tasks are completed.

join()

Waits until all jobs are completed.

priority_run(hosts, function, attempts=1)

Like run(), but adds the task to the front of the queue.

Parameters:

• hosts (string|list(string)|Host|list(Host)) – A hostname or Host object, or a list of them.
• function (function) – The function to execute.
• attempts (int) – The number of attempts on failure.

Return type:

object

Returns:

An object representing the task.

priority_run_or_raise(hosts, function, attempts=1)

Like priority_run(), but if a host is already in the queue, the existing host is moved to the top of the queue instead of enqueuing the new one.

Parameters:

• hosts (string|list(string)|Host|list(Host)) – A hostname or Host object, or a list of them.
• function (function) – The function to execute.
• attempts (int) – The number of attempts on failure.

Return type:

object

Returns:

A task object, or None if all hosts were duplicates.

reset()

Remove all accounts, hosts, etc.

run(hosts, function, attempts=1)

Add the given function to a queue, and call it once for each host according to the threading options. Use decorators.bind() if you also want to pass additional arguments to the callback function.

Returns an object that represents the queued task, and that may be passed to is_completed() to check the status.

Parameters:

• hosts (string|list(string)|Host|list(Host)) – A hostname or Host object, or a list of them.
• function (function) – The function to execute.
• attempts (int) – The number of attempts on failure.

Return type:

object

Returns:

An object representing the task.

run_or_ignore(hosts, function, attempts=1)

Like run(), but only appends hosts that are not already in the queue.

Parameters:

• hosts (string|list(string)|Host|list(Host)) – A hostname or Host object, or a list of them.
• function (function) – The function to execute.
• attempts (int) – The number of attempts on failure.

Return type:

object

Returns:

A task object, or None if all hosts were duplicates.

set_max_threads(n_connections)

Sets the maximum number of concurrent connections.

Parameters:n_connections (int) – The maximum number of connections.

shutdown(force=False)

Stop executing any further jobs. If the force argument is True, the function does not wait until any queued jobs are completed but stops immediately.

After emptying the queue it is restarted, so you may still call run() after using this method.

Parameters:force (bool) – Whether to wait until all jobs were processed.