Profile Configuration

ServerURL

The server URL is the single endpoint to which all requests are sent. It must be a valid URL, reachable from your namespace, and include at least a scheme and a server address; it may also include a port number and a path prefix. It must not include a trailing slash. The server URL is prefixed to an operation path from the API definition to create the target of a request. For example, given this URL definition from the configuration:

ServerURL: https://api-host:8000/apis

And this operation path from the API definition:

paths:
  /movies:
    post:

The auto-tester will send POST requests to this target URL:

https://api-host:8000/apis/movies

TimeToRun

The maximum length of time to run, in minutes: the auto-tester stops sending requests once this time period has elapsed. The overall running time will be longer because it includes additional time for setup and teardown.

For example, to set a maximum running time of ten minutes:

TimeToRun: 10

MaxRequests

The maximum number of requests to send: the auto-tester stops issuing requests once this number has been reached. The final total of requests may be greater if additional requests are needed for setup and teardown.

Set this to zero to disable the request limit, and then the auto-tester will issue as many requests as fit into the allotted time period.

For example, to set a maximum of two thousand requests:

MaxRequests: 2000

MaxErrors

The maximum number of error responses to allow: the auto-tester stops issuing requests once this number has been reached.

An error in this context is not an API error indicated by a 4xx (or even, sometimes, a 5xx) response, which is to be expected if the profile exercises all execution paths through the implementation. An error here is one where the response to a request is inconsistent with the API definition, which is a defect. Common cases include:

• the status code is not documented
• the payload does not conform to the schema
• any 500 error

There should be no such errors in a mature implementation so the limit is typically set low, so that the profile can terminate early if they occur. You can set the limit to zero and disable this stopping condition, which might be appropriate for a development environment where you know that some operations have defects (or you can disable those operations from ever being called as described under Operations).

For example, to set the error limit to three:

MaxErrors: 3

ParallelRequests

If this is true then the auto-tester spawns threads to issue multiple requests concurrently, otherwise it issues just one request at a time. The concurrent option increases throughput and can be helpful in exposing synchronization issues in the API.

The number of concurrent threads is fixed and cannot be changed.

For example, to disable concurrent requests:

ParallelRequests: false

ErrorInjection

If this is true then the auto-tester issues a certain percentage of malformed requests to further exercise the validation logic of the API, otherwise it only issues requests that are well-formed according to the API definition.

The percentage of requests is specified as ErrorPercent.

For example, to enable malformed requests:

ErrorInjection: true

ErrorPercent

The percentage of malformed requests, when ErrorInjection is true.

For example, to include 2% of malformed requests:

ErrorPercent: 2.0

Resources

The weights assigned to specific resource types. The value is an object, where the keys are resource names, and the values are non-negative integer weights. Resource names must be declared in the API extensions.

For example, to increase the test focus on Customer operations, and exclude Order operations entirely:

Resources:
  Customer: 2
  Order: 0

ResourceDefault

The default weight for any resource type not mentioned explicitly under Resources. The weight can be any non-negative integer value, but there are two common use cases:

• if the weight is 1 then all resources are selected with equal frequency, except where specified otherwise under Resources
• if the weight is 0 then all resources are disabled, except for those specified otherwise under Resources

For example, to disable all resources by default:

ResourceDefault: 0

In this case there must be at least one non-zero weight under Resources.

Instances

Upper and lower bounds on the number of resource instances that should exist at any time. The value is an object, where the keys are resource names, and the values specify lower and upper bounds for those resource types. Resource names must be declared in the API extensions, and the bounds must be non-negative integers that together define a non-empty range. The upper bound may also be unlimited in which case the number of instances depends upon the balance of create and delete requests issued for that type.

The auto-tester keeps track of the instances it creates for each resource type, in order to provide valid references and avoid too many 404 responses, and makes best efforts to keep the number of instances within any specified bounds.

For example, to place an upper bound on the number of Customer instances that should exist at any time:

Instances:
  Customer:
    maximum: 25

InstanceDefault

The default instance bounds for any resource type not mentioned explicitly under Instances.

For example, to make the default unlimited:

InstanceDefault:
  minimum: 1
  maximum: unlimited

Operations

The weights assigned to specific operations. The value is an object whose schema follows the pattern established by the Paths object in OpenAPI, but the values assigned to each operation are non-negative integer weights.

For example, to increase test focus on the put_customer operation, and avoid patch_customer completely:

Operations:
  /customers/{customer_id}:
    put: 10
    patch: 0

The special key _ can be used to match all methods defined on a path that are not called out explicitly. For example, to disable all but the put_customer operation:

Operations:
  /customers/{customer_id}:
    put: 10
    _: 0

Remember that the weights are interpreted only across operations within the same resource type, and not across all operations in the API, so the overall distribution is determined by both the weight on the resource and the weight on the operation.

OperationDefault

The default weight for any operations not mentioned explicitly under Operations. The value is an object where the keys are method names and the values are the default weights for those methods where they occur under any path. The special key _ matches all method names that are not listed separately.

For example, to disable all operations by default:

OperationDefault:
  _: 0

In this case there must be at least one non-zero weight under Operations.

Schemas

Data generation rules for named schemas found under definitions (OpenAPI 2.0) or components/schemas (OpenAPI 3.0).

For example, given the schema definition:

Movie:
  type: object
  properties:
    title:
      type: string
    release_date:
      type: string
      format: date
    language:
      description: Two-letter ISO 639-1 language code
      type: string
      pattern: '[a-z]{2}'

The default generation rule for the language property, derived from this schema, will produce random string values that match the specified pattern (two lower-case alphabetic characters) but many of these will not be valid language codes, potentially giving rise to an excessive number of 400 responses.

To override the default generation rule and specify instead some particular language codes sufficient for testing:

Datatypes:
  schemas:
    Movie:
      properties:
        language:
          enum:
            - en
            - fr
            - de

To inject a small percentage of invalid values:

Datatypes:
  schemas:
    Movie:
      properties:
        language:
          choice:
            - enum:
                - en
                - fr
                - de
              weight: 98
            - const: xx
              weight: 2

Operations

Data generation rules for operation parameters and payloads described by inline (or unnamed) schemas and types. The value is an object whose schema follows the pattern established by the Paths object in OpenAPI.

Note that the auto-tester only supports path and query parameters at present, and other parameter types are ignored.

For example, given these (abbreviated) operation definitions:

paths:
  /movies:
    get:
      parameters:
        - name: limit
          in: query
          description: Sets an upper bound on the number of movies returned
          schema:
            type: integer
    post:
      requestBody:
        content:
          application/json:
            schema:
              type: object
              allOf:
                - $ref: '#/definitions/Movie'
                - properties:
                    keywords:
                      type: array
                      items:
                        type: string

To set values for the limit query parameter, and the keywords property of the payload to post:

Datatypes:
  operations:
    /movies:
      get:
        parameters:
          - name: limit
            in: query
            data:
              # pick a random value between 0 and 500 (inclusive)
              minimum: 0
              maximum: 500
      post:
        requestBody:
          content:
            application/json:
              data:
                properties:
                  keywords:
                    # pick up to 3 items from the following list
                    items:
                      enum:
                        - avant-garde
                        - dystopia
                        - epic
                        - postmodern
                        - remake
                        - shootout
                    minItems: 0
                    maxItems: 3