Defining AWS Observer jobs

Using the AWS Observer, you can define jobs that read services data from the Amazon Web Services (AWS) through AWS SDK and then generate a topology. It is installed as part of the core installation procedure.

Before you begin

Important: The AWS Observer supports the cloud/SaaS AWS version 1.11.

Ensure you have the AWS details to hand, such as AWS Region, Access Key ID and Access Secret Key.

Remember: Swagger documentation for the observer is available at the following default location: https://<your host>/1.0/aws-observer/swagger

About this task

The AWS Observer supports EC2 service and currently processes EC2 resources in the following order. When the account in use has insufficient privileges to access a resource, the observer skips the resource(s) and proceed to the next one.
  • Instance
  • AvailabilityZones
  • NetworkInterfaces
  • Volumes
  • SecurityGroups
  • VPCs
  • Subnets
  • RouteTables
Important: As a minimum, the account in use must have the DescribeInstances policy, or the observer job fails. Please see the following for more information: Identity and access management for Amazon EC2
aws_observer_common.sh
The configuration file you use to customize AWS Observer settings.
The parameters defined here are then used by the aws_observer_load_start.sh to trigger the AWS Observer jobs.
Tip: Alternatively, you can set the appropriate environment variables. If an environment variable is set, it takes precedence over the configuration file settings.
You define and start the following job. You must edit the parameters in the configuration file before running this job.
Full Topology Upload job
A transient (one-off) job that loads all requested topology data.
This job is started by the aws_observer_load_start.sh script.
Required: In order for the AWS Observer to access the Amazon Web Services (AWS) account automatically, the accessKey, secretKey and region parameters are required.
  • The access key and the secret access key are not the standard user name and password, but are special tokens that allow the services to communicate with the AWS account by making secure REST or Query protocol requests to the AWS service API.
  • The region is the geographical location, for example US East (Ohio), Asia Pacific (Hong Kong), or EU (London).
    Note: The Full Topology Upload job also supports multi-region full loads, as well as properties filtering, but only via Swagger and UI.
    • If you wish to discover more than one region, you will need to comma-separate each region, for example ["eu-east-1","ap-south-1"].
    • If you wish to exclude more than one property, you will need to separate each property by comma, for example "kernelId,state-code,keyName".
    Multi-region full load and properties filtering are not supported in the aws_observer_load_start.sh script.

Procedure

To find your Access Key and Secret Access Key:

  1. Log in to your AWS Management Console.
  2. Click on your user name at the top right of the page.
  3. Click on the Security Credentials link from the drop-down menu.
  4. Find the Access Credentials section, and copy the latest Access Key ID.
  5. Click on the Show link in the same row, and copy the Secret Access Key.

To find the region

  1. Check the region at the following location:
    https://docs.aws.amazon.com/general/latest/gr/rande.html

To edit the parameters in the configuration file

  1. Open the aws_observer_common.sh configuration file and edit the following Load parameters:
    Table 1. AWS Observer parameters
    Parameter Action Details
    Unique ID Enter a unique name for the job Required
    Data Tenant Specify the unique data tenant name to track the discovered data for each job. Required
    Access Key Specify the AWS access key. Required
    Secret Key Specify the AWS secret key. Required. Must be encrypted.
    Region Specify the AWS region or multiple regions to discover. Required
    Property to exclude Single or multiple properties to exclude Optional
    Proxy Specify the proxy ('true' or 'false') Required. If set to 'true', you must supply the proxy host, port and protocol.
    Proxy Host Specify the proxy host Optional (unless proxy is set to 'true')
    Proxy Port Specify the proxy port Optional (unless proxy is set to 'true')
    Proxy Protocol Specify the proxy protocol Optional (unless proxy is set to 'true')
    Job schedule

    Specify when the job should run, and whether it should run at regular intervals.

    By default the job runs immediately, and only once.

    Optionally you can specify a future date and time for the job to run, and then set it to run at regular intervals after that.

    Optional. Transient (one-off) jobs only.

    If you set a job schedule, the run intervals must be at least 90 seconds apart, and if you set them at less than 15 minutes, a warning is displayed, as the frequency can impact system performance.
    Observer job description Enter additional information to describe the job. Optional
    Encryption requirement:
    The Load job requires the secretKey in the configuration file in encrypted form. To encrypt, run the encrypt_password.sh script in the ASM_HOME/bin directory: 
    ./bin/encrypt_password.sh
    Enter and then confirm the secret key. The encryption utility will return an encrypted secretKey.

To start the Load job

  1. To start the AWS Observer Full Topology Upload job, use the following command: 
    $ASM_HOME/bin/aws_observer_load_start.sh

Results

This job loads all requested topology data, and runs only once. Run this job whenever you need AWS topology data refreshed.

What to do next

You can also use the following scripts:
aws_observer_load_stop.sh
Stops the Load job
aws_observer_job_list.sh
Lists the status of current jobs
aws_observer_log_level.sh
Sets the log level
Remember: As an alternative to being configured using the Observer Configuration UI, observer jobs have scripts to start and stop all available jobs, to list the status of a current job, and to set its logging levels. These scripts can be run with -h or --help to display help information, and with -v or --verbose to print out the details of the actions performed by the script, including the full cURL command. For the on-prem version of Agile Service Manager, observer scripts are configured for specific jobs by editing the script configuration files.