Enabling partial query logs

DNS query logs detail all requests handled by the DNS nameservers, providing insight into DNS traffic, dead records, and growth analysis. On the IBM® NS1 Connect® platform, query logs are aggregated within time buckets (30 seconds) and emitted to a customer-defined S3 location (bucket and prefix). S3 objects are gzip-encoded JSONL (line-delimited JSON) where each line represents a single aggregation. The object keys are formatted with process times, whereas the logs are timestamped with the event time.

Granting access to the IBM NS1 Connect® platform

Follow the instructions provided in the AWS documentation to grant third-party access to your S3 location. Then, contact IBM support with the following information:

S3 bucket name, region, and prefix. Note: The prefix should terminate with a forward slash (‘/’).
Amazon Resource Name (ARN) of an AWS Identity and Access Management (IAM) role with the following policies:
- At least s3:PutObject permissions for the above S3 location.
- A trust policy granting NS1 sts:AssumeRole permissions to arn:aws:iam::025043166333:role/ service-role/pipeline-querylogs-role-ukj3oed7.
An sts:ExternalId on which the above trust policy is conditioned (typically a UUID).

Refer to the example Terraform configuration for creating the proper IAM roles and policies.

Example data

Object prefixes are partitioned by year, month, day, and hour (in GMT).

Example object key

s3://<customer_bucket>/<customer_prefix>dns.query.logs/2019/10/16/20/2019-10-16-20-46- 33.115951011.gz

Note: Each object is gzip-encoded. The uncompressed data format is line-deliminted JSON.

Example log line

{ 
   “count”: 10, 
   “customer”: 12345, 
   “domain”: “foo.bar.com”, 
   “metric_name”: “dns.query.logs”, 
   “network”: “0”, 
   “rectype”: “A”, 
   “timestamp”: 1571250180, 
   “zone”: “bar.com” 
}

Parameter	Description
`count`	The number of times this record was queried within the aggregation window (30 seconds).
`customer`	The IBM NS1 Connect® account ID.
`domain`	The queried record.
`metric_name`	`dns.queries.logs` is the only included data set at this time.
`network`	The unique network identifier for customers with dedicated networks (default is 0).
`rectype`	The type of DNS record queried.
`timestamp`	The query event time.
`zone`	The encompassing DNS zone for the queried record.

Example Terraform configuration

If you are using Terraform to manage your AWS resources, copy and paste the code below to apply the configuration via Terraform.

variable "ns1_querylogs_s3_bucket" {
  type        = string
  description = "The name of the destination bucket for NS1 query log objects."
}

variable "ns1_querylogs_s3_prefix" {
    type = string
    description = "The s3 prefix to prepend to all NS1 query log objects. Omit leading slash. Include trailing slash."
}

variable "ns1_querylogs_external_id" {
    type = string
    description = "An agreed-upon value for assuming external IAM roles (typically a UUID): https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html"
}

resource "aws_iam_role" "ns1_querylogs" {
  name = "ns1-querylogs-role"
  description = "The role that NS1 assumes to send query logs logs to this AWS account."
  path               = "/"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::025043166333:role/service-role/pipeline-querylogs-role-ukj3oed7"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "${var.ns1_querylogs_external_id}"
        }
      }
    }
  ]
}
EOF
}

resource "aws_iam_policy" "ns1_querylogs" {
  name = "ns1-querylogs-policy"
  description = "Allows s3 objects to be put to a specific bucket and prefix."
  path        = "/"
  policy      = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::${var.ns1_querylogs_s3_bucket}/${var.ns1_querylogs_s3_prefix}*"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "ns1_querylogs" {
  role       = aws_iam_role.ns1_querylogs.name
  policy_arn = aws_iam_policy.ns1_querylogs.arn
}