Enabling partial query logs

DNS query logs detail all requests handled by the DNS nameservers, providing insight into DNS traffic, dead records, and growth analysis. In IBM® NS1 Connect®, query logs are aggregated within time buckets (30 seconds) and emitted to a customer-defined S3 location (bucket and prefix). S3 objects are gzip-encoded JSONL (line-delimited JSON) where each line represents a single aggregation. The object keys are formatted with process times, whereas the logs are timestamped with the event time.

Granting access to NS1 Connect

Follow the instructions provided in the AWS documentation to grant third-party access to your S3 location. Then, contact IBM support with the following information:

S3 bucket name, region, and prefix. Note: The prefix should terminate with a forward slash (‘/’).
Amazon Resource Name (ARN) of an AWS Identity and Access Management (IAM) role with the following policies:
- At least s3:PutObject permissions for the above S3 location.
- A trust policy granting NS1 Connect sts:AssumeRole permissions to arn:aws:iam::025043166333:role/ service-role/pipeline-querylogs-role-ukj3oed7.
An sts:ExternalId on which the above trust policy is conditioned (typically a UUID).

Refer to the example Terraform® configuration for creating the proper IAM roles and policies.

Example data

Object prefixes are partitioned by year, month, day, and hour (in GMT).

Example object key

s3://<customer_bucket>/<customer_prefix>dns.query.logs/2019/10/16/20/2019-10-16-20-46- 33.115951011.gz

Note: Each object is gzip-encoded. The uncompressed data format is line-deliminted JSON.

Example log line

{ 
   “count”: 10, 
   “customer”: 12345, 
   “domain”: “abc.example.com”, 
   “metric_name”: “dns.query.logs”, 
   “network”: “0”, 
   "pop":"iad",
   “rectype”: “A”, 
   “timestamp”: 1571250180, 
   “zone”: “example.com” 
}


Parameter	Description
count	The number of times this record was queried within the aggregation window (30 seconds)
customer	The NS1 Connect account ID
domain	The queried record
metric_name	`dns.queries.logs` is the only included data set at this time.
network	The unique network identifier for customers with dedicated networks (default is 0)
pop	The physical location of the point of presence (PoP) name server
rectype	The type of DNS record queried
timestamp	The query event time
zone	The encompassing DNS zone for the queried record

Example Terraform configuration

If you are using Terraform to manage your AWS resources, copy and paste the code below to apply the configuration using Terraform.

variable "ns1_querylogs_s3_bucket" {
  type        = string
  description = "The name of the destination bucket for NS1 query log objects."
}

variable "ns1_querylogs_s3_prefix" {
    type = string
    description = "The s3 prefix to prepend to all NS1 query log objects. Omit leading slash. Include trailing slash."
}

variable "ns1_querylogs_external_id" {
    type = string
    description = "An agreed-upon value for assuming external IAM roles (typically a UUID): https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html"
}

resource "aws_iam_role" "ns1_querylogs" {
  name = "ns1-querylogs-role"
  description = "The role that NS1 assumes to send query logs logs to this AWS account."
  path               = "/"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::025043166333:role/service-role/pipeline-querylogs-role-ukj3oed7"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "${var.ns1_querylogs_external_id}"
        }
      }
    }
  ]
}
EOF
}

resource "aws_iam_policy" "ns1_querylogs" {
  name = "ns1-querylogs-policy"
  description = "Allows s3 objects to be put to a specific bucket and prefix."
  path        = "/"
  policy      = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::${var.ns1_querylogs_s3_bucket}/${var.ns1_querylogs_s3_prefix}*"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "ns1_querylogs" {
  role       = aws_iam_role.ns1_querylogs.name
  policy_arn = aws_iam_policy.ns1_querylogs.arn
}