Skip to main content

Streaming API

The OpsHelm Streaming API provides real time access to the data collected by the OpsHelm platform. By consuming the real time API, you can get immediate access to fully enriched and standardized cloud audit logs, including Create, Update, and Delete events with attribution for assets in your cloud, cost and pricing updates, and issues detected by OpsHelm.

Connecting to the API​

The base URL for the streaming api is:

https://streaming.opshelm.com/v1

Authentication​

Requests to the Streaming API require an API key. API keys should be specified as a bearer token in the request via the Authorization header as follows:

Authorization: Bearer <api key>

Replace <api key> with a valid API key.

API Reference

See the API Reference for information on how to consume the API.

Event Types

By default, events delivered by the streaming API conform to the CloudEvents specification. As such, events can be identified by examining the various metadata fields on the object. This is useful should you wish to route or handle certain types of events differently.

Event Fields​

The CloudEvents specification defines the following fields, which will be set by as described below:

source​

The source field identifies context within which the event occurred. Within the OpsHelm platform, events are generally associated with a specific cloud account or environment. This will generally be set to a value such as /cloudAccounts/01HKB3QFZK67MZSHZGYYPPF8Q, which identifies the cloud account by its unique ID within the platform.

type​

The type field describes the class of event. Events returned from the API will have a type field that starts with com.opshelm, such as com.opshelm.asset.create. See below for some event types and sample events.

subject​

The subject field identifies the entity that is the subject of a particular event. This will generally be a resource that exists within or is tracked by OpsHelm. For example, an event for a specific asset will have a subject that identifies the cloud account and asset: /cloudAccounts/01HKB3QFZK67MZSHZGYYPPF8QW/assets/01J05FDVTBZ9W2E7TWDP2R5KTV

data​

The data field will contain the event payload.

Inventory Events​

The streaming API provides events for changes to your cloud inventory as they're received and processed by the platform. The event types are documented below along with some sample events that show what an event might look like.

Inventory Event Fields​

An inventory record, which is located under either the current or previous field in the data field, contains the following fields:

meta​

The meta field contains metadata about the resource in question, such as the ID or the type of resource. E.g.:

"meta": {
    "arn": "arn:aws:ecs:us-west-2:123456789012:task-definition/api:504",
    "region": "us-west-2",
    "resource_type": "AWS::ECS::TaskDefinition"
}

attribution​

The attribution field includes information about the actor (IAM user, role, service) that created the version asset version included in the event and how the change was made (Console, Terraform, SDK). E.g.:

"attribution": {
    "IP": "1.2.3.4",
    "Agent": "AwsSdk",
    "Platform": "AWS",
    "UserAgent": "aws-sdk-nodejs/2.1614.0 linux/v20.13.1 amazon-ecs-deploy-task-definition-for-github-actions promise",
    "Principal": {
        "Type": "iam_user",
        "ID": "dev/ci/api-deploy",
        "ARN": "arn:aws:iam::123456789012:user/dev/ci/api-deploy",
        "RoleHistory": [],
        "CloudProvider": "aws"
    },
    "Version": 1
}

resource​

The resource field contains the full configuration of the resource.

Event Types​

Create​

type: com.opshelm.asset.create

Example: The following event is an asset creation event for a new ECS Task Definition version

Note: The data object of a creation event contains the configuration of the newly created resource under the current.resource attribute.

Show Event
{
  "specversion": "1.0",
  "id": "131962182531032158",
  "source": "/cloudAccounts/01HKB3QFZK67MZSHZGYYPPF8QW",
  "type": "com.opshelm.asset.create",
  "subject": "/cloudAccounts/01HKB3QFZK67MZSHZGYYPPF8QW/assets/01J05FDVTBZ9W2E7TWDP2R5KTV",
  "datacontenttype": "application/json",
  "time": "2024-06-12T05:41:04Z",
  "data": {
    "current": {
      "meta": {
        "arn": "arn:aws:ecs:us-west-2:123456789012:task-definition/api:504",
        "region": "us-west-2",
        "resource_type": "AWS::ECS::TaskDefinition"
      },
      "attribution": {
        "IP": "1.2.3.4",
        "Agent": "AwsSdk",
        "Platform": "AWS",
        "UserAgent": "aws-sdk-nodejs/2.1614.0 linux/v20.13.1 amazon-ecs-deploy-task-definition-for-github-actions promise",
        "Principal": {
          "Type": "iam_user",
          "ID": "dev/ci/api-deploy",
          "ARN": "arn:aws:iam::123456789012:user/dev/ci/api-deploy",
          "RoleHistory": [],
          "CloudProvider": "aws"
        },
        "Version": 1
      },
      "resource": {
        "Compatibilities": [
          "EC2"
        ],
        "ContainerDefinitions": [
          {
            "Command": null,
            "Cpu": 0,
            "CredentialSpecs": null,
            "DependsOn": null,
            "DisableNetworking": null,
            "DnsSearchDomains": null,
            "DnsServers": null,
            "DockerLabels": {
              "ECS_PROMETHEUS_EXPORTER_PORT": "9090",
              "ECS_PROMETHEUS_JOB_NAME": "api",
              "ECS_PROMETHEUS_METRICS_PATH": "/metrics"
            },
            "DockerSecurityOptions": null,
            "EntryPoint": null,
            "Environment": [
              {
                "Name": "AWS_REGION",
                "Value": "us-west-2"
              },
              {
                "Name": "ENVIRONMENT",
                "Value": "prod"
              }
            ],
            "EnvironmentFiles": null,
            "Essential": true,
            "ExtraHosts": null,
            "FirelensConfiguration": null,
            "HealthCheck": null,
            "Hostname": null,
            "Image": "123456789012.dkr.ecr.us-west-2.amazonaws.com/api:ee37de9",
            "Interactive": null,
            "Links": null,
            "LinuxParameters": null,
            "LogConfiguration": {
              "LogDriver": "awslogs",
              "Options": {
                "awslogs-group": "api",
                "awslogs-region": "us-west-2"
              },
              "SecretOptions": null
            },
            "Memory": null,
            "MemoryReservation": null,
            "MountPoints": [],
            "Name": "api",
            "PortMappings": [
              {
                "AppProtocol": "",
                "ContainerPort": 8080,
                "ContainerPortRange": null,
                "HostPort": 8080,
                "Name": null,
                "Protocol": "tcp"
              }
            ],
            "Privileged": null,
            "PseudoTerminal": null,
            "ReadonlyRootFilesystem": null,
            "RepositoryCredentials": null,
            "ResourceRequirements": null,
            "Secrets": null,
            "StartTimeout": null,
            "StopTimeout": null,
            "SystemControls": [],
            "Ulimits": null,
            "User": null,
            "VolumesFrom": [],
            "WorkingDirectory": null
          }
        ],
        "Cpu": "128",
        "DeregisteredAt": null,
        "EphemeralStorage": null,
        "ExecutionRoleArn": "arn:aws:iam::123456789012:role/dev/infra/ecs-api-execution",
        "Family": "api",
        "InferenceAccelerators": null,
        "IpcMode": "",
        "Memory": "256",
        "NetworkMode": "awsvpc",
        "PidMode": "",
        "PlacementConstraints": [],
        "ProxyConfiguration": null,
        "RegisteredAt": "2024-06-12T05:41:04.795Z",
        "RegisteredBy": "arn:aws:iam::123456789012:user/dev/ci/api-deploy",
        "RequiresAttributes": [
          {
            "Name": "com.amazonaws.ecs.capability.logging-driver.awslogs",
            "TargetId": null,
            "TargetType": "",
            "Value": null
          },
          {
            "Name": "ecs.capability.execution-role-awslogs",
            "TargetId": null,
            "TargetType": "",
            "Value": null
          },
          {
            "Name": "com.amazonaws.ecs.capability.ecr-auth",
            "TargetId": null,
            "TargetType": "",
            "Value": null
          },
          {
            "Name": "com.amazonaws.ecs.capability.docker-remote-api.1.19",
            "TargetId": null,
            "TargetType": "",
            "Value": null
          },
          {
            "Name": "com.amazonaws.ecs.capability.task-iam-role",
            "TargetId": null,
            "TargetType": "",
            "Value": null
          },
          {
            "Name": "ecs.capability.execution-role-ecr-pull",
            "TargetId": null,
            "TargetType": "",
            "Value": null
          },
          {
            "Name": "com.amazonaws.ecs.capability.docker-remote-api.1.18",
            "TargetId": null,
            "TargetType": "",
            "Value": null
          },
          {
            "Name": "ecs.capability.task-eni",
            "TargetId": null,
            "TargetType": "",
            "Value": null
          }
        ],
        "RequiresCompatibilities": null,
        "Revision": 504,
        "RuntimePlatform": null,
        "Status": "ACTIVE",
        "TaskDefinitionArn": "arn:aws:ecs:us-west-2:123456789012:task-definition/api:504",
        "TaskRoleArn": "arn:aws:iam::123456789012:role/dev/infra/ecs-api-task",
        "Volumes": []
      }
    }
  }
}

Update​

type: com.opshelm.asset.update

Example: The following event is an asset update event for an S3 Bucket

Note: The data object contains both the current and previous configurations of the asset, as well as an attribution with details about how the change was made.

Show Event
{
  "specversion": "1.0",
  "id": "3564325930759703107",
  "source": "/cloudAccounts/01HRD45D1VDRE9030GVD15XSCM",
  "type": "com.opshelm.asset.update",
  "subject": "/cloudAccounts/01HRD45D1VDRE9030GVD15XSCM/assets/01J05GPZ6XWEA720XZAEWB135X",
  "datacontenttype": "application/json",
  "time": "2024-06-12T06:14:17Z",
  "data": {
    "previous": {
      "meta": {
        "arn": "arn:aws:s3:::oh-example-bucket",
        "region": "us-west-2",
        "resource_type": "AWS::S3::Bucket"
      },
      "attribution": {
        "IP": "11.12.13.14",
        "Agent": "Terraform",
        "Platform": "AWS",
        "UserAgent": "APN/1.0 HashiCorp/1.0 Terraform/1.4.6 (+https://www.terraform.io) terraform-provider-aws/5.47.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.26.1 os/linux lang/go#1.22.0 md/GOOS#linux md/GOARCH#amd64 api/s3#1.27.4",
        "Principal": {
          "Type": "assumed_role",
          "ID": "terraform/aws-go-sdk-1717101594822223398",
          "ARN": "arn:aws:sts::123456789012:assumed-role/terraform/aws-go-sdk-1717101594822223398",
          "RoleHistory": [],
          "CloudProvider": "aws"
        },
        "Version": 1
      },
      "resource": {
        "BlockPublicAcls": true,
        "BlockPublicPolicy": true,
        "CreationDate": "2024-06-12T06:03:32Z",
        "IgnorePublicAcls": true,
        "IntelligentTieringConfigurations": null,
        "IsPublic": false,
        "LifecycleRules": null,
        "LoggingTargetBucket": null,
        "LoggingTargetPrefix": null,
        "Name": "oh-example-bucket",
        "OwnershipControls": [
          "BucketOwnerEnforced"
        ],
        "Policy": {
          "Statement": []
        },
        "Region": "us-west-2",
        "ReplicationRole": null,
        "ReplicationRules": null,
        "RestrictPublicBuckets": true,
        "SSEAlgorithm": "AES256",
        "Tags": null,
        "VersioningMfaDelete": "",
        "VersioningStatus": ""
      }
    },
    "current": {
      "meta": {
        "arn": "arn:aws:s3:::oh-example-bucket",
        "region": "us-west-2",
        "resource_type": "AWS::S3::Bucket"
      },
      "attribution": {
        "IP": "22.33.44.55",
        "Agent": "Console",
        "Platform": "AWS",
        "UserAgent": "[S3Console/0.4, aws-internal/3 aws-sdk-java/1.12.488 Linux/5.10.217-183.860.amzn2int.x86_64 OpenJDK_64-Bit_Server_VM/25.372-b08 java/1.8.0_372 vendor/Oracle_Corporation cfg/retry-mode/standard]",
        "Principal": {
          "Type": "assumed_role",
          "ID": "admin",
          "ARN": "arn:aws:sts::777777777777:assumed-role/admin",
          "RoleHistory": [
            {
              "Type": "assumed_role",
              "ID": "AWSReservedSSO_DevAdmin_46cc7c23e514bee5/kyle@opshelm.com",
              "ARN": "arn:aws:sts::123456789012:assumed-role/AWSReservedSSO_DevAdmin_46cc7c23e514bee5/kyle@opshelm.com"
            }
          ],
          "CloudProvider": "aws"
        },
        "Version": 1
      },
      "resource": {
        "BlockPublicAcls": false,
        "BlockPublicPolicy": false,
        "CreationDate": "2024-06-12T06:14:18Z",
        "IgnorePublicAcls": false,
        "IntelligentTieringConfigurations": null,
        "IsPublic": false,
        "LifecycleRules": null,
        "LoggingTargetBucket": null,
        "LoggingTargetPrefix": null,
        "Name": "oh-example-bucket",
        "OwnershipControls": [
          "BucketOwnerEnforced"
        ],
        "Policy": {
          "Statement": []
        },
        "Region": "us-west-2",
        "ReplicationRole": null,
        "ReplicationRules": null,
        "RestrictPublicBuckets": false,
        "SSEAlgorithm": "AES256",
        "Tags": null,
        "VersioningMfaDelete": "",
        "VersioningStatus": ""
      }
    }
  }
}

Delete​

type: com.opshelm.asset.delete

Example: The following event is an asset deletion event for an EC2 Instance

Note: The data object contains the last known configuration of the deleted resource under the previous attribute.

Show Event
{
  "specversion": "1.0",
  "id": "4562645149939754246",
  "source": "/cloudAccounts/01HRD45D1VDRE9030GVD15XSCM",
  "type": "com.opshelm.asset.delete",
  "subject": "/cloudAccounts/01HRD45D1VDRE9030GVD15XSCM/assets/01HXF5R7BTE9DV7ZAC8CZEQHX0",
  "datacontenttype": "application/json",
  "data": {
    "previous": {
      "meta": {
        "arn": "arn:aws:ec2:us-west-2:123456789012:instance/i-0cced4b94722a0043",
        "region": "us-west-2",
        "resource_type": "AWS::EC2::Instance"
      },
      "attribution": {
        "IP": "1.2.3.4",
        "Agent": "Console",
        "Platform": "AWS",
        "UserAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
        "Principal": {
          "Type": "aws_service",
          "ID": "autoscaling.amazonaws.com",
          "ARN": "arn:aws:sts::123456789012:assumed-role/AWSReservedSSO_DevAdmin_46cc7c23e514bee5/taaha@opshelm.com",
          "RoleHistory": [],
          "CloudProvider": "aws"
        },
        "Version": 1
      },
      "resource": {
        "AmiLaunchIndex": 0,
        "Architecture": "x86_64",
        "BlockDeviceMappings": [
          {
            "DeviceName": "/dev/xvda",
            "Ebs": {
              "AssociatedResource": null,
              "AttachTime": "2024-05-09T17:17:23Z",
              "DeleteOnTermination": true,
              "Status": "attached",
              "VolumeId": "vol-0f62d2de78acbcddd",
              "VolumeOwnerId": null
            }
          }
        ],
        "BootMode": "uefi-preferred",
        "CapacityReservationId": null,
        "CapacityReservationSpecification": {
          "CapacityReservationPreference": "open",
          "CapacityReservationTarget": null
        },
        "ClientToken": "ecf4c91f-1590-4f21-9e15-74815621356d",
        "CpuOptions": {
          "AmdSevSnp": "",
          "CoreCount": 1,
          "ThreadsPerCore": 1
        },
        "CurrentInstanceBootMode": "legacy-bios",
        "EbsOptimized": false,
        "ElasticGpuAssociations": null,
        "ElasticInferenceAcceleratorAssociations": null,
        "EnaSupport": true,
        "EnclaveOptions": {
          "Enabled": false
        },
        "HibernationOptions": {
          "Configured": false
        },
        "Hypervisor": "xen",
        "IamInstanceProfile": null,
        "ImageId": "ami-023e152801ee4846a",
        "InstanceId": "i-0cced4b94722a0043",
        "InstanceLifecycle": "",
        "InstanceType": "c5.4xlarge",
        "Ipv6Address": null,
        "KernelId": null,
        "KeyName": "prod-key",
        "LaunchTime": "2024-05-09T17:17:22Z",
        "Licenses": null,
        "MaintenanceOptions": {
          "AutoRecovery": "default"
        },
        "MetadataOptions": {
          "HttpEndpoint": "enabled",
          "HttpProtocolIpv6": "disabled",
          "HttpPutResponseHopLimit": 2,
          "HttpTokens": "required",
          "InstanceMetadataTags": "disabled",
          "State": "applied"
        },
        "Monitoring": {
          "State": "disabled"
        },
        "NetworkInterfaces": [
          {
            "Association": {
              "CarrierIp": null,
              "CustomerOwnedIp": null,
              "IpOwnerId": "amazon",
              "PublicDnsName": "ec2-34-219-64-236.us-west-2.compute.amazonaws.com",
              "PublicIp": "34.219.64.236"
            },
            "Attachment": {
              "AttachTime": "2024-05-09T17:17:22Z",
              "AttachmentId": "eni-attach-031bee16b7c746abb",
              "DeleteOnTermination": true,
              "DeviceIndex": 0,
              "EnaSrdSpecification": null,
              "NetworkCardIndex": 0,
              "Status": "attached"
            },
            "ConnectionTrackingConfiguration": null,
            "Description": "",
            "Groups": [
              {
                "GroupId": "sg-091f46a2e5e44f469",
                "GroupName": "public"
              }
            ],
            "InterfaceType": "interface",
            "Ipv4Prefixes": null,
            "Ipv6Addresses": [],
            "Ipv6Prefixes": null,
            "MacAddress": "02:85:e9:1a:20:07",
            "NetworkInterfaceId": "eni-06bb361799247c519",
            "OwnerId": "123456789012",
            "PrivateDnsName": "ip-172-31-28-185.us-west-2.compute.internal",
            "PrivateIpAddress": "172.31.28.185",
            "PrivateIpAddresses": [
              {
                "Association": {
                  "CarrierIp": null,
                  "CustomerOwnedIp": null,
                  "IpOwnerId": "amazon",
                  "PublicDnsName": "ec2-34-219-64-236.us-west-2.compute.amazonaws.com",
                  "PublicIp": "34.219.64.236"
                },
                "Primary": true,
                "PrivateDnsName": "ip-172-31-28-185.us-west-2.compute.internal",
                "PrivateIpAddress": "172.31.28.185"
              }
            ],
            "SourceDestCheck": true,
            "Status": "in-use",
            "SubnetId": "subnet-0e358d6b111b48ebf",
            "VpcId": "vpc-05705c528ab137937"
          }
        ],
        "OutpostArn": null,
        "Placement": {
          "Affinity": null,
          "AvailabilityZone": "us-west-2a",
          "GroupId": null,
          "GroupName": "",
          "HostId": null,
          "HostResourceGroupArn": null,
          "PartitionNumber": null,
          "SpreadDomain": null,
          "Tenancy": "default"
        },
        "Platform": "",
        "PlatformDetails": "Linux/UNIX",
        "PrivateDnsName": "ip-172-31-28-185.us-west-2.compute.internal",
        "PrivateDnsNameOptions": {
          "EnableResourceNameDnsAAAARecord": false,
          "EnableResourceNameDnsARecord": true,
          "HostnameType": "ip-name"
        },
        "PrivateIpAddress": "172.31.28.185",
        "ProductCodes": [],
        "PublicDnsName": "ec2-34-219-64-236.us-west-2.compute.amazonaws.com",
        "PublicIpAddress": "34.219.64.236",
        "RamdiskId": null,
        "RootDeviceName": "/dev/xvda",
        "RootDeviceType": "ebs",
        "SecurityGroups": [
          {
            "GroupId": "sg-091f46a2e5e44f469",
            "GroupName": "launch-wizard-12"
          }
        ],
        "SourceDestCheck": true,
        "SpotInstanceRequestId": null,
        "SriovNetSupport": null,
        "State": {
          "Code": 16,
          "Name": "running"
        },
        "StateReason": null,
        "StateTransitionReason": "",
        "SubnetId": "subnet-0e358d6b111b48ebf",
        "Tags": null,
        "TpmSupport": null,
        "UsageOperation": "RunInstances",
        "UsageOperationUpdateTime": "2024-05-09T17:17:22Z",
        "VirtualizationType": "hvm",
        "VpcId": "vpc-05705c528ab137937"
      }
    }
  }
}
Last updated on