This is documentation of the 2.0 version of the Alegion video annotation result format. This version aligns the video annotation result data format with Alegion image annotation and features several enhancements to make it more human-readable and easier to use in scripting.
| Data Type | Alegion-video |
| Format Version | 2.0 |
Format
fps- frame rate of the video, this should be generated by the Alegion video validator or through uploading to asset service and must match the asset at theurlprovided-
url- URL of video asset -
manifest- top-level global information about the video annotation record including entity information as well as metadata about the tooling -
frames- frame-level entity data including localization, classifications, and associative relationships
{
"fps" : "20",
"url" : "https://cdn.app.alegion.com/tenant-assets/dmcmorris/d0d5aba2-85a5-4644-abef-6a6321588ff7_ggt9_the_cool_one.mp4",
"s3ReferenceKey" : "s3://app-video-annotation-output/f00d2e32-1ce6-48c3-8d52-689cd70b4a30/1d89511b-defb-40ee-b61b-6b0c7c743cc6/video-annotation-result.json",
"videoAnnotationId" : "1d89511b-defb-40ee-b61b-6b0c7c743cc6"
"manifest": {...}
"frames": : [...]
}
Manifest
-
annotationTool- type and version of the tool/app that generated the annotation data -
resultFormat- type and version of result data format expected in the file, moving forward manifest and this field will always be present so parsers can determine what data to expect and if the format they are reading is supported -
annotatedAt- timestamp of when the annotation was completed -
entityTypes- list of all possible entity types configured for the annotation task, not all types are guaranteed to have entities annotated -
entityHierarchy- represents hierarchical entity information (previously called compositional relationships). These hierarchies are considered to be intrinsic to the entity types and do not change from frame to frame. All entities will have an entry in the hierarchy. Hierarchical entities cannot be nested currently, only one level is supported.
{
"annotationTool": {
"type": "video-annotation-component",
"version": "6.0"
},
"resultFormat": {
"type": "alegion-video",
"version": "2.0"
},
"annotatedAt": "2021-10-27T10:15:30Z",
"entityTypes": [
{
"name": "Person",
"shapeType": "bounding-box"
},
{
"name": "Head",
"shapeType": "polygon"
},
{
"name": "Item",
"shapeType": "bounding-box"
}
],
"entityHierarchy": [
{
"parent": {
"id": "3674c2f7-5c7b-4e54-b95a-ba278cf1e1cd",
"name": "Person 1",
"entityTypeName": "Person"
},
"children": [
{
"id": "d2763a48-98ff-4910-ba5b-885ee5d77403",
"name": "Head 1",
"entityTypeName": "Head"
}
]
},
{
"parent": {
"id": "1b56f688-e467-4918-8d08-01d56721b860",
"name": "Item 1",
"entityTypeName": "Item"
},
"children": []
}
]
}
Frames
-
frameNumber- 0 indexed number of frame data represents.frameNumberis expected to be sorted and without gaps with an entry for every frame in the video. -
entities- entity information for all entities visible/in-frame for the video at the providedframeNumber. Frames without any entity data should still be present, but will simply have an emptyentitieslist.
[
{
"frameNumber": 0,
"entities": []
},
{
"frameNumber": 1,
"entities": [
{...},
{...}
]
}
]
Entity Data
-
id- unique UUID for the entity, UUID is expected to be unique within the annotation record, but may be reused in other annotation records (happens when existing annotations are imported) -
name- the human-readable name for the entity, typically theentityTypeNamewith an index based on the order in which it was created -
shapeType- localization type, supported values arebounding-box,keypoint,polygon,line, andnone -
classifications- values for all classifications, if set, at the frame number for the entity regardless if they were altered on the frame or not -
relationships- associative relationships active for the entity at the given frame -
coordinates- localization coordinates for localizations types that require them (all butnone) -
keyframeType- a flag used to inform parsers what values were explicitly annotated on this entity for the frame number. possible values areclassification,position, andrelationship
{
"id": "6bc6b364-2d0b-4e21-a162-b1df50d0428c",
"name": "Person 1",
"entityTypeName": "Person",
"shapeType": "bounding-box",
"classifications": {},
"relationships": [
{
"objectId": "1b56f688-e467-4918-8d08-01d56721b860",
"objectInstanceName": "Item 1"
}
],
"coordinates": [
{
"x": 162,
"y": 254
},
{
"x": 162,
"y": 34
},
{
"x": 426,
"y": 34
},
{
"x": 426,
"y": 254
}
],
"keyframeType": [
"classification",
"position",
"relationship"
]
}
Differences from Version 1.0
-
ADDITIONAL added top level
manifest -
BREAKING CHANGE moved compositional relationships from within the frame-level data to
entityHierarchyinside of themanifest -
ADDITIONAL added
objectInstanceNameto value inside ofrelationshipsfor associative relationships
Example
5 frames of data with hierarchical entities, associative relationships, and classifications
{
"fps": "25",
"url": "https://alegion-public.s3-us-west-2.amazonaws.com/coltrane/sample-videos/basketball.mov",
"s3ReferenceKey": "s3://app-dev-video-annotation-output/65d08268-7f01-4733-8669-47c7a08e4646/bf0fbf7f-c609-4ba4-9961-9009f13717c6/video-annotation-result.json",
"videoAnnotationId": "bf0fbf7f-c609-4ba4-9961-9009f13717c6",
"manifest": {
"annotationTool": {
"type": "video-annotation-component",
"version": "6.0"
},
"resultFormat": {
"type": "alegion-video",
"version": "2.0"
},
"annotatedAt": "2021-10-27T10:15:30Z",
"entityTypes": [
{
"name": "Person",
"shapeType": "bounding-box"
},
{
"name": "Hand",
"shapeType": "bounding-box"
},
{
"name": "Item",
"shapeType": "bounding-box"
}
],
"entityHierarchy": [
{
"parent": {
"id": "6dd7678e-99c8-4321-84dd-17f4648bc7ab",
"name": "Person 1",
"entityTypeName": "Person"
},
"children": [
{
"id": "e6fa3f96-bceb-48db-a36b-1f1ec4ca7f44",
"name": "Hand 1",
"entityTypeName": "Hand"
}
]
},
{
"parent": {
"id": "d8edc6ba-c42a-4352-893e-bd7983e84e26",
"name": "Item 1",
"entityTypeName": "Item"
},
"children": []
}
]
},
"frames": [
{
"frameNumber": 0,
"entities": [
{
"shapeType": "bounding-box",
"name": "Person 1",
"entityTypeName": "Person",
"classifications": {},
"id": "6dd7678e-99c8-4321-84dd-17f4648bc7ab",
"keyframeType": [
"classification",
"position"
],
"relationships": [],
"coordinates": [
{
"x": 189,
"y": 825
},
{
"x": 189,
"y": 204
},
{
"x": 568,
"y": 204
},
{
"x": 568,
"y": 825
}
]
},
{
"shapeType": "bounding-box",
"name": "Item 1",
"entityTypeName": "Item",
"classifications": {
"type": "personal"
},
"id": "d8edc6ba-c42a-4352-893e-bd7983e84e26",
"keyframeType": [
"classification",
"position"
],
"relationships": [],
"coordinates": [
{
"x": 190,
"y": 587
},
{
"x": 190,
"y": 490
},
{
"x": 282,
"y": 490
},
{
"x": 282,
"y": 587
}
]
},
{
"shapeType": "bounding-box",
"name": "Hand 1",
"entityTypeName": "Hand",
"classifications": {
"side": "right"
},
"id": "e6fa3f96-bceb-48db-a36b-1f1ec4ca7f44",
"keyframeType": [
"classification",
"position",
"relationship"
],
"relationships": [
{
"objectInstanceName": "Item 1",
"objectId": "d8edc6ba-c42a-4352-893e-bd7983e84e26"
}
],
"coordinates": [
{
"x": 194,
"y": 520
},
{
"x": 194,
"y": 467
},
{
"x": 266,
"y": 467
},
{
"x": 266,
"y": 520
}
]
}
]
},
{
"frameNumber": 1,
"entities": [
{
"shapeType": "bounding-box",
"name": "Person 1",
"entityTypeName": "Person",
"classifications": {},
"id": "6dd7678e-99c8-4321-84dd-17f4648bc7ab",
"keyframeType": [],
"relationships": [],
"coordinates": [
{
"x": 184,
"y": 825
},
{
"x": 184,
"y": 204
},
{
"x": 568,
"y": 204
},
{
"x": 568,
"y": 825
}
]
},
{
"shapeType": "bounding-box",
"name": "Item 1",
"entityTypeName": "Item",
"classifications": {
"type": "personal"
},
"id": "d8edc6ba-c42a-4352-893e-bd7983e84e26",
"keyframeType": [],
"relationships": [],
"coordinates": [
{
"x": 184,
"y": 574
},
{
"x": 184,
"y": 477
},
{
"x": 276,
"y": 477
},
{
"x": 276,
"y": 574
}
]
},
{
"shapeType": "bounding-box",
"name": "Hand 1",
"entityTypeName": "Hand",
"classifications": {
"side": "right"
},
"id": "e6fa3f96-bceb-48db-a36b-1f1ec4ca7f44",
"keyframeType": [],
"relationships": [
{
"objectInstanceName": "Item 1",
"objectId": "d8edc6ba-c42a-4352-893e-bd7983e84e26"
}
],
"coordinates": [
{
"x": 185,
"y": 507
},
{
"x": 185,
"y": 454
},
{
"x": 257,
"y": 454
},
{
"x": 257,
"y": 507
}
]
}
]
},
{
"frameNumber": 2,
"entities": [
{
"shapeType": "bounding-box",
"name": "Person 1",
"entityTypeName": "Person",
"classifications": {},
"id": "6dd7678e-99c8-4321-84dd-17f4648bc7ab",
"keyframeType": [],
"relationships": [],
"coordinates": [
{
"x": 178,
"y": 825
},
{
"x": 178,
"y": 204
},
{
"x": 568,
"y": 204
},
{
"x": 568,
"y": 825
}
]
},
{
"shapeType": "bounding-box",
"name": "Item 1",
"entityTypeName": "Item",
"classifications": {
"type": "personal"
},
"id": "d8edc6ba-c42a-4352-893e-bd7983e84e26",
"keyframeType": [],
"relationships": [],
"coordinates": [
{
"x": 177,
"y": 560
},
{
"x": 177,
"y": 463
},
{
"x": 269,
"y": 463
},
{
"x": 269,
"y": 560
}
]
},
{
"shapeType": "bounding-box",
"name": "Hand 1",
"entityTypeName": "Hand",
"classifications": {
"side": "right"
},
"id": "e6fa3f96-bceb-48db-a36b-1f1ec4ca7f44",
"keyframeType": [],
"relationships": [
{
"objectInstanceName": "Item 1",
"objectId": "d8edc6ba-c42a-4352-893e-bd7983e84e26"
}
],
"coordinates": [
{
"x": 175,
"y": 494
},
{
"x": 175,
"y": 441
},
{
"x": 247,
"y": 441
},
{
"x": 247,
"y": 494
}
]
}
]
},
{
"frameNumber": 3,
"entities": [
{
"shapeType": "bounding-box",
"name": "Person 1",
"entityTypeName": "Person",
"classifications": {},
"id": "6dd7678e-99c8-4321-84dd-17f4648bc7ab",
"keyframeType": [
"position"
],
"relationships": [],
"coordinates": [
{
"x": 173,
"y": 825
},
{
"x": 173,
"y": 204
},
{
"x": 568,
"y": 204
},
{
"x": 568,
"y": 825
}
]
},
{
"shapeType": "bounding-box",
"name": "Item 1",
"entityTypeName": "Item",
"classifications": {
"type": "personal"
},
"id": "d8edc6ba-c42a-4352-893e-bd7983e84e26",
"keyframeType": [
"position"
],
"relationships": [],
"coordinates": [
{
"x": 171,
"y": 547
},
{
"x": 171,
"y": 450
},
{
"x": 263,
"y": 450
},
{
"x": 263,
"y": 547
}
]
},
{
"shapeType": "bounding-box",
"name": "Hand 1",
"entityTypeName": "Hand",
"classifications": {
"side": "right"
},
"id": "e6fa3f96-bceb-48db-a36b-1f1ec4ca7f44",
"keyframeType": [
"position"
],
"relationships": [
{
"objectInstanceName": "Item 1",
"objectId": "d8edc6ba-c42a-4352-893e-bd7983e84e26"
}
],
"coordinates": [
{
"x": 166,
"y": 481
},
{
"x": 166,
"y": 428
},
{
"x": 238,
"y": 428
},
{
"x": 238,
"y": 481
}
]
}
]
},
{
"frameNumber": 4,
"entities": [
{
"shapeType": "bounding-box",
"name": "Person 1",
"entityTypeName": "Person",
"classifications": {},
"id": "6dd7678e-99c8-4321-84dd-17f4648bc7ab",
"keyframeType": [],
"relationships": [],
"coordinates": [
{
"x": 173,
"y": 825
},
{
"x": 173,
"y": 204
},
{
"x": 568,
"y": 204
},
{
"x": 568,
"y": 825
}
]
},
{
"shapeType": "bounding-box",
"name": "Item 1",
"entityTypeName": "Item",
"classifications": {
"type": "personal"
},
"id": "d8edc6ba-c42a-4352-893e-bd7983e84e26",
"keyframeType": [],
"relationships": [],
"coordinates": [
{
"x": 171,
"y": 547
},
{
"x": 171,
"y": 450
},
{
"x": 263,
"y": 450
},
{
"x": 263,
"y": 547
}
]
},
{
"shapeType": "bounding-box",
"name": "Hand 1",
"entityTypeName": "Hand",
"classifications": {
"side": "right"
},
"id": "e6fa3f96-bceb-48db-a36b-1f1ec4ca7f44",
"keyframeType": [],
"relationships": [
{
"objectInstanceName": "Item 1",
"objectId": "d8edc6ba-c42a-4352-893e-bd7983e84e26"
}
],
"coordinates": [
{
"x": 166,
"y": 481
},
{
"x": 166,
"y": 428
},
{
"x": 238,
"y": 428
},
{
"x": 238,
"y": 481
}
]
}
]
}
]
}
Get In Touch