Video Annotation Data Format

  • Updated

This is documentation of the 2.0 version of the Alegion video annotation result format. This version aligns the video annotation result data format with Alegion image annotation and features several enhancements to make it more human readable and easier to use in scripting.

 

Data Type Alegion-video
Format Version 2.0

 

Format

  • fps - frame rate of video, this should be generated by the alegion video validator or through uploading to asset service and must match the asset at the url provided
  • url - url of video asset

  • s3ReferenceKey - optional - generated as part of result data generation… should be removed?

  • videoAnnotationId - optional - generated as part of annotation work/result data generation… should be removed?

  • manifest - top level global information about the video annotation record including entity information as well as metadata about the tooling

  • frames - frame level entity data including localization, classifications, and associative relationships

{
  "fps" : "20",
  "url" : "https://cdn.app.alegion.com/tenant-assets/dmcmorris/d0d5aba2-85a5-4644-abef-6a6321588ff7_ggt9_the_cool_one.mp4",
  "s3ReferenceKey" : "s3://app-video-annotation-output/f00d2e32-1ce6-48c3-8d52-689cd70b4a30/1d89511b-defb-40ee-b61b-6b0c7c743cc6/video-annotation-result.json",
  "videoAnnotationId" : "1d89511b-defb-40ee-b61b-6b0c7c743cc6"
  "manifest": {...}
  "frames": : [...]
}

 

Manifest

  • annotationTool - type and version of the tool/app that generated the annotation data

  • resultFormat - type and version of result data format expected in the file, moving forward manifest and this field will always be present so parsers can determine what data to expect and if the format they are reading is supported

  • annotatedAt - timestamp of when the annotation was completed

  • entityTypes - list of all possible entity types configured for the annotation task, not all types are guaranteed to have entities annotated

  • entityHierarchy - represents hierarchical entity information (previously called compositional relationships). These hierarchies are considered to be intrinsic to the entity types and do not change frame to frame. All entities will have an entry in the hierarchy. Hierarchical entities cannot be nested currently, only one level is supported.

{
    "annotationTool": {
        "type": "video-annotation-component",
        "version": "6.0"
    },
    "resultFormat": {
        "type": "alegion-video",
        "version": "2.0"
    },
    "annotatedAt": "2021-10-27T10:15:30Z",
    "entityTypes": [
        {
            "name": "Person",
            "shapeType": "bounding-box"
        },
        {
            "name": "Head",
            "shapeType": "polygon"
        },
        {
            "name": "Item",
            "shapeType": "bounding-box"
        }
    ],
    "entityHierarchy": [
        {
            "parent": {
                "id": "3674c2f7-5c7b-4e54-b95a-ba278cf1e1cd",
                "name": "Person 1",
                "entityTypeName": "Person"
            },
            "children": [
                {
                    "id": "d2763a48-98ff-4910-ba5b-885ee5d77403",
                    "name": "Head 1",
                    "entityTypeName": "Head"
                }
            ]
        },
        {
            "parent": {
                "id": "1b56f688-e467-4918-8d08-01d56721b860",
                "name": "Item 1",
                "entityTypeName": "Item"
            },
            "children": []
        }
    ]
}

 

Frames

  • frameNumber - 0 indexed number of frame data represents. frameNumber is expected to be sorted and without gaps with an entry for every frame in the video.

  • entities - entity information for all entities visible/in-frame for the video at the provided frameNumber. Frames without any entity data should still be present, but will simply have an empty entities list.

[
    {
        "frameNumber": 0,
        "entities": []
    },
    {
        "frameNumber": 1,
        "entities": [
            {...},
            {...}
        ]
    }
]

 

Entity Data

  • id - unique UUID for entity, UUID is expected to be unique within the annotation record, but may be reused in other annotation records (happens when existing annotations are imported)

  • name - human readable name for entity, typically the entityTypeName with a index based on order it was created

  • shapeType - localization type, supported values are bounding-box, keypoint, polygon, line, and none

  • classifications - values for all classifications, if set, at the frame number for the entity regardless if they were altered on the frame or not

  • relationships - associative relationships active for the entity at the given frame

  • coordinates - localization coordinates for localizations types that require them (all but none)

  • keyframeType - a flag used to inform parsers what values were explicitly annotated on this entity for the frame number. possible values are classification, position, and relationship

{
    "id": "6bc6b364-2d0b-4e21-a162-b1df50d0428c",
    "name": "Person 1",
    "entityTypeName": "Person",
    "shapeType": "bounding-box",
    "classifications": {},
    "relationships": [
        {
            "objectId": "1b56f688-e467-4918-8d08-01d56721b860",
            "objectInstanceName": "Item 1"
        }
    ],
    "coordinates": [
        {
            "x": 162,
            "y": 254
        },
        {
            "x": 162,
            "y": 34
        },
        {
            "x": 426,
            "y": 34
        },
        {
            "x": 426,
            "y": 254
        }
    ],
    "keyframeType": [
        "classification",
        "position",
        "relationship"
    ]
}

 

Differences from Version 1.0

  • ADDITIONAL added top level manifest

  • BREAKING CHANGE moved compositional relationships from within the frame level data to entityHierarchy inside of the manifest

  • ADDITIONAL added objectInstanceName to value inside of relationships for associative relationships

 

Example

5 frames of data with hierarchical entities, associative relationships, and classifications

{
  "fps": "25",
  "url": "https://alegion-public.s3-us-west-2.amazonaws.com/coltrane/sample-videos/basketball.mov",
  "s3ReferenceKey": "s3://app-dev-video-annotation-output/65d08268-7f01-4733-8669-47c7a08e4646/bf0fbf7f-c609-4ba4-9961-9009f13717c6/video-annotation-result.json",
  "videoAnnotationId": "bf0fbf7f-c609-4ba4-9961-9009f13717c6",
  "manifest": {
    "annotationTool": {
      "type": "video-annotation-component",
      "version": "6.0"
    },
    "resultFormat": {
      "type": "alegion-video",
      "version": "2.0"
    },
    "annotatedAt": "2021-10-27T10:15:30Z",
    "entityTypes": [
      {
        "name": "Person",
        "shapeType": "bounding-box"
      },
      {
        "name": "Hand",
        "shapeType": "bounding-box"
      },
      {
        "name": "Item",
        "shapeType": "bounding-box"
      }
    ],
    "entityHierarchy": [
      {
        "parent": {
          "id": "6dd7678e-99c8-4321-84dd-17f4648bc7ab",
          "name": "Person 1",
          "entityTypeName": "Person"
        },
        "children": [
          {
            "id": "e6fa3f96-bceb-48db-a36b-1f1ec4ca7f44",
            "name": "Hand 1",
            "entityTypeName": "Hand"
          }
        ]
      },
      {
        "parent": {
          "id": "d8edc6ba-c42a-4352-893e-bd7983e84e26",
          "name": "Item 1",
          "entityTypeName": "Item"
        },
        "children": []
      }
    ]
  },
  "frames": [
    {
      "frameNumber": 0,
      "entities": [
        {
          "shapeType": "bounding-box",
          "name": "Person 1",
          "entityTypeName": "Person",
          "classifications": {},
          "id": "6dd7678e-99c8-4321-84dd-17f4648bc7ab",
          "keyframeType": [
            "classification",
            "position"
          ],
          "relationships": [],
          "coordinates": [
            {
              "x": 189,
              "y": 825
            },
            {
              "x": 189,
              "y": 204
            },
            {
              "x": 568,
              "y": 204
            },
            {
              "x": 568,
              "y": 825
            }
          ]
        },
        {
          "shapeType": "bounding-box",
          "name": "Item 1",
          "entityTypeName": "Item",
          "classifications": {
            "type": "personal"
          },
          "id": "d8edc6ba-c42a-4352-893e-bd7983e84e26",
          "keyframeType": [
            "classification",
            "position"
          ],
          "relationships": [],
          "coordinates": [
            {
              "x": 190,
              "y": 587
            },
            {
              "x": 190,
              "y": 490
            },
            {
              "x": 282,
              "y": 490
            },
            {
              "x": 282,
              "y": 587
            }
          ]
        },
        {
          "shapeType": "bounding-box",
          "name": "Hand 1",
          "entityTypeName": "Hand",
          "classifications": {
            "side": "right"
          },
          "id": "e6fa3f96-bceb-48db-a36b-1f1ec4ca7f44",
          "keyframeType": [
            "classification",
            "position",
            "relationship"
          ],
          "relationships": [
            {
              "objectInstanceName": "Item 1",
              "objectId": "d8edc6ba-c42a-4352-893e-bd7983e84e26"
            }
          ],
          "coordinates": [
            {
              "x": 194,
              "y": 520
            },
            {
              "x": 194,
              "y": 467
            },
            {
              "x": 266,
              "y": 467
            },
            {
              "x": 266,
              "y": 520
            }
          ]
        }
      ]
    },
    {
      "frameNumber": 1,
      "entities": [
        {
          "shapeType": "bounding-box",
          "name": "Person 1",
          "entityTypeName": "Person",
          "classifications": {},
          "id": "6dd7678e-99c8-4321-84dd-17f4648bc7ab",
          "keyframeType": [],
          "relationships": [],
          "coordinates": [
            {
              "x": 184,
              "y": 825
            },
            {
              "x": 184,
              "y": 204
            },
            {
              "x": 568,
              "y": 204
            },
            {
              "x": 568,
              "y": 825
            }
          ]
        },
        {
          "shapeType": "bounding-box",
          "name": "Item 1",
          "entityTypeName": "Item",
          "classifications": {
            "type": "personal"
          },
          "id": "d8edc6ba-c42a-4352-893e-bd7983e84e26",
          "keyframeType": [],
          "relationships": [],
          "coordinates": [
            {
              "x": 184,
              "y": 574
            },
            {
              "x": 184,
              "y": 477
            },
            {
              "x": 276,
              "y": 477
            },
            {
              "x": 276,
              "y": 574
            }
          ]
        },
        {
          "shapeType": "bounding-box",
          "name": "Hand 1",
          "entityTypeName": "Hand",
          "classifications": {
            "side": "right"
          },
          "id": "e6fa3f96-bceb-48db-a36b-1f1ec4ca7f44",
          "keyframeType": [],
          "relationships": [
            {
              "objectInstanceName": "Item 1",
              "objectId": "d8edc6ba-c42a-4352-893e-bd7983e84e26"
            }
          ],
          "coordinates": [
            {
              "x": 185,
              "y": 507
            },
            {
              "x": 185,
              "y": 454
            },
            {
              "x": 257,
              "y": 454
            },
            {
              "x": 257,
              "y": 507
            }
          ]
        }
      ]
    },
    {
      "frameNumber": 2,
      "entities": [
        {
          "shapeType": "bounding-box",
          "name": "Person 1",
          "entityTypeName": "Person",
          "classifications": {},
          "id": "6dd7678e-99c8-4321-84dd-17f4648bc7ab",
          "keyframeType": [],
          "relationships": [],
          "coordinates": [
            {
              "x": 178,
              "y": 825
            },
            {
              "x": 178,
              "y": 204
            },
            {
              "x": 568,
              "y": 204
            },
            {
              "x": 568,
              "y": 825
            }
          ]
        },
        {
          "shapeType": "bounding-box",
          "name": "Item 1",
          "entityTypeName": "Item",
          "classifications": {
            "type": "personal"
          },
          "id": "d8edc6ba-c42a-4352-893e-bd7983e84e26",
          "keyframeType": [],
          "relationships": [],
          "coordinates": [
            {
              "x": 177,
              "y": 560
            },
            {
              "x": 177,
              "y": 463
            },
            {
              "x": 269,
              "y": 463
            },
            {
              "x": 269,
              "y": 560
            }
          ]
        },
        {
          "shapeType": "bounding-box",
          "name": "Hand 1",
          "entityTypeName": "Hand",
          "classifications": {
            "side": "right"
          },
          "id": "e6fa3f96-bceb-48db-a36b-1f1ec4ca7f44",
          "keyframeType": [],
          "relationships": [
            {
              "objectInstanceName": "Item 1",
              "objectId": "d8edc6ba-c42a-4352-893e-bd7983e84e26"
            }
          ],
          "coordinates": [
            {
              "x": 175,
              "y": 494
            },
            {
              "x": 175,
              "y": 441
            },
            {
              "x": 247,
              "y": 441
            },
            {
              "x": 247,
              "y": 494
            }
          ]
        }
      ]
    },
    {
      "frameNumber": 3,
      "entities": [
        {
          "shapeType": "bounding-box",
          "name": "Person 1",
          "entityTypeName": "Person",
          "classifications": {},
          "id": "6dd7678e-99c8-4321-84dd-17f4648bc7ab",
          "keyframeType": [
            "position"
          ],
          "relationships": [],
          "coordinates": [
            {
              "x": 173,
              "y": 825
            },
            {
              "x": 173,
              "y": 204
            },
            {
              "x": 568,
              "y": 204
            },
            {
              "x": 568,
              "y": 825
            }
          ]
        },
        {
          "shapeType": "bounding-box",
          "name": "Item 1",
          "entityTypeName": "Item",
          "classifications": {
            "type": "personal"
          },
          "id": "d8edc6ba-c42a-4352-893e-bd7983e84e26",
          "keyframeType": [
            "position"
          ],
          "relationships": [],
          "coordinates": [
            {
              "x": 171,
              "y": 547
            },
            {
              "x": 171,
              "y": 450
            },
            {
              "x": 263,
              "y": 450
            },
            {
              "x": 263,
              "y": 547
            }
          ]
        },
        {
          "shapeType": "bounding-box",
          "name": "Hand 1",
          "entityTypeName": "Hand",
          "classifications": {
            "side": "right"
          },
          "id": "e6fa3f96-bceb-48db-a36b-1f1ec4ca7f44",
          "keyframeType": [
            "position"
          ],
          "relationships": [
            {
              "objectInstanceName": "Item 1",
              "objectId": "d8edc6ba-c42a-4352-893e-bd7983e84e26"
            }
          ],
          "coordinates": [
            {
              "x": 166,
              "y": 481
            },
            {
              "x": 166,
              "y": 428
            },
            {
              "x": 238,
              "y": 428
            },
            {
              "x": 238,
              "y": 481
            }
          ]
        }
      ]
    },
    {
      "frameNumber": 4,
      "entities": [
        {
          "shapeType": "bounding-box",
          "name": "Person 1",
          "entityTypeName": "Person",
          "classifications": {},
          "id": "6dd7678e-99c8-4321-84dd-17f4648bc7ab",
          "keyframeType": [],
          "relationships": [],
          "coordinates": [
            {
              "x": 173,
              "y": 825
            },
            {
              "x": 173,
              "y": 204
            },
            {
              "x": 568,
              "y": 204
            },
            {
              "x": 568,
              "y": 825
            }
          ]
        },
        {
          "shapeType": "bounding-box",
          "name": "Item 1",
          "entityTypeName": "Item",
          "classifications": {
            "type": "personal"
          },
          "id": "d8edc6ba-c42a-4352-893e-bd7983e84e26",
          "keyframeType": [],
          "relationships": [],
          "coordinates": [
            {
              "x": 171,
              "y": 547
            },
            {
              "x": 171,
              "y": 450
            },
            {
              "x": 263,
              "y": 450
            },
            {
              "x": 263,
              "y": 547
            }
          ]
        },
        {
          "shapeType": "bounding-box",
          "name": "Hand 1",
          "entityTypeName": "Hand",
          "classifications": {
            "side": "right"
          },
          "id": "e6fa3f96-bceb-48db-a36b-1f1ec4ca7f44",
          "keyframeType": [],
          "relationships": [
            {
              "objectInstanceName": "Item 1",
              "objectId": "d8edc6ba-c42a-4352-893e-bd7983e84e26"
            }
          ],
          "coordinates": [
            {
              "x": 166,
              "y": 481
            },
            {
              "x": 166,
              "y": 428
            },
            {
              "x": 238,
              "y": 428
            },
            {
              "x": 238,
              "y": 481
            }
          ]
        }
      ]
    }
  ]
}
Share This