camera-cards

suite of scripts for processing digital files from various camera manufacturers in their original structue and format

camera_cards microservice and instructions for use

By default, this script will detect a born-digital camera card structure and transform the original camera card directory structure into a usable AIP. The script will:

• Concatenate video files into a single file
• Generate technical metadata files, including a tree and MediaInfo, FFprobe, and Exiftool reports
• Make a new directory structure, with an "objects" directory for audiovisual files, a "camera_metadata" directory for significant camera-generated metadata files, and a "nmaahc_metadata" directory for files generated by the script
• Create a log of these changes

The script can currently recognize the following camera cards:

• AVCHD (e.g. Canon C100)
• Canon XF (e.g. Canon C300)
• P2 (Panasonic)
• XAVC (Sony)
• XDCAM EX (e.g. Canon XF100)

If you prefer to package files as-is to preserve your original camera directories, you can choose to compress your directory structure into a tarball. To select this option, use the '-t' flag.

If your camera card structure is not recognized, the script will prompt you to choose to restructure and/or package your card as-is. If you choose to restructure your camera card, the script applies a generic profile, which identifies video and audio files using MediaInfo and sorts them into an "objects" directory; all other camera-generated files are sorted into a "camera_metadata" directory.

This script is still a work in progress. In all cases, it is highly recommended that you review the output to make sure the script was able to process your camera files properly.

usage

• This script takes a set of options. Options can be combined, and the order of the options does not matter, as long as they are in between camera_cards and your input. Some options require an argument following immediately after the option flag (e.g., -d must be followed with the path of the directory you specify: -d [directory/path]); these arguments are detailed below. None of the options are required to run the command; the script will interview the operator if any of the options are left out.

  • -m: specify MEDIAID (type media id for final package, e.g. SC0001_20190101_CAMERA_OUTPUT)
    • Your command will look like this: camera_cards SC0001_20190101_CAMERA_OUTPUT /path/to/camera_card_directory
  • -o: specify AIP_DESTINATION (the output directory for the package)
    • If the AIP_DESTINATION is not supplied by the operator, the script will look for it in nmaahcconfig.
    • Your command will look like this: camera_cards -o /path/to/aip_destination /path/to/camera_card_directory
  • -c: combine multiple camera cards into one output package
    • The script will ask you to drag in one at a time, or you can supply both on the command line, as below.
    • Your command will look like this: camera_cards -c /path/to/camera_card_directory_1 /path/to/camera_card_directory_2
  • -t: tar camera files and folders into a tarball, compressed with gzip
    • Your command will look like this: camera_cards -t /path/to/camera_card_directory
  • -a: create a packaged AIP; this is the default option, but you can use this flag to specify you want an AIP alongside a tarball (e.g. '-ta')
    • Your command will look like this: camera_cards -a /path/to/camera_card_directory
  • -h: display help text in the terminal
    • Your command will look like this: camera_cards -h

• Your overall command will look like this: camera_cards [ -m MEDIAID ] [ -t ] /path/to/camera_card_directory

script logic

The script currently detects camera card structure based on directory or file name. It looks for the following directories or files in quotation marks:

• AVCHD: The "PRIVATE" directory contains all the camera card's video files.
• MXF (Canon C300): The "CLIPS001" folder directory contains all the camera card's video files.
• P2: The "LASTCLIP.TXT" file identifies the last clip recorded to the card.
• XAVC: The "XDROOT" directory is the parent directory for the P2 format.
• XDCAM EX: The "BPAV" directory is the parent directory for the XDCAM EX format.

If the camera card directory contains none of these files or subdirectories, the script applies the generic profile.

The script identifies and concatenates all audiovisual files. It preserves only significant camera-generated metadata files. "Significant" camera-generated metadata files are determined based on community members' assessments of whether these files are essential for playback, or hold important information about the audiovisual files' properties. Other standard metadata sidecar files for audiovisual content are also generated.

camera card structures: before and after restructuring

generic profile

If your camera card structure is not recognized, the script will prompt you to choose to restructure and/or package your card as-is. If you choose to restructure your camera card, the script applies the following generic profile:

    MEDIAID
    ├── nmaahc_metadata
    │   ├── MEDIAID_camera_card_directory.tree.txt
    │   ├── VIDEOFILE1_mediainfo.txt
    │   ├── VIDEOFILE1_exiftool.txt
    │   ├── VIDEOFILE1_ffprobe.xml
    │   ├── VIDEOFILE2_mediainfo.txt
    │   ├── VIDEOFILE2_exiftool.txt
    │   └── VIDEOFILE2_ffprobe.xml [etc.]
    ├── camera_metadata
    │   └── [all non-audiovisual files generated by camera]
    └── objects [all audiovisual files generated by camera]
        ├── VIDEOFILE1.ext
        ├── VIDEOFILE2.ext [etc.]
        ├── AUDIOFILE1.ext [etc., if present]
        └── MEDIAID.ext [concatenated video file created by script]

packaging files as-is

If you prefer to package files as-is to preserve your original camera directories, you can choose to compress your directory structure into a tarball. To select this option, use the '-t' flag. Your output will look like this:

sources and further resources

Thank you to the moving image archiving community members who have researched and written about born-digital camera card sturctures:

general resources and overviews:

• Claire Fox, "Not Normalized: Born-Digital Camera Original Video Formats in the Archives" (MIAP thesis, 2020), https://miap.hosting.nyu.edu/program/student_work/2020spring/20s_thesis_Fox_deposit_copy_y.pdf
  • An exploration of issues in documenting born-digital file-based camera original video formats for archivists.
• FADGI Audio-Visual Working Group, "Creating and Archiving Born Digital Video, Part II. Eight Federal Case Histories" (2014), http://www.digitizationguidelines.gov/guidelines/FADGI_BDV_p2_20141202.pdf
  • Case histories for archiving born-digital video (particularly referencing Crystal Sanchez, Stephanie Christensen, and Isabel Meyer's NMNH/DAMS case study)
• Kieran O'Leary, concat.py (script), https://github.com/kieranjol/IFIscripts/blob/master/concat.py
  • Python script that concatenates video files for XDCAM workflows at the Irish Film Institute
• Angelo Sacerdote, "Digital Dilemmas: Dealing with Born-Digital and Digital Surrogate Audio and Audio-Visual Collections" (SAA presentation, 2008), https://1library.net/document/qvg4m1dq-digital-dilemmas-dealing-digital-digital-surrogate-visual-collections.html
  • Introduction to born-digital audiovisual formats, including discussion of P2 camera cards
• Crystal Sanchez, Taylor McBride, and contributors, "Born-Digital Camera-Original Video: Practices and Risks" (Google spreadsheet), https://docs.google.com/spreadsheets/d/1OvZkGkizNnx_nZ9OVDOkJJIVFuIMK_7FYKC77YhoUac/edit#gid=0
  • Crowdsourced overview of born-digital camera-original formats and challenges particular to each
• Jonah Volk, "A Producer's Guide to File-Based Digital Video Preservation" (MIAP thesis, 2010), https://miap.hosting.nyu.edu/program/student_work/2010spring/JonahVolk_Thesis_20110908__y.pdf
  • An exploration of issues in preserving file-based digital video and a proposed workflow for digital video producers.

resources for specific camera card formats

• AVCHD (e.g. Canon C100): documentation thanks to Robert Hoffman, summer 2020 NMAAHC intern; see also Claire Fox's thesis, page 50
• Canon XF (e.g. Canon C300): documentation thanks to Robert Hoffman, summer 2020 NMAAHC intern, and Rupert Howe, http://www.premierepro.org/2016/10/canon-xf-c300-xf305-folder-structure-and-file-naming-explained/
• P2 (Panasonic): see Jonah Volk's thesis, page 6, and Angelo Sacerdote's presentation, slide 9
• XAVC (Sony): see Claire Fox's thesis, pages 50—51
• XDCAM EX (e.g. Canon XF100): see Jonah Volk's thesis, page 9, and Claire Fox's thesis, page 50