The Norman Police Department serves more than 117,000 residents and is recognized as a leader in adopting modern surveillance technologies such as body-worn and in-vehicle cameras. With the growing use of public safety video, the department expects video files to increase from approximately 50 terabytes to approximately 5 petabytes by 2019.
Like the Norman Police Department, public agencies across the U.S. are facing a similar growth in video files. Due to a number of highly publicized public safety events, many agencies are turning to video to help protect citizens and staff. A Department of Justice survey found nearly 90 percent of local police departments are using some type of video camera technology, ranging from license plate readers to video surveillance cameras. Sixty-eight percent of the 12,000 police departments surveyed use in-vehicle cameras while another 32 percent report at least some officers are equipped with body-worn cameras. Globally, the amount of surveillance footage captured is expected to reach approximately 3.3 trillion hours by 2020.
While video is a welcomed safety tool, agencies must determine how to effectively and efficiently store and manage these massive files. Data lakes are proving to be an effective storage solution to this challenge. A data lake is an architecture that consolidates individual pools of surveillance information for collection, analysis and storage on a common, secure platform. The following best practices should be considered when designing a data lake platform.
Best Practice 1: Identify file owners, file types and storage requirements.
- Identify file owners. Organizations must work closely with departmental users to determine potential use cases for the files and understand various users’ skill levels when it comes to handling and analyzing unformatted data. In addition, organizations should identify who owns the data and is ultimately responsible for who views, manipulates and retires it.
- Identify surveillance file types and technology requirements. Organizations should take an inventory of all their “active-duty” surveillance devices, documenting not only the number of devices but also resolution, frame rate, compression and other details impacting storage needs.
- Understand laws and guidelines for storing video files. Organizations must comply with various laws and internal best practices depending on the contents of their video files and where they are stored. For example, some files are subject to the FBI’s Criminal Justice Information Services (CJIS) policies and/or Federal Risk Management and Authorization Program (FedRAMP) guidelines regarding the storage of citizens’ data in the cloud. Some require data protection and creation of an audit trail. To remain in compliance, avoid lawsuits, preserve the public’s trust and ensure the most effective use of video files, it’s important to understand and follow legal requirements and generally accepted guidelines related to storage duration, security, privacy, redactions and chain of custody.
- Consider the entire video life cycle. Organizations need to develop a systematic strategy for when, where and how to move files to the most appropriate storage at different phases in their life cycle. In doing so, they must consider the long-term implications of storing evidence over many years. In some cases, the evidence will have a longer shelf life than the device on which it is stored. Organizations need to consider the cost of storing this type of data (whether on premises or in the cloud) and ensure their storage platform has the flexibility to accommodate such a scenario.
Best Practice 2: Consider an open, scalable platform.
- Look into open standards-based technology. A storage platform based on open standards is an important differentiator among video storage solutions; it allows organizations to create best-of-breed solutions that meet their unique and evolving needs.
- Plan for scalability. With increasing use of video tools, it’s likely a public safety organization’s storage needs will grow significantly in the near future. A sustainable data lake platform must expand capacity and performance quickly to support storage needs regardless of the number of surveil¬lance devices, the size of the video files they create and the data retention laws that apply to them. In more technical terms, this means the platform should optimize data (video file) ingestion; balance data across storage clusters; and leverage shared processing, caching and storage.
- Work with a proven vendor. Organizations should ensure their vendor is experienced in providing storage solutions for unstructured data such as video, has a track record of growth and innovation in the storage arena, and demonstrates long-term financial stability.
Best Practice 3: Implement a platform that can accommodate a variety of storage scenarios and models.
- On premises. In this model, storage, management and analysis capabilities reside within the organization’s data center, and the organization has direct control of all video files and content. Many public safety agencies need to store and manage at least some subset of video data on premises (e.g., for easy access to active cases or to comply with citizen privacy mandates); however, on-premises solutions often require significant infrastructure and operational overhead.
- Cloud based. In a cloud-based model, a service provider offers storage and other capabilities over the Internet. The cloud service may be shared with other parties (public), or it may be dedicated solely to the public safety organization (private). Because high-definition video files are large, they require a lot of bandwidth for file transmission and can accrue expensive video streaming charges. For this reason, cloud storage is sometimes a better fit for file backup or long-term storage rather than streaming or active use.
- Hybrid. This model uses a combination of on-premises and cloud-based services to give organizations flexibility and control in managing and storing video files. Organizations can outsource long-term stor¬age and backup to cloud-based services while keeping active and high-security cases on premises.
