Owing to the massive amount of video data being generated as a result of high proliferation of surveillance cameras, the manpower to monitor such system is relatively expensive. Passively monitoring surveillance video however, incapacitates the usefulness of surveillance camera. Therefore, a drive to monitor events as they happen is expedient to fully harness the massive data generated by surveillance cameras. This is the main goal of video analytics. In this chapter, we extend the notion of surveillance. Surveillance refers not only to monitoring for security or safety purposes but encapsulates all aspects of monitoring to capture the dynamics of different application domains including retail, transportation, service industries and healthcare. This chapter presents a detailed survey of video analytics as well as its application. We present advances in video analytics research and emerging trends from subdomains such as behavior analysis, moving object classification, video summarization, object detection, object tracking, congestion analysis, abnormality detection and information fusion from multiple cameras. We also summarize recent development in video analytics and intelligent video systems (IVS). We evaluated the state-of-the-art approach to video analytics including deep learning approach and outlined research direction with emphasis on algorithm-based analytics and applications. Hardware-related issues are excluded from this chapter.