Recent advances in visual sensing technology and unmanned aerial vehicle (UAV) provide an effective tool to capture the as-is conditions of infrastructure and thus have gained their popularity in infrastructure inspection and documentation. To facilitate this process, several recent studies have proposed automated methods for detecting concrete cracks from UAV-based images. This study is aimed at proposing a new method for automating both the detection and quantification of concrete cracks. The proposed method advances state-of-the-art image-based concrete crack detection methods by integrating the RGB images with the lidar data collected by UAV. The key innovations focus on two aspects: (1) by recognizing objects of interest in the lidar data, regions of interest can be extracted automatically from the images; (2) by retrieving the depth information through the lidar data, the actual pixel sizes can be estimated to facilitate both the detection and quantification of concrete cracks. In order to validate the proposed method, a customized UAV platform that was equipped with a high-resolution camera and a Velodyne VLP-16 lidar scanner was developed to scan the substructure elements of an in-service bridge where multiple concrete cracks can be observed. The effectiveness of the proposed method in recognizing and quantifying concrete cracks is validated quantitively against manually annotated images and physical measurements. The results indicate that the proposed approach can recognize crack pixels with an accuracy of 85% on average as well as quantify the recognized concrete cracks with an error less than 10%.