The color scheme of the building façades is a critical visual impression of the city’s context and characteristics. Analyzing the building colors can provide a way to get an objective and holistic view of the city and a foundation to enact regulations and ordinances on the building color and texture. Traditional color analytical processes are usually carried out by manual survey and sampling, which can take a lot of time and be high-cost in large-scale urban areas, and are limited by the number of color samples and accidental factors of the environment. To this end, we proposed an automated color sampling and analysis method by using street view images as the data source. The images were semantically segmented to extract the façade areas with a convolutional neural network. The dominant colors of the building façades were then identified and transformed to the Munsell color system for further analysis and comparison between different districts. The research can provide a repeatable and objective urban building color analysis method and a low-cost, high-efficiency tool for urban color surveys.