Green and blue spaces are vital natural environments that offer physical and mental health benefits. Accurately identifying and quantifying the health-promoting natural elements in these spaces is crucial for advancing nature-based health interventions. Given the growing use of nature videos to simulate and enhance the health-promoting effects of exposure to green and blue spaces, developing reliable methods to analyze these scenes is increasingly important. However, traditional color-based methods have demonstrated limited effectiveness, while pre-trained models struggle to segment green and blue spaces in video data. To address these limitations, this paper introduced a video-based green and blue spaces (VGBs) dataset for semantic segmentation (SS). The dataset comprised 2,333 images from 89 videos, annotated with four labels: vegetation, water, sky, and other. We evaluated the performance of different segmentation models and conducted a comparative analysis between models trained on VGBs and pretrained models. Experimental results indicated that DeepLabV3 trained on VGBs achieved superior segmentation accuracy for vegetation, sky, and other elements with per-class Intersection over Union (IoU) values of 92.51%, 92.12%, 87.41% respectively, while highlighting the need for improvement in water (per-class IoU = 75.14%) segmentation. This dataset provides a foundation for developing and benchmarking advanced algorithms for segmenting green and blue spaces videos. The publicly available deep learning (DL) models trained on VGBs support video-based applications, including real-time landscape analysis, nature exposure assessment, and virtual nature generating.