Deep-learning-ready RGB-depth images of seedling development

Abstract

In the era of machine learning-driven plant imaging, the production of annotated datasets is a very important contribution. In this data paper, a unique annotated dataset of seedling emergence kinetics is proposed. It is composed of almost 70,000 RGB-depth frames and more than 700,000 plant annotations. The dataset is shown valuable for training deep learning models and performing high-throughput phenotyping by imaging. The ability of such models to generalize to several species and outperform the state-of-the-art owing to the delivered dataset is demonstrated. We also discuss how this dataset raises new questions in plant phenotyping.

Background

Plants continuously metamorphose throughout their development, with specific stages occurring in a predetermined order. This paper introduces a dataset on key seedling development stages from germination to first leaf formation. During this period, plants must adapt to their environment for successful photosynthesis and growth. Seedlings undergo photomorphogenesis upon emerging from the soil, which involves reduced hypocotyl growth, cotyledon opening, photosynthesis initiation, and meristem activation. These processes require extensive genomic reprogramming and are challenging to study owing to seedling population variability. Understanding seedling development is essential for determining crop yield, as uneven emergence can lead to lower yields and poor farmer acceptance. More precisely, the kinetics of seedling development is a measure of seed vigor which is an important agricultural trait defined by (i) rapid and uniform germination and (ii) seedling growth, that is determined by physiological and sanitary quality [1]. While germination kinetics can be monitored in soilless systems [2], measuring seedling growth under real growth conditions (i.e. with a culture substrate) is not yet possible. The method proposed in this data paper enables the measurement of seedling growth in a high throughput manner with fine temporal resolution. This approach could be used to assess the vigor of seed lots from different genotypes or varieties, under both suboptimal and optimal conditions (e.g. temperature, humidity) and during exposure with a diversity of microorganisms.

High-throughput kinetic monitoring of seedling growth has been performed on RGB-depth images in recent studies of Samiei et al. [3], Garbouge et al. [4], and Couasnet et al. [5]. Color imaging was first successfully coupled with deep learning models by Samiei et al. [3] to monitorMedicago. The added value of fusing RGB with depth was introduced by Garbouge et al. [4] to monitor several varieties of beans. To reproduce such experiments, a software was developed and provided to the scientific community by Couasnet et al. [5]. In this data paper, we provide a dataset of RGB depth monitoring data of seedling kinetics extended to more plant species. This dataset is annotated and functional for deep learning use, i.e., it is reusable for scientists who are eager to train new deep learning architectures.

We demonstrate the value of the delivered dataset by training a new deep learning model that shows good generalization performance on various species in comparison with the types of models optimized for single species by Samiei et al. [3], Garbouge et al. [4], and Couasnet et al. [5]. Finally, we discuss the perspectives of the available research owing to the accessibility of this data set.

Construction and content

Room and equipment

Data acquisition was performed via eight Intel RealSense D435 cameras, strategically positioned above two tables on either side of the room. These cameras were interconnected with a Raspberry Pi 3 model B, referred to as the clients, where RGB, depth, and infrared images were temporarily stored. At hourly intervals, each of the eight clients transmitted the images via a local Wi-Fi network to a ninth Raspberry Pi 3 model B, identical to the others, designated as the server, which was equipped with a 4TB external hard drive. After image transmission, the sizes of the original images were compared with those of the transmitted images, and if they matched, the original images were deleted from the clients. A system monitoring script on the server regularly verified the integrity of data acquisition, and in the event of any issues, a warning email was sent to maintenance. Each client was also equipped with Power over Ethernet (PoE), enabling the Raspberry Pis to be powered via Ethernet cables, and a 64 GB Samsung microSDXC card, allowing for continued data acquisition during network disconnection (with a capacity of several days). Potential power interruptions were mitigated by the presence of an uninterruptible power supply (UPS) capable of providing backup power for 30 min to 1 h. Last, the clients and server were interconnected on the same local area network, with Wi-Fi connectivity for the server and wired connections for the clients, linked to a TP-Link T1500-28PCT switch and administered by a TP-Link Archer C9 router. The hardware components necessary for installing an 8-camera acquisition system in a controlled environment are listed in Table 1.

Table 1 List of equipment used to equip the growth chamber

The controlled environment growth chamber was equipped with eight Intel RealSense D435 cameras, periodically positioned across two rows on either side of the room. The cameras were not always operational simultaneously, and the number and format of germination trays positioned beneath each camera varied. Two types of trays were used: 40-pot trays (5 columns $\times$ 8 rows) and 84-pot trays (7 columns $\times$ 12 rows). A detailed description of each experiment is provided in the subsequent section. To provide explicit insight into the data acquisition environment, a map of the experiment, including the corresponding camera designations, is presented in Fig. 1.

Fig. 1

Detailed visualisation of the experimental setup used to produce the delivered dataset. A RGB-Depth Intel Realsense D435 camera (top left). Cameras positioned above the plants for top view monitoring (top right). The cameras are labeled with a numerical identifier ranging from 3 to 10. The chamber map is shown at the bottom left

Although images are acquired every 15-minutes, the acquisition scripts are designed to activate the camera sensors 5 min before acquisition. The images are subsequently taken and saved, and the sensors are then deactivated for approximately 10 min before the next acquisition cycle. A detailed explanation of this process can be found in Garbouge et al. [4]. In the meanwhile, a data transfer script exports the images to an external hard drive connected to the server. To compensate for data loss resulting from network interruptions, power failures, or hardware malfunctions, we have developed a script to detect troubleshooting in the arrival of data on the server. This script verifies that images are received by the server at hourly intervals (with an additional check performed an hour later if necessary) and that the image size exceeds a predetermined threshold (corresponding to an empty image of identical size and format). If any of these conditions are not met, an alert is triggered, and an email is sent detailing the problem(s) identified and the hardware concerned. Additionally, a Raspberry Pi installed in the offices performs hourly checks to ensure that the room’s server is accessible. This supplementary security measure prevents emails alert system failures in the event of an internet access interruption. Despite the implementation of these security systems (UPS, acquisition verification script, and server accessibility verification), we were unable to cope with power interruptions exceeding 1 h or system shutdowns occurring during nighttime or public holidays. Consequently, image acquisition may have been interrupted on several occasions, resulting in potential gaps in the datasets provided. Nevertheless, it is important to note that the acquisition time remains accurate. Therefore, spurious missing data do not affect the timeline. The data loss is estimated to be around $\%$ which is considered negligible. We assume that the missing data are uniformly distributed over the time and do not interfere with the biological process being monitored.

Intel RealSense D435 cameras include RGB camera, two infrared (IR) cameras, and an IR projector. The product data sheet [6] provides a detailed explanation of how the depth image is calculated from the stereo IR sensor. The IR projector enhances image quality on surfaces with minimal texture. The pixel values in the depth image represent distances to the camera in millimeters. Notably, the depth field exceeds the color field in terms of spatial extent. The raw resolution of the images is 1920 $\times$ 1080 pixels for RGB and 1280 $\times$ 720 pixels for depth. To facilitate alignment of the RGB and depth images, the depth image undergoes cropping and resizing via nearest neighbor interpolation. This transformation is performed via the PyRealSense2 package associated with the camera, which relies on intrinsic camera parameters. To provide readily usable data, the dataset includes the RGB full frame and the aligned depth frame. The concatenation of these four channels is referred to as an RGB-depth full frame. As illustrated in Fig. 2, the RGB full frame presents a top view of the tray, enabling visualization of all the plants with their distance-to-camera values linked to their height.

Fig. 2

Cropped RGB full frame (left) and Depth aligned frame (right). The images are cropped to exclude distant objects and emphasize depth dynamics

Data description

The available database comprises multiple sequences of RGB-depth full frames, collectively referred to as full time lapses. In contrast, we define pots time lapses as the cropped pot regions extracted from full time lapses. Throughout the time lapse sequences, we observed the growth and development of the plants, which can be described by biologists in distinct developmental stages. Four main stages of development can be observed in the pots time lapses. As illustrated in Fig. 3, this inclues the soil (when the seedling is still fully in the soil), first appearance of the cotyledons, opening of the cotyledons and appearance of the first leaf. The dataset includes annotations for a set of dicotyledon species, which were manually obtained through collaborative efforts with biologists.

Fig. 3

RGB images extracted from a pots time-lapse at each stage

The datasets presented in this article include full time lapses of various plant species, including rapeseed, tomatoes, and beans, acquired during the monitoring of 11 distinct trials conducted throughout 2022. A comprehensive list of these 11 trials, accompanied by relevant information (camera specifications, tray configurations, plant varieties, germination pot details, start and end dates, and image acquisition quantities), is provided in the Appendix section. Phenotyping was performed on beans, rapeseed, and tomato species. A summary of the phenotyped plants per species is presented in Table 2.

Table 2 Synthesis of the full time-lapse and RGB-Depth full frame quantity per species

The dataset is publicly accessible in the DATA INRAE repository, DOI: https://doiorg.publicaciones.saludcastillayleon.es/10.57745/AMFJTK. The file tree structure is illustrated in Fig. 4. The dataset is organized into 11 compressed .zip files, each corresponding to a distinct trial. Within these files, images are sorted chronologically by acquisition start date, then by camera, and stored in .png format within dedicated color and depth folders. Labels are also stored in .zip files, sorted by trial and camera, with one file per plant. The corresponding .xlsx tables provide a summary of the image and label contents. General information about the dataset is provided in the Readme.txt file.

Fig. 4

Data repository tree and comments available in the DATA INRAE repository, DOI: https://doiorg.publicaciones.saludcastillayleon.es/10.57745/AMFJTK

We now position the provided dataset with the related literature. There are currently many publicly available plant datasets, as reviewed by Kurtser and Lowry [7], Das Choudhury et al. [8], Lu and Young [9] and Cao et al. [10]. In this study, we provide a multispecies, indoor, multimodal RGB-depth time-lapse (15-minute frame rate) dataset annotated for the classification of seedling growth stages from a top view. The main features of the most related datasets and our dataset are provided in Table 3. In Cruz et al. [11], authors operated with two species (including beans as in our dataset) viewed from the top view in time lapse (1-hour frame rate) of RGB-depth images. This dataset also includes fluorescence and infrared but provides data for only 21 plants. Additionally, [12] operated with two species from a top view in time lapses acquired at a frame rate of 20 min. This dataset includes only RGB data and is limited to 147 plants. Similarly, [13] provides RGB top view images. Only one species is studied and 200 plants are phenotyped with a very low frame rate (18 days). The dataset is designed to perform vegetative and reproductive stage classification. Finally [14], provided RGB-depth from the top view, with acquisition every 4 h. This dataset is dedicated to leaf segmentation; however, it contains only 5 plants and is limited to only one type of plant, Komatsuna.¹

The number of phenotyped plants is crucial for capturing the biological diversity. We achieved a dataset size comparable in magnitude to the largest dataset reported in the literature. This enhances the robustness of the knowledge to be extracted from this dataset. The largest data sets reported by Gené-Mola et al. [15], Lac et al. [16], and Genze et al. [17] do not involve time-lapse imaging, and/or rely solely on RGB data. This strongly differs from our multimodal, time-resolved approach which appears as an original dataset in comparison with the most related publicly available datasets.

Table 3 Comparison of key characteristics among datasets from previous most related studies and our dataset

Utility and discussion

We now detail the utility of the delivered dataset and discuss further developments opened with this dataset.

Pot scale classification

Before processing, the Pots frames undergo normalization. The RGB channels, which are represented as 8-bit integer data, are divided by 255. The depth channel contains missing values, which are filled by inpainting via anisotropic diffusion. A Laplacian filter is applied iteratively, and through diffusion, missing values are progressively replaced. The depth pixels, represented as 16-bit integers, are divided by 65,535. This normalization between 0 and 1 preserves precision. The RGB and depth ranges are the same to facilitate weight optimization during deep learning training.

The software GrowthData [5] was provided with a model trained on images of Bean. With the dataset provided in this article, we trained a multi-species model. The annotated data set was split into training, validation, and testing sets, with the following ratio 80%, 10%, and 10%, respectively. Because of the similarity between successive pot frames, a pot time lapse lies entirely within one of the three subsets. Detailed information regarding the number of Pots time lapses and frames is provided in Table 4.

Table 4 Details of the data split per species in terms of Pots time-lapses and Pots frame

The multispecies model (MS model) was trained on the entire training dataset. To increase the variability in the training set, classical data augmentation techniques, including rotation ($90^{\circ }$, $180^{\circ }$, or $270^{\circ }$), and vertical and horizontal symmetry, were employed. For comparison, we trained the Rp model via rapeseed data, the T model via tomato data, and the B model via Bean data. All the models utilized the architecture of the original convolutional neural network (CNN) of Garbouge et al. [4] presented in Fig. 5. The model includes four convolution blocks with 32, 64, 128, and 128 filters; a fully connected layer with 256 neurons; a dropout with a drop rate of 0.5; and a final fully connected layer with four outputs, corresponding to the number of classes. Each convolution block performs $3\times 3$ pixel filtering, activation, and $2\times 2$ pixel max pooling. Following convolution and a fully connected layer, all the activations were rectified linear units (ReLUs), except for the last one, which was a Softmax function.

Fig. 5

Convolutional neural network used as a baseline to perform growth stage classification from RGB depth images. Orange, red, and blue blocks represent convolution, maximum pooling, and fully connected layers, respectively

The CNN we designed serves as a simple baseline approach. It predicts the class of each frame in the time-lapse independently, without considering the order in which the frames occur. Given a training set of N pairs of images $x_i$ and labels $y_i$, we trained the parameters $\theta$ of the network f using stochastic gradient descent to minimize empirical risk:

$$\begin{aligned} \theta ^* = \underset{\theta }{\operatorname {argmin}} \sum _{i=1}^N \mathcal {L}(y_i,f(x_i,\theta ))\,, \end{aligned}$$

(1)

where $\mathcal {L}$ denotes the loss function chosen as the categorical cross entropy. This loss function is defined in Demirkaya et al. [21] as

$$\begin{aligned} \mathcal {L}(y_i,f(x_i,\theta )) = - \sum _{k=1}^4 \delta (y_i-k) \log (f_k(x_i,\theta ))\,, \end{aligned}$$

(2)

where k is the class index, $\delta (y_i-k)$ is equal to 1 if $y_i=k$ and 0 otherwise and $f_k(x_i,\theta )$ denotes the probability that $x_i$ belongs to class k. The gradient descent aims the minimization using the Adam optimizer algorithm introduced in Kingma and Ba [22]. We adopted an adaptive learning rate. If the validation loss does not improve for 10 consecutive epochs, the learning rate is halved. For training the T, B, and Rp models, the initial learning rate is $6.10^{-5}$, while for training the MS model, it is $3.10^{-5}$. The batch size is 2048 for training the MS, T, and B models, and 4096 for the Rp model.

The class frequency in the training set depends on the stage duration. Images within a time series are classified independently, which may not accurately predict the evolution of growth stages. To address this limitation, predictions are post-processed with a smoothing filter based on a sliding window computing a majority voting by finding the median of classes

$$\begin{aligned} s = \lfloor * \rfloor \Bigg \{ \Bigg ( \frac{n+1}{2} \Bigg )^{th} \Bigg \}\,, \end{aligned}$$

(3)

where s represent the smoothed predictions and n is the window’s length (taken as 5 in this study). Then, smoothed class is forced to be uniform between the first and last appearance of a stage.

We evaluated the raw predictions via loss and accuracy (Rw-Acc). Additionally, we evaluated the corrected predictions in terms of accuracy over rapeseed data (Rp-Acc), tomato data (T-Acc), bean data (B-Acc), and overall data (O-Acc). The training process was repeated 10 times, and we computed the average performance. The results are presented in Table 5. The indicated uncertainty represents the 95% confidence interval computed from the standard deviation under the Gaussian hypothesis.

Table 5 Average performance of the model trained with different species in terms of loss and accuracy

With respect to accuracy per species, each test set is best predicted by its respective specific model. The Rp model achieves an accuracy of 89.2% for rapeseed, the T model 87.9% for tomatoes, and the B model 94.8% for beans. The application of mono-species models to other species yields unsatisfactory results. The most notable example of transferability is the B model on tomato data, with an accuracy of 68.5%. In other cases, the accuracy does not exceed 57.2%. Across all species, the MS model reached an overall accuracy of 88.7%. The MS model performs slightly less accurately for tomatoes, with an accuracy of 88.3%, and more accurately for beans, with an accuracy of 89.7%. Although the global performance of the MS model is lower than that of each mono-species model, this represents a gain in generality. By acquiring a large, mixed-species dataset, we developed a model capable of processing multiple species with a comparable level of accuracy. This demonstrates the added value of the dataset delivered.

Exploring research trends

Several other uses of the delivered dataset can be envisioned, as detailed in this section. First, one can seek to outperform the baseline we provided in the previous section for the classification of seedling development stages. The way we perform individual pot frame classification with a memoryless convolutional neural network can be seen as a brute-force approach since it does not consider any prior information on plant development. There is, of course, a sequential causal aspect in plant growth, where each stage is intrinsically dependent on the preceding stages. An alternative neural network architecture that would incorporate this prior knowledge would be expected to provide higher performance than the architecture delivered with this data paper. One could consider recurrent neural networks (RNNs) or long short-term memory (LSTM) networks to model temporal dependencies. However, as demonstrated by Garbouge et al. [23], this introduces a dependency on the speed of plant growth and the frame rate of data capture. Faster-growing plants or higher frame rate data may skew the learning algorithms, introducing inaccuracies in the phenotypic analysis. Another approach would be to use ordinal loss functions, as proposed by Cao et al. [24], and Shi et al. [25]. The ordinal loss functions assign different penalties to errors based on their ordinal relationships, thus maintaining some degree of temporal causality. Such new investigations are now accessible to the reader via the dataset delivered in this article.

The dataset was produced with periodic sampling. However, one could be interested in acquiring data at non-periodic sampling rate. Different existing approaches [26, 27] for non-periodic sampling strategies could be tested indeed to reduce the cost of data storage [28, 29]. While the real interest would be to perform such non periodic sampling on-the-fly, it is possible to simulate and investigate non-periodic sampling strategies via sub-sampling with the dataset provided thanks to the relatively high frame rate used.

Another limitation of classifying plant growth at the pot scale is the issue of overlapping plants. As the duration of data acquisition increased, the size of the seedlings exceeded the size of the pots, and occlusions of the plants appeared. Also, such occlusions would be likely to appear in the study of mixture of species which is important in agroecology. Advanced techniques for instantaneous leaf segmentation have shown promise in initiating occlusion detection [30, 31]. Occlusion detection enables the system to either stop processing or adopt a different processing method, ensuring classification performance. To extend the processing time, some methods exploit the circadian cycle to capture unobstructed frames [32] without any tracking method. To allow further investigation, including other tasks than classification (including tracking, segmentation, object detection, ...), we provide the full frames in the dataset.

Moreover, since we have described how to reproduce the data acquisition process in this article, we allow the reader to reproduce similar experiments with other species. One open question would then be how to adapt the already trained model with our dataset to new species. The dataset comprises three species. This already allows researchers to simulate such situations. For example, one can explore model adaptation techniques (domain adaptation, fine-tuning, transfer learning).

Finally, the provided dataset can serve any deep learning methodological task. To point a few, this could include neural architecture search [33], distillation knowledge to propose deep learning models that could run on the edge [34], i.e. on the minicomputer of the imaging system. One could also think of reducing the complexity of the dataset via data distillation [35] or the use of hybrid agronomical models that mixes mechanistic and machine learning approaches [36, 37]. As a last interesting perspective opened with the provided dataset, one could think of using the RGB-Depth images to serve other common plant imaging modalities (thermography, chlorophyll fluorescence, spectral imaging, ...) via style transfer [38, 39].

Conclusion

The dataset proposed in this data paper provides, to the best of our knowledge, the largest annotated dataset (70,000 frames and 700,000 annotations) on multispecies seedling development when viewed indoors via RGB-depth in time-lapse from a top view. The dataset has shown interest by comparison with the most related works and opens new perspectives to scientific communities, either for plant biologists interested in using this dataset for their own species or for computer vision experts interested in developing new computer vision tools for plant phenotyping.

Appendix

The appendix contains 5 tables listing the trials. They describe the cameras, the stages used and the number of images. See Tables 6, 7, 8, 9, and 10.

Table 6 Synthesis of trial information - part 1

Table 7 Synthesis of trial information - part 2

Table 8 Synthesis of trial information - part 3

Table 9 Synthesis of trial information - part 4

Table 10 Synthesis of trial information - part 5