The cocoa beans were harvested in a farm located in the town of Rionegro, Santander, Colombia (7° 15' 51" N, 73° 08' 58" W). These beans were extracted from cocoa pods within the same hectare and then fermented by a local farmer. Experts from the National Cocoa Federation in Colombia selected 30 samples for each of the three fermentation levels: (i) slightly fermented (LF), (ii) correctly fermented (CF), and (iii) highly fermented (HF). In total, 90 cocoa beans were selected.

Afterwards, the beans were evaluated in an optical laboratory and their spectral data were captured using the experimental setup described below. The environmental conditions in the laboratory included a temperature of 23 °C and a humidity of 60 %.

An optical assembly was built in our laboratory to capture hyperspectral images of the cocoa beans. The built testbed is shown in Figure 1. The scene (cocoa beans) was illuminated with a tunable light source (Oriel Instruments, TLS-300 XR) that decomposes the illumination from a halogen light source in its corresponding monochromatic wavelengths, with steps of two nanometers within the spectral range between 350 and 950 nm. Such monochromatic light is propagated through a bifurcated optical fiber (Illumination Technologies, 9145HT dual 6” light line) towards two lamps that illuminate the scene.

Figure 1. Figure 1. Top view of the testbed we used to obtain the spectral images Source: Authors’ own work.

A monochromatic sensor (AVT Stingray F-080B) captures the intensity of the light reflected by the cocoa beans (G_ref (λ)). Each obtained hyperspectral image exhibits 1032 x 776 pixels of spatial resolution (M x N) and 301 spectral bands (L). Also, a white scene (G_inc (λ)) was acquired to calibrate the hyperspectral information as. Six beans were organized for each scene, considering the focus and field of vision of the setup. In total, five data cubes were obtained for each of the three fermentation categories, which resulted in 15 spectral images, F ∈ R ^{M × N × L} , each with six beans of the same category (i.e., a total of 90 cocoa beans captured).

Figure 2 presents six of the 301 spectral bands of one random bean, which were acquired following the proposed setup. In addition, it spectral band includes the spectral range (VIS by visible or NIR by Near-infrared) and wavelength of each band in nanometers. We freely published the spectral images of the 90 cocoa beans on IEEE DataPort [40]. Figure 3 plots the mean spectral responses of the three fermentation categories. After the spectral information of the cocoa beans was obtained, the spectral images were processed.

Figure 2. Figure 2. Examples of some spectral bands captured by the optical assembly for each bean Source: Authors’ own work.

Figure 3. Figure 3. Mean spectral signatures of the 90 cocoa beans classified into three categories and their standard deviations: 30 slightly fermented beans (in green), 30 correctly fermented beans (in blue), and 30 highly fermented beans (in red) Source: Authors’ own work.

The first stage in HSI processing consists in identifying and extracting the pixels containing the information of interest, that is, the beans. Since the background of the images is black, a binary mask (D) that identifies the position of the beans was created using thresholding.

Additionally, morphological operations were employed. The values of the pixels in the binary image, D, are adjusted based on the value of other pixels in its neighborhood, such that the binary result ( D ‾ ∈ R ^M×N)is a given closed figure ( B ). The closing operator consists of a dilation operator and an erosion operator. The dilation operator, on the one hand, adds pixels to the boundaries of the objects in the image and is mathematically expressed as (1).

The erosion operator, on the other hand, removes the pixels in the boundaries of the objects and is expressed as (2).

In this study, following the shape of the beans, B is chosen as an oval shape of 200 x 300 pixels. Then, the result of the closing operation is given by (3).

Finally, the binarized image, D ‾ , is multiplied with the hyperspectral image, F ∈ R^{M × N × L}, of each sample to obtain the reflectance values of the cocoa beans as follows (4):

Where ∘ is the Hadamard product; and F ˜ , the HSI without background information. Figure 4 shows a HSI, F; its binary mask, D ‾ , which was obtained with (3); and the respective dataset, F ˜ , without background, which was calculated using (4).

Figure 4. Figure 4. (a) a HSI F, (b) its binary mask ,D_ and (c) the HSI F~ without background Source: Authors’ own work.

The second stage in the proposed framework involves extracting the classification features from F ˷ . Recent works have demonstrated that segmenting spectral information into spatial superpixels reduces the computational cost required by supervised classification methods and increases accuracy [36], [37], [38], [39]. Since traditional superpixel algorithms operate on three-band images (e.g., RGB images), we propose the following to extract the classification features: first, a three-band (RGB) image of the F ˜ hyperspectral data cube is extracted. Notice that F ˜ is acquired from the range between 350 and 950 nm. Then, from the F ˜ data cube, the spectral bands corresponding to 460, 530, and 670 nm are selected to form a three-band image (R,G,B). The three wavelengths represent the spectral response peaks of the blue, green, and red channels, as shown in Figure. 5.

Figure 5. Figure 5. Theoretical spectral responses of the red, green, and blue channels Source: Adapted from [41].

Afterwards, a segmentation algorithm is applied to the three-band image to find a superpixel map. For this purpose, we specifically use the well-known Simple Linear Iterative Clustering (SLIC) algorithm [35].

The SLIC algorithm works in the five-dimensional space, where the two coordinates (x, y) correspond to the spatial location of the superpixel and the other three components depict the RGB channels. The input variable of this algorithm is the number of desired superpixels (N_spx ). Given N_spx , where the approximate size of each superpixel is MN ⁄ N_spx , the SLIC algorithm defines a cluster center at every grid interval, S, as follows (5):

Hence, the algorithm chooses N_spx superpixel cluster centers, C_j = [r_j, g_j, b_j, x_j y_j ]^T , with j = [1, N_spx].

The SLIC algorithm assumes that the pixels associated with a cluster lie in a 2S × 2S area around the superpixel center on the (x, y) plane. Therefore, this is the pixel search area near each cluster center. The center is transferred to the lowest gradient position in a 3 × 3 neighborhood to prevent it from remaining on the edge of an object. In the next step, for each cluster center, the algorithm assigns the best-matching pixels from the search area according to the distance measure H_t defined as follows (6), (7), (8):

where H_c is the sum of the RGB distance and the xy plane distance normalized by the grid interval S; and r_j, g_j , and b_j denote the color of the j- th superpixel cluster center, while j' indexes each pixel, j' = [1, MN]. The value of m controls the compactness of a superpixel, which can be in the [1, 20] range. Usually, it is chosen as =10 [39]. In this paper, we used N_spx =500. Figure 6 is an RGB image segmented into superpixels with the SLIC algorithm and displayed with a false-color composite.

Figure 6. Figure 6. A cocoa bean RGB image segmented into 500 superpixels with the SLIC algorithm Source: Authors’ own work.

Finally, the spatial information of the superpixel map is matched with the hyperspectral information of the data cube F ˜ to calculate the average spectral signature of each superpixel and build an array with these spectral signatures, i.e., a matrix of size L×N_spx , as explained below.

Let F ˜ ∈ R ^{L × MN} be the unfolded matrix of the hyperspectral image ( F ˜ ) reorganized as F ˜ =[ f ˜ ₍₁₎,..., f ˜ _(MN)], where f ˜ _(k) ∈ R ^L represents the spectral signature of the k-th pixel.

Mathematically, the matrix with the average spectral signatures Y ∈ R ^L×Nspx is created as (9):

where H ∈ R ^MN×Nspx is an average sorting matrix, in which the different values of zero for each column (ℓ) have a weight 1 ⁄ ( N ^ℓ _s ), such that N ^ℓ _s is the number of pixels grouped in a superpixel. Then, the matrix Y= [y ₁,…,y _Nspx] of spectral signatures will be classified as explained in the next subsection.

The last step in the proposed framework is to classify the spectral signatures of each superpixel in Y using the Support Vector Machine (SVM) algorithm.

For this purpose, we denote the information of n spectral signatures used in the training step as Θ= {y₁,…,y_n }, and their respective class labels as Ω={ω₁, ω₂, ω₃ }, where ω₁, ω₂, ω₃ ∈ R ^n/3 represent the slight, correctly, and high fermentation levels, respectively.

In this subsection, we evaluate the classification proposed in (12). The objective is to label each bean with its predominant fermentation category. Table 2 lists some parameters of this classification.

Table 2 Table 2 Classification parameters

Number of cocoa beans	90
Superpixels per bean	500
Beans per category	30
Spectral superpixels per category	15000
Spectral superpixels in total	45000
Percentage of spectral signatures used for training the SVM classifier	15 %

Source: Authors’ own work.

In total, we classified 90 cocoa beans, each one spatially divided into 500 regions (superpixels); that is, we classified a total of 45,000 superpixels. Let us remember that each superpixel has an associated spectral signature consisting of 301 reflectance values measured between 350 and 950 nm in steps of 2 nm. For this classification, we used 15 % of the samples to train the SVM classifier. Figure 7 shows the result of classifying one of the 90 cocoa beans using the proposed method. The framework assigns a label to the spectral signature of each superpixel and, subsequently, assigns the predominant label to each bean.

Figure 7. Figure 7. Visual results of the classification by superpixels (top) and predominant label (bottom) Source: Authors’ own work.

As mentioned in Subsection 2.1, we also had the reference label of each bean, which was carefully assigned by a professional technical team. Therefore, after assigning a label to each bean, this method evaluated the precision of the classification proposed in this paper by comparing the results with the reference labels.

The confusion matrix in Table 3 details the number of beans correctly and incorrectly labeled with the proposed method. Each column in the matrix represents the number of predictions of each class, while each row represents the ground truth.

Table 3 Table 3 Confusion matrix of all the beans classified based on the predominant label (Equation 12)

	Slight	Correct	High
Slight	30	0	0
Correct	0	30	0
High	0	0	30

Source: Authors’ own work.

From the results in the confusion matrix, we calculated the values of recall, precision, truth overall, and overall classification per class as in [44]. These results are shown in Table 4.

Table 4 Table 4 Classification metrics from the confusion matrix

Truth overall class Slight	30	Recall class Slight	100 %
Truth overall class Correct	30	Recall class Correct	100 %
Truth overall class High	30	Recall class High	100 %
Classification overall class Slight	30	Precision class Slight	100 %
Classification overall class Correct	30	Precision class Correct	100 %
Classification overall class High	30	Precision class High	100 %
Overall Accuracy		100 %

Source: Authors’ own work.

Note that all Slightly, Correctly, and Highly fermented cocoa beans were correctly labeled by the classifier (Table 3). In addition, the metrics in Table 4 show the high performance of the proposed classification approach. The overall precision of assigning the predominant label to 90 cocoa beans was 100 %.

Entities that regulate the international marketing of cocoa beans, such as The Federation of Cocoa Commerce London (FCC), require the quality of the seeds in each ton to be acceptably uniform. The objective is to guarantee the homogeneity of the products derived from cocoa.

An approach such as the one proposed here can be used to analyze the uniformity of cocoa loads. It was shown that the 90 beans used in this study were correctly classified, 30 into each one of the fermentation categories. However, note that the initial labeling of the bean (bottom, Figure 7) disregards the fact that some superpixels can be classified into other fermentation levels different from the predominant one (top, Figure 7). Therefore, classifying each superpixel in a grain separately, we can analyze how uniform each bean is.

In this subsection, we discuss a more detailed classification in which bean fermentation uniformity is calculated. This type of analysis has not yet been used commercially in the cocoa industry. Furthermore, it is rarely used in general food quality control, where uniformity is assessed at the collective level (batch uniformity).

Table 5 shows the result of classifying the average spectral signature of each one of the 45,000 superpixels. We used the general bean category as the reference label for each superpixel in the confusion matrix. Note that, compared to Table 3, the matrix in Table 5 is not diagonal. It is understood that the matrix is not diagonal because there may be regions (superpixels) with different degrees of fermentation despite being on the same grain.

Table 5 Table 5 Confusion matrix of all the classified spectral superpixels (Eq 11)

	Slight	Correct	High
Slight	12,915	765	1,320
Correct	435	12,360	2,205
High	210	2,610	12,180

Source: Authors’ own work.

Table 6 shows the percentage of superpixel classification in each class. The rows represent the actual labels (full-grain categories); therefore, the sum of the values in each row is equal to 100 %. In turn, the columns show the tags assigned by the classifier to the superpixels. The last row, Total, shows the percentage of superpixels labeled by the model in each fermentation level. Note that, although there are 30 beans in each category, regions of correct and high fermentation predominate, with 34.96 % and 34.89 % of the areas, respectively.

Table 6 Table 6 Confusion matrix of all the classified superpixels in percentages

	Slight	Correct	High
Slight	86.1 %	5.1 %	8.8 %
Correct	2.9 %	82.4 %	14.7 %
High	1.4 %	17.4 %	81.2 %
Total	30.13 %	34.96 %	34.89 %

Source: Authors’ own work

Furthermore, Table 6 shows that, in highly fermented beans, there is a considerable amount of material that is well fermented (17.4 %), and vice versa (14.7 %). In comparison, only small portions of the beans with high and correct fermentation have slight fermentation (1.4 % and 2.9 %, respectively). Regarding slightly fermented beans, there are small burned regions (high, 8.8 %) and a minority of well-fermented areas (5.1 %).

This analysis could contribute to the implementation of fermentation techniques that produce a more homogeneous drying.

In this subsection, we vary the number of spectral signatures per category used in the SVM classifier training in order to analyze its influence on the precision of the classification and determine how many spectral signatures are necessary to obtain acceptable results.

The training was programmed to randomly select a certain percentage of spectral samples, the same number of signatures from each category. The system was trained, and the beans were classified using the predominant label method. Specifically, the percentage of training samples was varied between 1 % and 20 % of the total signatures (45,000), as seen on the x-axis in Figure 8. A 10 % of each resulting training set was used as validation to tune the model.

Figure 8. Figure 8. Overall classification accuracy as a function of the percentage of training samples Figure 8 shows the average accuracy of the test set over ten experiments. Evidently, as the number of training samples increases, the classifier improves its precision. Source: Authors’ own work.

Figure 8 shows the average accuracy of the test set over ten experiments. Evidently, as the number of training samples increases, the classifier improves its precision.

Note that the class most easily recognized by the classifier is Slight fermentation, followed by Correct and High, with any proportion of training samples.

Specifically, using only 2.5 % of the training samples (which is equivalent to 1,125 spectral signatures), the system correctly labeled 90 % of the cocoa beans as Slight fermentation, i.e., 27 of the 30 beans in this class. Likewise, with the same training percentage, the classifier accurately detected 76.6 % of the beans with Correct fermentation (23 beans), and 63.3 % of those with High fermentation (19 beans).

Furthermore, in Figure 8, when the percentage of training samples exceeds 12.5 %, the overall accuracy of the classification is quite acceptable. In fact, let us remember that the results in Subsections 3.1. and 3.2. were obtained from classifications performed with 15 % of the training samples.