Data Science Institute Supports UA Grad Research

Processing and Analyzing Aerial as well as Ground Images of Different Crops in order to Estimate their Different Agricultural Phenotypes

The Transportation Energy Resources from Renewable Agriculture Phenotyping Reference Platform (TERRA-REF https://terraref.org(link is external)) aims to transform plant breeding by using remote sensing to quantify plant traits such as plant architecture, carbon uptake, tissue chemistry, water use, and other features to predict the yield potential and stress resistance of crops such as sorghum and lettuce.

Recent achievements on the TERRA-REF project, specifically in preprocessing and stitching the raw image patches coming from the Gantry machine that scans the crop fields, have been extremely useful. Ultimately, in collaboration with plant scientists, we will use the clean data to analyze different phenotypes of different variations of crops in order to measure the efficiencies of different gene modifications.

I have been working with the image data coming from the Gantry machine for the past few months. The first problem that we have been trying to tackle was to stitch the patches coming from the machine. These patches are associated with GPS coordinates but since GPS is noisy, there are lots of misalignments in the final Orthomosaic image. We have tackled this problem by incorporating Visual Features to correct the misalignments. We basically used the GPS coordinates to find the overlapped regions for each two given patches. We then used the Scale-Invariant feature transform (SIFT) method to detect important feature points. SIFT uses edges and corners in different scales to detect key points in an image. These feature points are matched to key points from another image based on similarity of features. Then a transformation between the two images can be estimated in order to place them accurately and merge them into a new image. This transformation can be directly estimated or we can use the RANSAC method to estimate its parameters. In RANSAC, we randomly pick minimum number of matches points needed for estimating a basic transformation, then we see how many of other matched points are considered inlier using this transformation. We keep doing this until we reach a satisfying confidence level. Using RANSAC and affine transformations, we found the best transformation between the two given patches and using that we stitched them together. The results of our method are highly satisfactory and we will step forward in the following weeks to start working on the other problems in this project which includes identifying individual crops and classify different phenotypes of them. The following figures compares the result of our method with the alternative method in which only GPS coordinates are used to stitch the patches.

Figure 1 Stitched images of TERRA-REF lettuce field

My research focuses on machine learning applications in astronomical and planetary sciences. Simply put, we try to infer patterns from data which help enhance our understanding of the universe.

I work with researchers in the Department of Astronomy and Steward Observatory on predicting types of astronomical events known as transients. Transients are a class of astronomical events that happen relatively quickly (thus the name transient). Supernovae, or the explosions marking the end of a star’s life, are some of the best-known transients. However, there are many different types of supernovae and transient events in general, which happen with different rates of rarity and provide answers to different scientific questions. A huge telescope being set up in Chile right now, the Large Synoptic Survey Telescope (LSST), is anticipated to alert on 10 million transient events a night – but although it knows something happened, it does not know what. Since 10 million events is far too many for us to let astronomers look at each one in detail, we use data about the galaxies in which the events reside in order to predict what type of event an alert most likely corresponds to. We do this using a machine learning technique known as kernel density estimation – which helps us infer the patterns existing in the data that correspond to each type of event. When the LSST hits, we can tell astronomers right away, with a certain measure of confidence, what type of event we think an alert corresponds to based on the galaxy in which it resides. This will allow for exceptionally important or rare events to be observed in detail and advance our astronomical understanding of them before they fade away.

I also work with researchers in Lunar and Planetary Laboratory on predicting the mineral makeup on the surface of Mars, in anticipation of the Mars 2020 rover which will be collecting samples from the surface in about a year. Although a particular landing site has been selected (Jezero crater, pictured in Figure 1) there are still too many grains of sand over the 3-12 miles of terrain the rover is projected to travel on for it to manually inspect and identify which samples are most indicative of life on Mars. To solve this problem, we use images of the surface of Mars, taken by a satellite (the Mars Reconnaissance Orbiter, MRO) revolving around the planet. Unlike our everyday cameras which are limited to visible light, the instrument aboard the MRO captures images in ranges of wavelengths extending to infrared light, for each pixel. The resulting measurements are known as a spectra, which are unique to different materials and let us infer the material makeup of the surface. We consider both the spectra and the proximity of different regions of the crater to one another in what we call a spectral-spatial machine learning model in order to infer the mineral makeup of the surface. When we are done, we can provide the scientific community the anticipated mineral makeup on each part of the surface, with a certain level of confidence. This projected mineral makeup can then be used by the Mars 2020 rover to prioritize areas for sample collection which are most able to answer questions about the potential for life on Mars.

Figure 1 Jezero crater as imaged by a high-resolution camera aboard the Mars Reconnaissance Orbiter (MRO). The crater is the circular shape in the middle, and was selected as the landing site for the next Mars rover, Mars 2020, which is scheduled to launch this summer and  land here on February 18^th, 2021. Our work aims to map out the mineral makeup of the crater and surrounding areas, in order to help the rover determine the regions corresponding to the most promising signs of previous microbial life. Mars 2020 will then collect rock and soil samples from those areas and store them somewhere on the surface, where they will be eventually collected and brought back to Earth for further analysis. 

The first step in the detection of valley fever by exploring chest X-ray images via deep learning

Figure 1. Examples of a pulmonary nodule (lung lesion) and cardiomegaly. The area encompassing the abnormalities is shown in red overlay.

Valley fever (coccidioidomycosis) is a fungal infection that has been endemic in the southwestern areas of the United States for hundreds of years. Valley fever causes pneumonia-like symptoms and may be disseminated leading to other systemic infections that may manifest in meningitis, leading to cognitive impairment, paralysis or death. We investigate the use of deep learning for the classification of pulmonary nodules as the first step in the detection of valley fever.

In recent years, due to the abundance of chest X-ray (CXR) images, deep learning (DL) has gained wide popularity in the analysis of radiographic images and is anticipated to help radiologists in the context of disease detection and management. Valley fever (coccidioidomycosis) is a fungal infection that has been endemic in the southwestern areas of the United States for hundreds of years. It causes pneumonia-like symptoms and may be disseminated leading to other systemic infections that may manifest in meningitis, leading to cognitive impairment, paralysis or death. Valley fever may also manifest as pulmonary nodules which may be mistaken for lung cancer.  In the past decade, the infection incidents of this disease have skyrocketed, reaching 22 500 cases in 2011, up from 2265 in 1998. Furthermore, the difficulty in monitoring and diagnosing valley fever has led to misdiagnoses; thus, the statistics may not fully characterize the magnitude of this disease . Following the detection of a pulmonary nodule, depending on the size, diagnostic differentiation methods available are limited and primarily involve sequential imaging with CXR, CT, PET-CT or invasive tissue sampling. An automated method can provide a second opinion and thus mitigate the problem of misdiagnosis. Furthermore, the existence of a fully automated method can help with diagnosis in less developed regions that don’t have access to an experienced physician.

Introducing a non-invasive infrared thermography based method for measurement of Skin Conductance Response (SCR)

Figure 2: The number of detected sweat glands per frame for a 30 seconds period (~900 frames) along with the histogram of their size distribution for 3 sample frames.

Electrodermal responses have been among the most widely employed psychophysiological measures of sympathetic nervous system (SNS) activity. Both numbers of active sweat glands within a defined area of a fingertip also known as palmar sweat index (PSI) and standard electrodermal measures, require skin contact sensor systems. Identification of active sweat glands may serve as a surrogate for the skin conductance response (SCR) also known as the electrodermal response when a contact method is either unavailable or undesirable.

Different noncontact measures of SNS activity includes ocular-base variables such as saccade, slow eye movement, pupil dilation, eyeblink, or eyelid closure. Such procedures are not practical for investigations of SNS responses to trauma reminders or other variably timed, amygdala-activating stimuli in patients with PTSD. Individuals with PTSD, in whom heightened arousal (hypervigilance, irritability, and emotional or physiological responses to conditioned cues) may be triggered by threat cues of contact-based methods of measuring pore gland activities.  Also, the typical PSI technique suffers from limitations including poor temporal resolution and the tedious nature of its application and analysis. Non-contact methodologies for observing sweat gland activity have also been studies, including video microscopy and Optical Coherence Tomography. More recent methods quantify the pores count from thermal imagery. Pavlidis et al. use wavelets which provide an abstract aggregate measure of sweat activity based on spatial and temporal frequency variations in the imagery. Krzywicki et al. identify individual active sweat glands on the face and on the volar surfaces of the index and middle fingers using a matched filter technique (pore template was derived from a 7 × 7 pixel thermal image of a representative active pore.

In this study, non-invasive infrared thermography has been used to visualize individual sweat pores. Sweat secretion from individual pores in circumscribed areas was imaged using a high spatial resolution FLIR camera. Further different enhancement techniques are applied to the video frames to facilitate the detection of sweat glands. Multi-scale methods are central to image analysis, but they are also prone to halos. Achieving artifact-free results requires sophisticated edge-aware techniques and careful parameter tuning. Fast local Laplacian filter followed by a manually designed filter (Fig. 2), H-maxima transformation (to suppress all maxima in the intensity image), and finally an averaging filter, is applied to the raw frames to enhance the sweat glands’ visibility, reduce the background variability and thus facilitate the segmentation process and improve its accuracy. Further, a locally adaptive image threshold method that uses local mean intensity (first-order statistics) around each pixel with a sensitivity factor in the range of [0,1] (sensitivity factor determines which pixels get thresholded as foreground pixels. High sensitivity values lead to thresholding more pixels as foreground) was used to segment the output of the prior enhancement method. Fig. 1 shows the occurrence distribution of active sweat glands through a 30 seconds period and their size distribution for three sample frames.

Figure 3: Enhancement of raw image and removal of its background

Finally, to track the detected active sweat glands, a Kalman filter is used to predict the glands’ location in each following frame and determine the likelihood of that detection being assigned to an existing active sweat gland (detected through the prior algorithm). Further, this algorithm keeps track of the number of consecutive frames that each pore stayed active taking into account a grace period (due to the possible failure of the first algorithm to detect the active glands in some frames). To assign each of Kalman filter predictions to segmented sweat glands, a Kuhn-Munkres algorithm also known as Hungarian matching algorithm was used to computes an assignment that minimizes the total cost. Fig. 3 shows an example of the tracked pores for one frame.

Figure 4: Image to the right shows the segmentation mask (detected sweat pores) for the frame image shown on left. The numbers overlaid on top of the image to the left shows the IDs for tracked sweat glands, the tags that include ‘predicted’ are estimated location of pores on the prior frames that were not active during this frame.