Quantitative Measuring of Neuronal Reconstructions – The DIADEM Metric

Summary

Summary: This post summarizes some recent efforts in the fields of neuroscience and computer vision to improve neuronal reconstructions, particularly the DIADEM metric, a measure used to evaluate reconstructions produced by algorithms.

Citations:

Gillette TA, Brown KM, Ascoli GA. The DIADEM Metric: Comparing multiple reconstructions of the same neuron. Neuroinformatics, 9:233-245 (2011).

Gillette TA, Brown KM, Svoboda K, Liu Y, Ascoli GA. DIADEMchallenge.Org: A Compendium of Resources Fostering the Continuous Development of Automated Neuronal Reconstruction. Neuroinformatics, 9:303–304 (2011).

Almost all fields, particularly in biology, are greatly benefited by having a large data set. Morphological reconstructions of neurons obviously benefit the study of neuronal morphology, but also the related fields of neuron growth, neuron function with regards to drug and/or dysfunction, connectomics, among many others. Until recently reconstructions were generated and analyzed for a particular study, occasionally being reused when another lab requested it or for a follow-up study. NeuroMorpho.Org helped to change this by bringing together reconstructions from labs around the world doing a variety of experiments. Now will over 10,000 reconstructions there are many questions that can be asked that could not be before so many were brought together.

Nonetheless, 10,000 is not so many when one considers the vast diversity of ...

Summary: This post summarizes some recent efforts in the fields of neuroscience and computer vision to improve neuronal reconstructions, particularly the DIADEM metric, a measure used to evaluate reconstructions produced by algorithms.

Citations:

Gillette TA, Brown KM, Ascoli GA. The DIADEM Metric: Comparing multiple reconstructions of the same neuron. Neuroinformatics, 9:233-245 (2011).

Gillette TA, Brown KM, Svoboda K, Liu Y, Ascoli GA. DIADEMchallenge.Org: A Compendium of Resources Fostering the Continuous Development of Automated Neuronal Reconstruction. Neuroinformatics, 9:303–304 (2011).

Almost all fields, particularly in biology, are greatly benefited by having a large data set. Morphological reconstructions of neurons obviously benefit the study of neuronal morphology, but also the related fields of neuron growth, neuron function with regards to drug and/or dysfunction, connectomics, among many others. Until recently reconstructions were generated and analyzed for a particular study, occasionally being reused when another lab requested it or for a follow-up study. NeuroMorpho.Org helped to change this by bringing together reconstructions from labs around the world doing a variety of experiments. Now will over 10,000 reconstructions there are many questions that can be asked that could not be before so many were brought together.

Nonetheless, 10,000 is not so many when one considers the vast diversity of neurons and potential sources in terms of species, brain regions, experimental conditions, and many other variables. Moreover, producing a connectome/wiring diagram for a single species under a single experimental condition would require massive amounts of reconstructed neurons. Several bottlenecks exist, but largest among them is the reconstruction process itself. Depending on the neuron and what components are being traced, it could take weeks or even many months to fully reconstruct a neuron. Automated and semi-automated techniques have made a big difference in this process, especially for cases in which the underlying microscopy is of high quality. Unfortunately there are many circumstances which the underlying data contains imperfections that tend to cause problems for algorithms.

The DIADEM challenge (short for Digital Reconstruction of Axonal and Dendritic Morphology) was posed as a way to determine what the current state of the art was in automated neuronal reconstruction, determine the existing challenges facing researchers, and set a path towards solving those challenges. Measuring the quality of algorithms was an important component of the process, and several ideas were put forth and used in different parts of the competition. One measure, used in the qualification stage, was a subjective qualitative judgement by expert data users, i.e. those researchers who use the reconstructions in their projects. A second measure, used in the qualifier and final rounds, was the amount of time saved in reconstructing a neuron compared to current manual (via human-directed tracing programs) techniques. An objective, algorithmic measure was also posed, named the DIADEM metric (though technically it is not a metric).

The DIADEM metric was designed to score reconstructions based on the more difficult and more biologically important aspects of neuronal reconstruction, namely, the branching structure. It requires a “gold standard” reconstruction produced by an expert in the particular cell type being traced. An open-source post-challenge version was released that had greater flexibility in allowing users to set weighting of branch importance, however the default version made branches with larger subtrees count for more given their impact on connectivity and signal integration. Path length was taken into account, though more as a method to validate parent-child relationships than as a criteria for success. It was assumed that path tracing was a relatively easier problem and thus did not require checking.

The article discusses variations that might be useful to developers interested in validating other aspects of their algorithms and resulting reconstructions. Moreover, variations to the algorithm could prove useful to other research questions, such as changes in neuronal morphology over time during development or due to some experimental condition. Alternatives to the single gold standard are also considered.

It was ultimately concluded that the DIADEM metric was successful in that it correlated with subjective expert judgement as well as various experts’ judgements correlated with each other. While expert judgement is itself somewhat variable, the options provided with the metric make it possible for someone wishing to test their algorithm, or an algorithm they are considering using in their lab, to set their own standard of quality.

Skip to toolbar