Hand gestures such as gripping, pinching and grasping are driven by movements of muscles in our forearm. These movements generate small electrical signals that can be read by sensors on the skin, a technique called electromyography.

“Using sensors and machine learning, we can recognize gestures based on muscle activity,” said Jonathon Schofield, professor of mechanical and aerospace engineering at UC Davis and senior author on the paper.

EMG-based controls perform well in a lab setting and with limbs at rest. But there is a well-known problem of “position and load.” If you move your arm to a different position — say, shoulder height, or over your head — or grasp objects of different weights, the measurements change.

“In the real world, every time you move a limb and grasp something the measurement is going to change,” said graduate student Peyton Young, first author on the paper. “The neutral position (where the limb is held passively next to the body) is very different to moving around.”

Combining EMG and FMG

To address this, Young and Schofield experimented with a different type of measurement, alone and in combination with EMG. Force myography (FMG) measures how muscles in the arm bulge as they contract.

Young constructed a cuff that goes round the forearm and includes both EMG and FMG sensors. He used this device with a series of able-bodied volunteers in the lab who performed a series of arm gestures with while participants held different loads with different hand grasps. Data from the sensors was fed to a machine learning algorithm to classify the different movements into pinch, pick, fist and so on. The algorithm was trained on either EMG or FMG signals alone, or on a combination.

For each experiment, the algorithm was trained on some of the data and scored on its ability to accurately classify the rest.

“We train the classifier on data from the gestures, then score it on its ability to predict them,” Young said.

They found that position and loading did indeed affect the accuracy of classification of gestures. Overall, a combination of EMG and FMG gave over 97 percent classification accuracy, compared to 92 percent for FMG alone and 83 percent for EMG alone.

Young is now working on a combined FMG/EMG sensor and the team is working towards an experimental prosthetic limb that uses the technology.

The approach could have a wide range of applications for prosthetics and robotics as well as for virtual reality tools, Schofield said. The team benefits enormously from being able to collaborate with clinical prosthetics experts, surgeons and biologists across UC Davis, he said.

“We wouldn’t be able to do it without exposure to actual patients and clinicians,” Schofield said.

Additional authors on the paper are Kihun Hong, Eden Winslow, Giancarlo Sagastume, Marcus Battraw and Richard Whittle, all at UC Davis. Battraw is now on the faculty at California State University, Chico.

Cell migration influences many processes in the body, including how immune cells travel to an infection site, how the brain develops and how wounds are repaired. It is also exploited by disease cells, such as in metastatic cancer. While cell movement is known to be directed by the interaction between two protein families, GPCRs and chemokines, the vast similarities between members of each family have presented a challenge in understanding how the correct pairs form and control the movement of relevant cells. The researchers developed data science approaches to identify the exact parts of each protein governing their molecular interactions.

“We found that cells have an elegant system that uses structure and disorder together to control cell migration,” said senior co-corresponding author M. Madan Babu, PhD, FRS, St. Jude Senior Vice President of Data Science and Center of Excellence for Data-Driven Discovery director, Department of Structural Biology. “With that understanding, we can now rationally introduce small changes in a chemokine’s structure to ultimately alter cell migration in desired ways.”

Small, disordered regions provide order to chemokines-GPCR pairs

The scientists uncovered how chemokines and their receptors bind select members of the GPCR family by data mining protein sequences and structural information. They compared all human chemokine-binding GPCRs and all chemokines, then compared similar chemokines and GPCRs from other species. They also looked at each protein individually at a population level, finding places that stayed the same across groups and those that differed.

“Through our data analysis, we discovered that the information for how chemokines and GPCRs select for each other is stored in small, discrete packages of highly unstructured, disordered regions,” said first and co-corresponding author Andrew Kleist, MD, PhD, St. Jude Center of Excellence for Data-Driven Discovery, Department of Structural Biology. “The mix of those small packages from both the chemokine and receptor results in the unique interaction, similar to website data encryption keys, which governs cell migration.” Kleist started the work as a graduate student in the laboratory of co-corresponding author Brian Volkman, PhD, Professor of Biochemistry at the Medical College of Wisconsin.

Websites keep sales secure with public and private digital keys. The seller and buyer each possess a public key and a private key, both of which are prime numbers. When the private and public keys are multiplied together, the resulting unique number ensures that only the two parties taking part in the transaction can exchange information while protecting that information from bad actors. The scientists found the disordered regions in these proteins acted like private keys, while the structured regions acted like public keys. The interactions of a chemokine’s disordered region with a GPCR’s structured region, within the greater context of the highly structured portions of each protein, provide cells with a unique chemical identifier for that chemokine-GPCR pair, just like verifying a pair of public and private keys. That unique identifier contains the information for cells to respond appropriately to a particular chemokine-GPCR binding, migrating towards more of that chemokine.

“Once we understood how these proteins interacted, we demonstrated we could rationally mutate them to have different properties,” Babu said. The researchers changed the regions determining the selectivity of a chosen chemokine to alter its receptor binding preferences. Co-author Lindsay Talbot, MD, St. Jude Department of Surgery, showed that the scientists could change how T cells, a type of white blood cell, move, turning down a signal that normally stops their movement.

Making forward movements with chemokines and GPCRs

“Now that we’ve shown a proof of concept, our approach will guide exploration into new medicines and improvements for existing cellular therapies,” Kleist said. “For example, it may be possible to create molecules that better lead immune cells to cancers or help recruit more blood stem cells for bone marrow transplants. In theory, any therapy using cell movement could benefit from applying these principles.”

To enable scientists and clinicians to test this, the collaborators published their data science framework online. The resource is the first step in pushing cell movement manipulation from concept into reality for patients.

“When people think about the body, we think every cell stays in place, but that’s a simplistic view,” Babu said. “Depending on the tissue, cells are moving all the time, and our new understanding of those systems opens novel avenues for therapeutic development.”

The framework to assist the rational design of chemokines and receptors is freely available at: https://github.com/andrewbkleist/chemokine_gpcr_encoding.

The glasses are one of the wizard’s many deceits; the city viewed through green-tinted glasses would, of course, only look more green.

But using a new technique called “Oz,” scientists at the University of California, Berkeley, have found a way to manipulate the human eye into seeing a brand-new color — a blue-green color of unparalleled saturation that the research team has named “olo.”

“It was like a profoundly saturated teal … the most saturated natural color was just pale by comparison,” said Austin Roorda, a professor of optometry and vision science at UC Berkeley’s Herbert Wertheim School of Optometry & Vision Science, and one of the creators of Oz.

Oz works by using tiny doses of laser light to individually control up to 1,000 photoreceptors in the eye at one time. Using Oz, the team is able to show people not only a green more stunning than anything in nature, but also other colors, lines, moving dots and images of babies and fish.

The platform could also be used to answer basic questions about human sight and vision loss.

“We chose Oz to be the name because it was like we were going on a journey to the land of Oz to see this brilliant color that we’d never seen before,” said James Carl Fong, a doctoral student in electrical engineering and computer sciences (EECS) at UC Berkeley.

“We’ve created a system that can track, target and stimulate photoreceptor cells with such high precision that we can now answer very basic, but also very thought-provoking, questions about the nature of human color vision,” Fong said. “It gives us a way to study the human retina at a new scale that has never been possible in practice.”

The Oz technique is described in a new study published last week in the journal Science Advances. The work was funded in part by federal grants from the National Institutes of Health and the Air Force Office of Scientific Research.

Untapped photoreceptors

Humans are able to see in color thanks to three different types of photoreceptor “cone” cells embedded in the retina. Each type of cone is sensitive to different wavelengths of light: S cones detect shorter, bluer wavelengths;, M cones detect medium, greenish wavelengths; and L cones detect longer, reddish wavelengths.

However, due to an evolutionary quirk, the light wavelengths that activate the M and L cones are almost entirely overlapping. This means that 85% of the light that activates M cones also activates L cones.

“There’s no wavelength in the world that can stimulate only the M cone,” said study senior author Ren Ng, a professor of EECS at UC Berkeley, “I began wondering what it would look like if you could just stimulate all the M cone cells. Would it be like the greenest green you’ve ever seen?”

To find out, Ng teamed up with Roorda, who had created a technology that used tiny microdoses of laser light to target and activate individual photoreceptors. Roorda calls the technology “a microscope for looking at the retina,” and it is already being used by ophthalmologists to study eye disease.

But for a human to actually perceive a whole new color, Ng and Roorda would need to find a way to activate not just one cone cell, but thousands of them.

A movie screen the size of a fingernail

Fong first started working on the Oz project in 2018 as an undergraduate engineering student, and has created much of the complex software needed to translate images and colors into thousands of tiny laser pulses directed at the human retina.

“I joined after meeting this other student who was working with Ren, who told me that they were shooting lasers into people’s eyes to make them see impossible colors,'” Fong said.

For Oz to work, first you need a map of the unique arrangement of the S, M and L cone cells on an individual’s retina. To get these maps, the researchers collaborated with Ramkumar Sabesan and Vimal Prahbhu Pandiyan at the University of Washington, who have developed an optical system that can image the human retina and identify each cone cell.

With an individual’s cone map in hand, the Oz system can be programmed to rapidly scan a laser beam over a small patch of the retina, delivering tiny pulses of energy when the beam reaches a cone that it wants to activate, and otherwise staying off.

The laser beam is just one color — the same hue as a green laser pointer — but by activating a combination of S, M and L cone cells, it can trick the eye into seeing images in full technicolor. Or, by primarily activating the M cone cells, Oz can show people the color olo.

“If you look at your index fingernail at arm’s length, that’s about the size of the display,” said Roorda. “But if we could, we would have filled the entire visual space like an IMAX.”

The ‘wow’ experience

Hannah Doyle, a doctoral student in EECS and co-lead author of the paper, designed and ran the human experiments with Oz. Five human subjects got the chance to see the color olo, including Roorda and Ng, who were aware of the purpose of the study, but not the specifics of what they would see.

In one experiment, Doyle asked the participants to compare olo to other colors. They described it as blue-green or peacock green, and reported that it was much more saturated than the nearest monochromatic color.

“The most saturated colors you can experience in nature are the monochromatic ones. Light from a green laser pointer is one example,” Roorda said. “When I pinned olo up against other monochromatic light, I really had that ‘wow’ experience.”

Doyle also tried “jittering” the Oz laser, directing it ever-so-slightly off target so the light pulses hit random cones rather than only M cones. The participants immediately stopped seeing olo and started seeing the regular green of the laser.

“I wasn’t a subject for this paper, but I’ve seen olo since, and it’s very striking. You know you’re looking at something very blue-green,” Doyle said. “When the laser gets jittered, the normal color of the laser almost looks like yellow because the difference is so stark.”

Probing the nature of color vision

Oz isn’t just useful for projecting tiny movies into the eye. The research team is already finding ways to use the technique to study eye disease and vision loss.

“Many diseases that cause visual impairment involve lost cone cells,” Doyle said. “One application that I’m exploring now is to use this cone by cone activation to simulate cone loss in healthy subjects.”

They are also exploring whether Oz could help people with color blindness to see all the colors of the rainbow, or if the technique could be used to allow humans to see in tetrachromatic color, as if they had four sets of cone cells.

It may also help answer more fundamental questions about how the brain makes sense of the complex world around us.

“We found that we can recreate a normal visual experience just by manipulating the cells — not by casting an image, but just by stimulating the photoreceptors. And we found that we can also expand that visual experience, which we did with olo,” Roorda said. “It’s still a mystery whether, if you expand the signals or generate new sensory inputs, will the brain be able to make sense of them and appreciate them? And, you know, I like to believe that it can. I think that the human brain is this really remarkable organ that does a great job of making sense of inputs, existing or even new.”

Additional authors of the study include Congli Wang, Alexandra E. Boehm, Sophie R. Herbeck, Brian P. Schmidt, Pavan Tiruveedhula, John E. Vanston and William S. Tuten of UC Berkeley. This work was supported by a Hellman Fellowship, FHL Vive Center Seed Grant, Air Force Office of Scientific Research grants (FA9550-20-1-0195, FA9550-21-1-0230), National Institutes of Health grant (R01EY023591, R01EY029710, U01EY032055) and a Burroughs Wellcome Fund Career Award at the Scientific Interface.