Seeing Sounds and Hearing Colors: The Vibrant World of Synesthesia

Seeing Sounds and Hearing Colors: The Vibrant World of Synesthesia

Speakers: Kaitlyn Hova, Software Executive, JP Morgan

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=kHTgzGgYgv8

Overview

Kaitlyn Hova, a distinguished software executive at JP Morgan with a unique background spanning music and neuroscience, delivered a captivating talk at RSAC 2024 that ventured beyond traditional cybersecurity discourse. Titled "Seeing Sounds and Hearing Colors: The Vibrant World of Synesthesia," Hova's presentation offered a deeply personal and insightful exploration into the phenomenon of synesthesia, a neurological condition where stimulation of one sensory or cognitive pathway leads to automatic, involuntary experiences in a second sensory or cognitive pathway. While the RSAC conference is renowned for its focus on information security, Hova's talk provided a compelling departure, inviting attendees to consider the profound intricacies of human perception and cognitive diversity.

The talk was not merely an academic overview but a vibrant, first-person account of living with synesthesia, enriched by Hova’s multidisciplinary expertise. She skillfully wove together anecdotes from her own life, scientific insights, and an innovative technological demonstration to illustrate how her world is perceived through a unique cross-sensory lens. The significance of this presentation lies in its ability to broaden understanding of the human mind, challenging conventional notions of sensory experience and highlighting the hidden cognitive landscapes that shape individual realities. For an audience accustomed to discussions of threats and defenses, Hova's talk served as a powerful reminder of the complex, often beautiful, variability inherent in human cognition.

Background

▶ Watch: The surprising discovery of synesthesia (0:40)

The concept of synesthesia defines a neurological trait characterized by the involuntary and automatic experience of cross-modal perceptions. In essence, stimulation in one sense or cognitive pathway triggers a consistent, additional experience in another. As Hova described, it’s a condition where, for example, hearing a sound might evoke a visual perception of color, or seeing a number might involuntarily conjure a specific taste. This is not a metaphor or an active imagination; it is a genuine, physical experience for those who possess it. Hova discovered her own synesthetic experiences later in life, realizing that her perception of music and numbers was fundamentally different from others.

One of the most common forms, and one that Hova experiences, is grapheme-to-color synesthesia. Individuals with this type perceive numbers and letters as inherently colored. For Hova, this meant that the letter 'H' was always green, and 'M' was red, influencing seemingly trivial choices like the color of school binders for history and math. This ingrained association highlights the automatic and involuntary nature of the condition; these are not conscious decisions but rather intrinsic elements of perception. Research suggests that synesthesia is surprisingly prevalent, affecting approximately one in 23 people, or roughly 4% of the global population. To put this into perspective, it is statistically more common than having red hair, indicating that many individuals likely know someone with synesthesia, or may even possess it themselves without realizing it.

Hova also touched upon the fascinating phenomenon of synesthetic "rubbing," where having one type of synesthesia increases the likelihood of experiencing others, often leading to unique cognitive quirks. She provided a relatable example: her difficulty remembering names like "Roger" if the person associated with it projects a "green" aura, as "Roger" is a "red" word for her, leading her to inadvertently call them "Greg" (a "green" word). This illustrates how synesthesia can subtly, yet significantly, influence daily interactions and memory. The talk further suggested that synesthesia exists on a spectrum, with even non-synesthetes experiencing mild forms of cross-sensory association. A widely cited example is the Kiki-Boba effect, where people overwhelmingly associate a spiky, angular shape with the non-word "Kiki" and a round, blob-like shape with "Boba," demonstrating an innate tendency to link visual forms with phonetic qualities. This phenomenon underscores the idea that cross-sensory experiences might be a more fundamental part of average human perception than commonly understood, suggesting a continuum rather than a binary state of having or not having synesthesia.

Key Findings

▶ Watch: Synesthesia is surprisingly common (1 in 23 people) (2:50)

Kaitlyn Hova's presentation illuminated several key findings regarding synesthesia, drawing from both scientific research and her unique personal experience. Foremost among these is the prevalence of synesthesia, which is significantly higher than often assumed. Contrary to being a rare anomaly, studies suggest that approximately one in 23 people, or roughly 4% of the world's population, experience some form of synesthesia. This statistic challenges the perception of synesthesia as an exotic condition, positioning it instead as a notable aspect of human cognitive diversity that is far more common than traits like red hair (1-2% prevalence). This finding encourages a broader awareness and understanding of how diverse human brains can be.

Another critical insight is the diverse and highly personal manifestation of synesthetic experiences. While two individuals might both have grapheme-to-color synesthesia, the specific colors they associate with letters or numbers can vary wildly. For Hova, 'A' might be orange and 'B' purple, while for another synesthete, these associations could be entirely different. Crucially, Hova emphasized that both experiences are equally valid. This highlights that while the underlying neurological mechanism of cross-activation is consistent, the specific mapping of sensory inputs to outputs is highly individualized. This personal, yet consistent, nature of synesthetic perception underscores its involuntary and automatic character, distinguishing it from conscious association or imagination.

The talk also underscored the idea that cross-sensory experiences are a fundamental part of human perception, existing on a spectrum accessible to everyone. The Kiki-Boba effect serves as a compelling example, demonstrating that even individuals who do not identify as synesthetes exhibit an innate tendency to link abstract visual forms with specific sounds. This suggests that the neurological pathways responsible for cross-modal processing might be present to some degree in all individuals, with synesthetes simply having more pronounced or active connections. This perspective reframes synesthesia not as an anomaly, but as an amplified expression of a universal cognitive mechanism, potentially making the world "even more beautiful than the world itself" for those who experience it.

Finally, Hova's narrative revealed how synesthesia can profoundly impact daily life and explain unique cognitive quirks. Her struggles with remembering names that conflict with the "color" of a person, or her adamant insistence on specific binder colors in school, illustrate how these involuntary perceptions shape memory, social interactions, and even mundane organizational tasks. These examples provide concrete evidence of how synesthesia is not merely a curiosity but a deeply integrated aspect of an individual's cognitive architecture, influencing their processing and interpretation of the world in systematic and often surprising ways. The ability of synesthesia to explain such previously inexplicable "goofiness" further solidifies its status as a significant and impactful neurological phenomenon.

Technical Deep Dive

▶ Watch: How synesthesia creates cognitive quirks (e.g., names) (3:20)

While Kaitlyn Hova’s talk primarily explores the fascinating world of human perception, the "technical" aspects can be understood through the lens of neuroscience, the systematic nature of synesthetic experiences, and the engineering behind her demonstration. At its core, synesthesia represents a remarkable instance of cross-sensory wiring within the brain. Though the transcript does not delve into specific neuroanatomical structures, the very definition — "stimulation in one sense causes an automatic involuntary experience in another" — implies a unique neurological architecture where typical sensory pathways exhibit atypical interconnections. For a synesthete, the brain regions responsible for processing, say, visual information (like graphemes) are hyper-connected or cross-activated with areas responsible for color perception, leading to the consistent and involuntary pairing of letters/numbers with specific hues.

Hova's experience with grapheme-to-color synesthesia provides a clear technical example of this phenomenon. For her, each letter and number possesses an inherent, consistent color: 'H' is green, 'M' is red, 'C' is red, 'D' is blue, 'E' is yellow, 'F' is light green, 'G' is super green, 'A' is orangish red, and 'B' is purple. These are not learned associations but automatic perceptions, so deeply ingrained that they influenced her childhood decisions, such as demanding specific colored binders for school subjects. This systematic internal mapping, while personal to each synesthete, demonstrates a rule-bound, albeit non-conscious, processing system. The consistency of these associations over time is a key diagnostic criterion for synesthesia, distinguishing it from mere memory tricks or drug-induced hallucinations.

The intersection of Hova’s background in music and neuroscience further enriches the technical understanding. Her question, "how does music theory work if you can't see the sounds that you're analyzing?" highlights a profound integration of sensory input in her cognitive process. For Hova, musical notes are not just auditory phenomena; they are visually colored entities that contribute to her analytical understanding of music. This suggests that her brain leverages these additional visual cues to process and interpret complex auditory information, potentially offering a unique cognitive advantage in fields like music composition or analysis. The "cognitive quirks" she describes, such as misremembering names based on conflicting color associations, further illustrate the pervasive and sometimes challenging technical implications of this cross-wiring on daily cognitive functions like memory and social interaction. A name like "Roger" might consistently evoke a "red" visual, creating a conflict if the person's 'aura' or perceived personality is "green," making it difficult for Hova to recall their actual name, preferring a "green" name like "Greg." This is a tangible example of how internal sensory mappings dictate external interactions.

The culmination of this "technical deep dive" is the Hovalin, a tangible demonstration of synesthetic experience. This 3D-printed violin is not merely an instrument but a carefully engineered device designed to externalize Hova's internal sensory world. It is equipped with LEDs (light-emitting diodes), which are precisely programmed to illuminate in the specific colors Hova perceives when playing corresponding musical notes. This involves:

1. Sensory Input Mapping: Translating auditory input (musical notes played on the violin) into specific color outputs based on Hova's personal synesthetic mapping (e.g., C-note to red light, D-note to blue light).
2. Hardware Integration: Embedding LEDs within the 3D-printed structure of the violin, requiring careful consideration of power, wiring, and light diffusion to ensure a clear visual representation.
3. Software Development: Writing code to interface with a sensor (likely a MIDI output or pitch detection system) that identifies the played note, and then triggers the appropriate LED sequence and color output.

The Hovalin thus serves as a sophisticated proof of concept, demonstrating that a highly subjective and internal neurological phenomenon can be systematically observed and even replicated through the thoughtful application of engineering principles and technology. It transforms an abstract, personal experience into a shareable, objective phenomenon, allowing others to "see" the sounds as Hova does, providing a window into the vibrant world of synesthetic perception.

Demo / Proof of Concept

▶ Watch: Introducing the Hovalin: a violin that visualizes sound (4:10)

The most compelling demonstration of Kaitlyn Hova's unique perceptual world was the Hovalin, an innovative 3D-printed violin that she co-created with her partner. This instrument is not just a musical device but a sophisticated proof of concept designed to externalize and share her personal experience of chromesthesia, a type of synesthesia where sounds (in this case, musical notes) automatically and involuntarily evoke perceptions of color.

The core of the Hovalin's demonstration lies in its integrated LEDs (light-emitting diodes). These LEDs are strategically embedded within the violin's structure and are meticulously programmed to illuminate in the precise colors that Hova sees in real-time as she plays specific musical notes. This means that as her fingers move across the strings and the bow draws out melodies, the Hovalin visually expresses her internal synesthetic landscape.

Hova provided a clear mapping of these note-to-color associations during her talk:

• C note: illuminates red
• D note: illuminates blue
• E note: illuminates yellow
• F note: illuminates a "weird light green"
• G note: illuminates "super green"
• A note: illuminates "orangish red"
• B note: illuminates purple
• C note (octave higher): illuminates red again, demonstrating consistency across octaves.

This real-time, dynamic visualization allows the audience to directly observe a phenomenon that is typically entirely internal and subjective. The Hovalin effectively bridges the gap between Hova's private sensory experience and the public's understanding, transforming an abstract neurological trait into a tangible and interactive display. The demonstration showcased not only the beauty of synesthesia but also the power of combining art, engineering (3D printing and electronics), and programming to explore and communicate complex aspects of human cognition. It underscores the potential for technology to provide unique insights into subjective experiences, making the invisible visible and the unheard seen.

Defensive Implications

▶ Watch: Live performance on the Hovalin visualizing synesthetic colors (5:00)

While Kaitlyn Hova's presentation on synesthesia offers a fascinating and deeply personal exploration of human perception, it is crucial to state that the content of this talk does not directly address cybersecurity threats, vulnerabilities, or traditional defensive strategies. The discussion centers on a neurological phenomenon, not on digital security, network defense, or incident response. Therefore, there are no immediate or direct defensive implications for security professionals in the conventional sense of safeguarding digital assets, infrastructure, or data.

However, if we interpret "defensive implications" in a broader, more philosophical sense related to understanding the human element within security, some tangential points can be considered. Cybersecurity is inherently a human-centric field, involving human users, attackers, and defenders. Understanding the vast spectrum of human perception and cognitive processing, as highlighted by synesthesia, can offer indirect benefits in areas such as security awareness training and user interface (UI) design for security tools.

For instance, recognizing that individuals process information through different sensory and cognitive lenses might inform more effective, multi-modal communication strategies in security awareness campaigns. If some users are more visually oriented, others auditory, and others might even have cross-sensory associations, training materials that incorporate diverse sensory cues could be more impactful and memorable. Similarly, in the design of security dashboards, alerts, or authentication systems, considering diverse perceptual experiences could lead to more intuitive, less error-prone, and more accessible interfaces. While a direct link to synesthesia might be rare, the broader principle of cognitive diversity in user experience design is highly relevant. If a designer understands that certain color combinations or auditory cues might be perceived differently or even trigger unintended cross-sensory responses for a small segment of the user base, it could lead to more robust and inclusive designs.

Furthermore, the talk implicitly encourages an appreciation for cognitive diversity within security teams. The unique ways in which synesthetes process information – for example, seeing patterns in data that others might miss due to their color associations – could potentially be leveraged in complex problem-solving scenarios, threat analysis, or anomaly detection. While highly speculative in the context of this specific talk, fostering environments where unique ways of thinking are valued and understood can enhance a team's collective intelligence and ability to identify non-obvious threats or solutions. Ultimately, while Hova's talk is not a security briefing, it serves as a powerful reminder of the intricate and varied nature of human cognition, a factor that is always present in the complex landscape of cybersecurity.

Key Takeaways

• Synesthesia is a common neurological phenomenon: Affecting approximately one in 23 people (4% of the population), synesthesia involves automatic, involuntary cross-sensory experiences, such as seeing sounds or tasting words.
• Diverse Manifestations: The most common type is grapheme-to-color synesthesia, where letters and numbers evoke specific colors, but experiences are highly personal and consistent for each individual.
• Impact on Daily Life: Synesthesia profoundly influences perception, memory, and cognitive processes, leading to unique "cognitive quirks" such as associating colors with people's names or specific preferences for object colors.
• A Spectrum of Experience: The Kiki-Boba effect demonstrates that even non-synesthetes exhibit mild cross-sensory associations, suggesting that synesthesia exists on a continuum of human perception.
• Technology for Externalization: The Hovalin, a 3D-printed violin with programmed LEDs, serves as an innovative proof of concept to externalize and share the subjective experience of chromesthesia (musical notes evoking colors) in real-time.
• Value of Cognitive Diversity: The talk highlights the rich and varied nature of human cognition, encouraging a broader understanding and appreciation for diverse ways of perceiving and processing the world.

About the Speaker(s)

Kaitlyn Hova is a dynamic and multi-talented professional who seamlessly blends expertise across diverse fields. Currently, she serves as a software executive at JP Morgan, demonstrating her prowess in the financial technology sector. However, her professional journey is far from conventional, rooted deeply in music and neuroscience. This unique interdisciplinary background has shaped her perspective, leading her to describe herself as a "very curious person" with a passion for "multi-faceted problems."

Beyond her corporate role, Hova is recognized as an entrepreneur, musician, and code engineer. Her personal experience with synesthesia, which she discovered she possessed during her college years, has been a driving force in her exploration of human perception. This genetic condition allows her to experience the world through cross-sensory perceptions, such as seeing colors when hearing sounds or analyzing music. Her work, including the co-creation of the Hovalin, exemplifies her innovative spirit and her ability to bridge the gap between scientific understanding, artistic expression, and technological application.