Search

Speakers design communication for their audience, providing more information in both speech and gesture when their listener is naïve to the topic. We test whether the hippocampal declarative memory system contributes to multimodal audience design. The hippocampus, while traditionally linked to episodic and relational memory, has also been linked to the ability to imagine the mental states of others and use language flexibly. We examined the speech and gesture use of four patients with hippocampal amnesia when describing how to complete everyday tasks (e.g., how to tie a shoe) to an imagined child listener and an adult listener. Although patients with amnesia did not increase their total number of words and instructional steps for the child listener, they did produce representational gestures at significantly higher rates for the imagined child compared to the adult listener. They also gestured at similar frequencies to neurotypical peers, suggesting that hand gesture can be a meaningful communicative resource, even in the case of severe declarative memory impairment. We discuss the contributions of multiple memory systems to multimodal audience design and the potential of gesture to act as a window into the social cognitive processes of individuals with neurologic disorders. [ABSTRACT FROM AUTHOR]

Object naming involves accessing meaning and retrieving the associated word form from remote semantic memory. Historically, previously acquired semantic knowledge (i.e., remote semantic memory) was thought to be independent of the hippocampus via neocortical consolidation. This view is based on evidence demonstrating a dissociation in behavior in patients with hippocampal amnesia: amnesic patients are impaired in acquiring new vocabulary yet can name and define previously acquired words. More recently, the view that remote semantic memory is hippocampus‐independent has been challenged by the documentation of disruptions in aspects of remote semantic memory in patients with hippocampal amnesia, particularly in language use and depth of semantic knowledge. Based on these findings, we hypothesized that the hippocampus plays a long‐term role in remote semantic memory. We tested amnesic patients and demographically matched healthy comparison participants in an extensive naming task using photographic images of objects normalized for familiarity, object agreement, and visual complexity. Amnesic patients were less likely to correctly name objects than healthy comparison participants. Further, amnesic patients' performance worsened for words that were less familiar, more visually complex, and had less object agreement. These findings suggest that the hippocampus may play a long‐term role in semantic memory processes, rather than a time‐limited role in the initial acquisition of semantic information, and that hippocampal damage can disrupt object naming. [ABSTRACT FROM AUTHOR]

As reliance on digital communication grows, so does the importance of communicating effectively with text. Yet when communicating with text, benefits from other channels, such as hand gesture, are diminished. Hand gestures support comprehension and disambiguate characteristics of the spoken message by providing information in a visual channel supporting speech. Can emoji (pictures used to supplement text communication) perform similar functions? Here, we ask whether emoji improve comprehension of indirect speech. Indirect speech is ambiguous, and appropriate comprehension depends on the receiver decoding context cues, such as hand gesture. We adapted gesture conditions from prior research (Kelly et al., 1999, Experiment 2) to a digital, text-based format, using emoji rather than gestures. Participants interpreted 12 hypothetical text-message exchanges that ended with indirect speech, communicated via text only, text+emoji, or emoji only, in a between-subjects design. Like that previously seen for hand gesture, emoji improved comprehension. Participants were more likely to correctly interpret indirect speech in the emoji-only condition compared with the text+emoji and the text-only conditions, and more likely in the text+emoji condition compared to the text-only condition. Thus, emoji are not mere decoration, but rather are integrated with text to communicate and disambiguate complex messages. Similar to gesture in face-to-face communication, emoji improve comprehension during text-based communication. [ABSTRACT FROM AUTHOR]

When we communicate, we alter our language and gesture based on the mutually shared knowledge - common ground - that we have with our listener. How memory supports these alterations remain unclear. We asked healthy adults and patients with hippocampal amnesia to engage in a referential communication task. Previous work suggests that common ground can be encoded by distinct memory systems; Amnesic patients show normal learning and referential label use as common ground increases, but inconsistently mark these labels with definite determiners (e.g., the vs. a windmill). Which memory systems support the ability to mark common ground via hand gesture? We found that gestures of both healthy participants and amnesic patients reflected common ground status. Both groups produced high gestures when common ground was lacking, and were less likely to do so as common ground increased. These findings suggest that gesture can reflect common ground status during conversation, potentially via non-declarative memory. [ABSTRACT FROM AUTHOR]

During conversation, people integrate information from co-speech hand gestures with information in spoken language. For example, after hearing the sentence, “A piece of the log flew up and hit Carl in the face” while viewing a gesture directed at the nose, people tend to later report that the log hit Carl in the nose (information only in gesture) rather than in the face (information in speech). The cognitive and neural mechanisms that support the integration of gesture with speech are unclear. One possibility is that the hippocampus – known for its role in relational memory and information integration – is necessary for integrating gesture and speech. To test this possibility, we examined how patients with hippocampal amnesia and healthy and brain-damaged comparison participants express information from gesture in a narrative retelling task. Participants watched videos of an experimenter telling narratives that included hand gestures that contained supplementary information. Participants were asked to retell the narratives and their spoken retellings were assessed for the presence of information from gesture. For features that had been accompanied by supplementary gesture, patients with amnesia retold fewer of these features overall and fewer retellings that matched the speech from the narrative. Yet their retellings included features that contained information that had been present uniquely in gesture in amounts that were not reliably different from comparison groups. Thus, a functioning hippocampus is not necessary for gesture-speech integration over short timescales. Providing unique information in gesture may enhance communication for individuals with declarative memory impairment, possibly via non-declarative memory mechanisms. [ABSTRACT FROM AUTHOR]

Co-speech hand gesture facilitates learning and memory, yet the cognitive and neural mechanisms supporting this remain unclear. One possibility is that motor information in gesture may engage procedural memory representations. Alternatively, iconic information from gesture may contribute to declarative memory representations mediated by the hippocampus. To investigate these alternatives, we examined gesture's effects on word learning in patients with hippocampal damage and declarative memory impairment, with intact procedural memory, and in healthy and in braindamaged comparison groups. Participants learned novel label-object pairings while producing gesture, observing gesture, or observing without gesture. After a delay, recall and object identification were assessed. Unsurprisingly, amnesic patients were unable to recall the labels at test. However, they correctly identified objects at above chance levels, but only if they produced a gesture at encoding. Comparison groups performed well above chance at both recall and object identification regardless of gesture. These findings suggest that gesture production may support word learning by engaging nondeclarative (procedural) memory. [ABSTRACT FROM AUTHOR]

Hippocampal functioning contributes to our ability to generate multifaceted, imagistic event representations. Patients with hippocampal damage produce event narratives that contain fewer details and fewer imagistic features. We hypothesized that impoverished memory representations would influence language at the word level, yielding words lower in imageability and concreteness. We tested this by examining language produced by patients with bilateral hippocampal damage and severe declarative memory impairment, and brain-damaged and healthy comparison groups. Participants described events from the real past, imagined past, imagined present, and imagined future. We analyzed the imageability and concreteness of words used. Patients with amnesia used words that were less imageable than those of comparison groups across time periods, even when accounting for the amount of episodic detail in narratives. Moreover, all participants used words that were relatively more imageable when discussing real past events than other time periods. Taken together, these findings suggest that the memory that we have for an event affects how we talk about that event, and this extends all the way to the individual words that we use. [ABSTRACT FROM AUTHOR]

Viewing co-speech hand gestures with spoken phrases enhances memory for phrases, as compared to when the phrases are presented without gesture. Prior work investigating the mechanism underlying the effect of gesture on memory has implicated engagement of the motor system; when the hands are engaged in an unrelated motor task when viewing gesture, the beneficial effect of gesture is absent. However, one alternative interpretation of these findings is that the beneficial effect of gesture disappears due to mismatched contexts at encoding and retrieval: The hands are engaged during either encoding or retrieval, but not during both stages. Here we examined whether matching the motor context at encoding and retrieval plays a role in the beneficial effect of gesture on memory during a phrase recall task. Participants were presented with phrases that were viewed with and without gesture. Participants were assigned to one of four conditions that determined whether they would complete an unrelated motor task at (1) encoding only, (2) retrieval only, (3) both encoding and retrieval, or (4) neither. During stages in which they were not completing a motor task, participants' hands were in their laps. We found that gesture enhanced memory for phrases both when participants engaged in an unrelated motor task at encoding and retrieval and when they did not complete the motor task during either stage. Furthermore, phrases observed with gesture were more likely to be paraphrased than to be recalled literally. Together, these findings demonstrate that gesture can enhance memory even when the motor system is engaged in another task, as long as that same task is performed at retrieval. [ABSTRACT FROM AUTHOR]