Search

Speech-driven animation has gained significant traction in recent years, with current methods achieving near-photorealistic results. However, the field remains underexplored regarding non-verbal communication despite evidence demonstrating its importance in human interaction. In particular, generating laughter sequences presents a unique challenge due to the intricacy and nuances of this behaviour. This paper aims to bridge this gap by proposing a novel model capable of generating realistic laughter sequences, given a still portrait and an audio clip containing laughter. We highlight the failure cases of traditional facial animation methods and leverage recent advances in diffusion models to produce convincing laughter videos. We train our model on a diverse set of laughter datasets and introduce an evaluation metric specifically designed for laughter. When compared with previous speech-driven approaches, our model achieves state-of-the-art performance across all metrics, even when these are re-trained for laughter generation.

One of the most pressing challenges for the detection of face-manipulated videos is generalising to forgery methods not seen during training while remaining effective under common corruptions such as compression. In this paper, we examine whether we can tackle this issue by harnessing videos of real talking faces, which contain rich information on natural facial appearance and behaviour and are readily available in large quantities online. Our method, termed RealForensics, consists of two stages. First, we exploit the natural correspondence between the visual and auditory modalities in real videos to learn, in a self-supervised cross-modal manner, temporally dense video representations that capture factors such as facial movements, expression, and identity. Second, we use these learned representations as targets to be predicted by our forgery detector along with the usual binary forgery classification task; this encourages it to base its real/fake decision on said factors. We show that our method achieves state-of-the-art performance on cross-manipulation generalisation and robustness experiments, and examine the factors that contribute to its performance. Our results suggest that leveraging natural and unlabelled videos is a promising direction for the development of more robust face forgery detectors., Comment: CVPR 2022. Code: https://github.com/ahaliassos/RealForensics

We present RAVEn, a self-supervised multi-modal approach to jointly learn visual and auditory speech representations. Our pre-training objective involves encoding masked inputs, and then predicting contextualised targets generated by slowly-evolving momentum encoders. Driven by the inherent differences between video and audio, our design is asymmetric w.r.t. the two modalities' pretext tasks: Whereas the auditory stream predicts both the visual and auditory targets, the visual one predicts only the auditory targets. We observe strong results in low- and high-resource labelled data settings when fine-tuning the visual and auditory encoders resulting from a single pre-training stage, in which the encoders are jointly trained. Notably, RAVEn surpasses all self-supervised methods on visual speech recognition (VSR) on LRS3, and combining RAVEn with self-training using only 30 hours of labelled data even outperforms a recent semi-supervised method trained on 90,000 hours of non-public data. At the same time, we achieve state-of-the-art results in the LRS3 low-resource setting for auditory speech recognition (as well as for VSR). Our findings point to the viability of learning powerful speech representations entirely from raw video and audio, i.e., without relying on handcrafted features. Code and models are available at https://github.com/ahaliassos/raven., Comment: ICLR 2023. Code: https://github.com/ahaliassos/raven

Video-to-speech synthesis (also known as lip-to-speech) refers to the translation of silent lip movements into the corresponding audio. This task has received an increasing amount of attention due to its self-supervised nature (i.e., can be trained without manual labelling) combined with the ever-growing collection of audio-visual data available online. Despite these strong motivations, contemporary video-to-speech works focus mainly on small- to medium-sized corpora with substantial constraints in both vocabulary and setting. In this work, we introduce a scalable video-to-speech framework consisting of two components: a video-to-spectrogram predictor and a pre-trained neural vocoder, which converts the mel-frequency spectrograms into waveform audio. We achieve state-of-the art results for GRID and considerably outperform previous approaches on LRW. More importantly, by focusing on spectrogram prediction using a simple feedforward model, we can efficiently and effectively scale our method to very large and unconstrained datasets: To the best of our knowledge, we are the first to show intelligible results on the challenging LRS3 dataset., accepted to INTERSPEECH 2022 (Oral Presentation)

Chemical composition, in vitro digestibility of dry matter (IVDDM), organic matter (IVDOM) and cell wall (IVDCW), green dry matter production (GDMP), plant re-growth (PR) and stem/leaf ratio (S/L) of five cultivars of the genus Cynodon at two levels of N fertilization (0 and 400 kg/ha) in the summer and spring are provided. CP did not differ (P > 0.05) from other grasses in chemical composition. However differences (P ≤ 0.05) occurred due to nitrogen level and cut intervals. Values for ADF and NDF were affected (P ≤ 0,05) by grass, nitrogen level and cut intervals. Grasses showed statistical differences (P ≤ 0.05) for IVDDM, IVDOM and IVDCW. Tifton 44 showed the highest digestibility for all parameters, while Estrela Porto Rico the lowest digestibility for IVDCW. Other grasses did not differ from each other. There were no differences (P > 0.05) among the fertilized cultivars for GDMP. Under no fertilization GDMP of Tifton 85 cultivar was higher than that of Tifton 44 and coast cross cultivars. Under N application PR of Tifton 85 was higher than that of Estrela Porto Rico and purple star grasses. S/L ratio of the Tifton 85 cultivar was different (P < 0.05) from Estrela Porto Rico e purple star grasses ratios. Application of N increased GDMP, PR and S/L of the cultivars. O experimento foi realizado com o objetivo de determinar a composição química, a digestibilidade in vitro da matéria seca (DIVMS), da matéria orgânica (DIVMO), da parede celular (DIVPC), a produção de matéria seca verde total (PMSVT), vigor da rebrota (VR) e relação colmo/lâmina foliar (C/L) de cinco cultivares do gênero Cynodon (Tifton 44, Tifton 85, “Coast-cross”, Estrela Porto Rico e Estrela Roxa), em 4 cortes a cada 35 dias no verão (PV) e 2 cortes a cada 70 dias no inverno (PI) com e sem N (0 e 400 kg/ha/ano). Na avaliação da composição química, a PB não diferiu (P ≤ 0,05) entre as gramíneas, porém diferiu (P ≤ 0,05) quanto ao nível de nitrogênio e ao corte. A concentração da FDA e FDN apresentou diferença estatística (P ≤ 0,05) entre as gramíneas, nível de adubação e cortes realizados. As gramíneas apresentaram diferenças estatísticas entre si (P ≤ 0,05) para a DIVMS, DIVMO e DIVPC. A Tifton 44 apresentou digestibilidade maior para todos os parâmetros, enquanto que a cultivar Estrela Porto Rico apresentou DIVPC menor em relação às demais, e as outras não diferiram entre si. A PMSVT diferiu (P < 0,05) entre níveis de N para o PV (P < 0,05) e entre cultivares e níveis para o PI. O VR diferiu entre níveis (P < 0,05) no PV e entre cultivares e níveis no PI. A C/L diferiu (P < 0,05) para cultivares e para níveis para ambos os períodos testados. Tifton 85 apresentou melhor C/L e Estrela Porto Rico pior para ambos os períodos e tratamentos. O N propiciou uma maior PMSVT, VR e C/L nos cultivares testados.

Orientador: Jose Sidney Flemming Dissertação (mestrado) - Universidade Federal do Parana Resumo: A técnica da espectrofotometria de reflectância no infravermelho proximal (NIRS) tem demonstrado ser uma técnica apta para predizer o valor nutritivo das forragens. Com o objetivo de avaliar o sistema de análise por infravermelho na predição da MS, PB, FDA, FDN, Ca e P foi conduzido um experimento na região do Município de Guarapuava, Paraná - Brasil. Foram utilizadas 360 amostras em 6 cortes com intervalo entre cortes de 61, 60, 57, 143 e 103 dias, e uma altura de corte de 5 cm durante o período de 1 ano. As amostras foram compostas por Lolium multiflorum, Trifolium repens, Trifolium pratense e Lotus comiculatus no inverno e pasto nativo no verão. As amostras foram secas em estufa com circulação forçada a 60° C e moídas em moinho Willey em peneira de 1 mm. O espectro de absorção das amostras foi obtido através de um aparelho NR 4500 com monocromador na NIRSystems e o sinal foi digitalizado e gravado na forma de log (1/R). Foram selecionadas para calibração um total de 27, 31, 30, 29, 29, 29 amostras e, após feita a calibração, os coeficientes de regressão (R2) para as frações MS, PB, FDA, FDN, Ca e P foram respectivamente: 0,92; 0,92; 0,93; 0,97; 0,49, 0,64 e um erro padrão de calibração de respectivamente 1,67; 0,58; 1,52; 1,15; 0,10; 0,03. Com base nos valores acima descritos se concluiu que com a utilização do NI RS pode-se predizer de forma aceitável o valor nutritivo das forragens descritas neste experimento. Abstract: The Near Infrared Spectroscopy (NIRS) technique has been shown to be adequate to predict forage's nutrient content. With the goal to evaluate the analysis system based on NIRS on the prediction of DM (Dry Matter), CP (Crude Protein), ADF (Acid Detergent Fiber), NDF (Neutral Detergent Fiber), Ca, and P it was conducted a trial on Guarapuava County, Parana State, Brazil. In this study 360 samples divided on 6 cuttings have been utilized, with intervals between cuttings of 61, 60, 57, 143, and 103 days, and cutting's height of 5 cm during a one year period. The samples were consisted of Lolium multiflorum, Trifolium repens, Trifolium pratense, and Lotus corniculatus on the winter, and native pasture on the summer. The samples were dried using a forced-air convection oven at 65°C, and they were grinded in Willey grinder with 1.0 mm screen. The absorption spectra was obtained using a NR4500 monochromator instrument from NIRSystems, and the measurements were transformed into log of inverse reflectance (log(1/R)). For the purpose of calibration 27, 31, 30, 29, 29 and 29 samples were selected and, after calibration, the regression coefficients (R ) for DM, CP, ADF, NDF, Ca, and P were, respectively: 0.92, 0.92, 0.93, 0.97, 0.49, and 0.64, while the calibration's standard error were, respectively: 1.67, 0.58, 1.52, 1.15, 0.10, and 0.03. Based on the results of this experiment it was concluded that NIRS analysis can predict nutrient quality of the forages selected on this trial.