Abstract: Recently, the self-attention mechanism (Transformer) has shown its advantages in various natural language processing (NLP) tasks. Since positional information is crucial to NLP tasks, the ...

Most languages use word position and sentence structure to extract meaning. For example, "The cat sat on the box," is not the same as "The box was on the cat." Over a long text, like a financial ...

*Our UCPE introduces a geometry-consistent alternative to Plücker rays as one of the core contributions, enabling better generalization in Transformers. We hope to inspire future research on ...

Instead of using RoPE’s low-dimensional limited rotations or ALiBi’s 1D linear bias, FEG builds position encoding on a higher-dimensional geometric structure. The idea is simple at a high level: Treat ...

One of the best scenes in the movie “Moneyball” is when Billy Beane and Peter Brand go to the parking garage to discuss Brand’s work for the Cleveland front office. Eventually, Brand goes on a tangent ...

As Large Language Models (LLMs) are widely used for tasks like document summarization, legal analysis, and medical history evaluation, it is crucial to recognize the limitations of these models. While ...

Spiking neural networks (SNNs) are bio-inspired networks that mimic how neurons in the brain communicate through discrete spikes, which have great potential in various tasks due to their energy ...