RecurrentGemma: Moving Past Transformers for Efficient Open Language Models

Authors: Botev, Aleksandar, De, Soham, Smith, Samuel L, Fernando, Anushan, Muraru, George-Cristian, Haroun, Ruba, Berrada, Leonard, Pascanu, Razvan, Sessa, Pier Giuseppe, Dadashi, Robert, Hussenot, Léonard, Ferret, Johan, Girgin, Sertan, Bachem, Olivier, Andreev, Alek, Kenealy, Kathleen, Mesnard, Thomas, Hardin, Cassidy, Bhupatiraju, Surya, Pathak, Shreya, Sifre, Laurent, Rivière, Morgane, Kale, Mihir Sanjay, Love, Juliette, Tafti, Pouya, Joulin, Armand, Fiedel, Noah, Senter, Evan, Chen, Yutian, Srinivasan, Srivatsan, Desjardins, Guillaume, Budden, David, Doucet, Arnaud, Vikram, Sharad, Paszke, Adam, Gale, Trevor, Borgeaud, Sebastian, Chen, Charlie, Brock, Andy, Paterson, Antonia, Brennan, Jenny, Risdal, Meg, Gundluru, Raj, Devanathan, Nesh, Mooney, Paul, Chauhan, Nilay, Culliton, Phil, Martins, Luiz Gustavo, Bandy, Elisa, Huntsperger, David, Cameron, Glenn, Zucker, Arthur, Warkentin, Tris, Peran, Ludovic, Giang, Minh, Ghahramani, Zoubin, Farabet, Clément, Kavukcuoglu, Koray, Hassabis, Demis, Hadsell, Raia, Teh, Yee Whye, de Frietas, Nando

Subject Terms: Computer Science - Machine Learning, Computer Science - Artificial Intelligence, Computer Science - Computation and Language

Access URL: http://arxiv.org/abs/2404.07839

Gemma: Open Models Based on Gemini Research and Technology

Authors: Gemma Team, Mesnard, Thomas, Hardin, Cassidy, Dadashi, Robert, Bhupatiraju, Surya, Pathak, Shreya, Sifre, Laurent, Rivière, Morgane, Kale, Mihir Sanjay, Love, Juliette, Tafti, Pouya, Hussenot, Léonard, Sessa, Pier Giuseppe, Chowdhery, Aakanksha, Roberts, Adam, Barua, Aditya, Botev, Alex, Castro-Ros, Alex, Slone, Ambrose, Héliou, Amélie, Tacchetti, Andrea, Bulanova, Anna, Paterson, Antonia, Tsai, Beth, Shahriari, Bobak, Lan, Charline Le, Choquette-Choo, Christopher A., Crepy, Clément, Cer, Daniel, Ippolito, Daphne, Reid, David, Buchatskaya, Elena, Ni, Eric, Noland, Eric, Yan, Geng, Tucker, George, Muraru, George-Christian, Rozhdestvenskiy, Grigory, Michalewski, Henryk, Tenney, Ian, Grishchenko, Ivan, Austin, Jacob, Keeling, James, Labanowski, Jane, Lespiau, Jean-Baptiste, Stanway, Jeff, Brennan, Jenny, Chen, Jeremy, Ferret, Johan, Chiu, Justin, Mao-Jones, Justin, Lee, Katherine, Yu, Kathy, Millican, Katie, Sjoesund, Lars Lowe, Lee, Lisa, Dixon, Lucas, Reid, Machel, Mikuła, Maciej, Wirth, Mateo, Sharman, Michael, Chinaev, Nikolai, Thain, Nithum, Bachem, Olivier, Chang, Oscar, Wahltinez, Oscar, Bailey, Paige, Michel, Paul, Yotov, Petko, Chaabouni, Rahma, Comanescu, Ramona, Jana, Reena, Anil, Rohan, McIlroy, Ross, Liu, Ruibo, Mullins, Ryan, Smith, Samuel L, Borgeaud, Sebastian, Girgin, Sertan, Douglas, Sholto, Pandya, Shree, Shakeri, Siamak, De, Soham, Klimenko, Ted, Hennigan, Tom, Feinberg, Vlad, Stokowiec, Wojciech, Chen, Yu-hui, Ahmed, Zafarali, Gong, Zhitao, Warkentin, Tris, Peran, Ludovic, Giang, Minh, Farabet, Clément, Vinyals, Oriol, Dean, Jeff, Kavukcuoglu, Koray, Hassabis, Demis, Ghahramani, Zoubin, Eck, Douglas, Barral, Joelle, Pereira, Fernando, Collins, Eli, Joulin, Armand, Fiedel, Noah, Senter, Evan, Andreev, Alek, Kenealy, Kathleen

Subject Terms: Computer Science - Computation and Language, Computer Science - Artificial Intelligence

Access URL: http://arxiv.org/abs/2403.08295

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

Authors: De, Soham, Smith, Samuel L., Fernando, Anushan, Botev, Aleksandar, Cristian-Muraru, George, Gu, Albert, Haroun, Ruba, Berrada, Leonard, Chen, Yutian, Srinivasan, Srivatsan, Desjardins, Guillaume, Doucet, Arnaud, Budden, David, Teh, Yee Whye, Pascanu, Razvan, De Freitas, Nando, Gulcehre, Caglar

Subject Terms: Computer Science - Machine Learning, Computer Science - Computation and Language

Access URL: http://arxiv.org/abs/2402.19427

ConvNets Match Vision Transformers at Scale

Authors: Smith, Samuel L., Brock, Andrew, Berrada, Leonard, De, Soham

Subject Terms: Computer Science - Computer Vision and Pattern Recognition, Computer Science - Machine Learning, Computer Science - Neural and Evolutionary Computing

Access URL: http://arxiv.org/abs/2310.16764

Unlocking Accuracy and Fairness in Differentially Private Image Classification

Authors: Berrada, Leonard, De, Soham, Shen, Judy Hanwen, Hayes, Jamie, Stanforth, Robert, Stutz, David, Kohli, Pushmeet, Smith, Samuel L., Balle, Borja

Subject Terms: Computer Science - Machine Learning, Computer Science - Computer Vision and Pattern Recognition, Computer Science - Computers and Society

Access URL: http://arxiv.org/abs/2308.10888

Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues

Authors: Orvieto, Antonio, De, Soham, Gulcehre, Caglar, Pascanu, Razvan, Smith, Samuel L.

Subject Terms: Computer Science - Machine Learning, Computer Science - Neural and Evolutionary Computing

Access URL: http://arxiv.org/abs/2307.11888

Resurrecting Recurrent Neural Networks for Long Sequences

Authors: Orvieto, Antonio, Smith, Samuel L, Gu, Albert, Fernando, Anushan, Gulcehre, Caglar, Pascanu, Razvan, De, Soham

Subject Terms: Computer Science - Machine Learning

Access URL: http://arxiv.org/abs/2303.06349

Differentially Private Diffusion Models Generate Useful Synthetic Images

Authors: Ghalebikesabi, Sahra, Berrada, Leonard, Gowal, Sven, Ktena, Ira, Stanforth, Robert, Hayes, Jamie, De, Soham, Smith, Samuel L., Wiles, Olivia, Balle, Borja

Subject Terms: Computer Science - Machine Learning, Computer Science - Cryptography and Security, Computer Science - Computer Vision and Pattern Recognition, Statistics - Machine Learning

Access URL: http://arxiv.org/abs/2302.13861

Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation

Authors: He, Bobby, Martens, James, Zhang, Guodong, Botev, Aleksandar, Brock, Andrew, Smith, Samuel L, Teh, Yee Whye

Subject Terms: Computer Science - Machine Learning, Computer Science - Artificial Intelligence, Computer Science - Computation and Language, Statistics - Machine Learning

Access URL: http://arxiv.org/abs/2302.10322

Unlocking High-Accuracy Differentially Private Image Classification through Scale

Authors: De, Soham, Berrada, Leonard, Hayes, Jamie, Smith, Samuel L., Balle, Borja

Subject Terms: Computer Science - Machine Learning, Computer Science - Cryptography and Security, Computer Science - Computer Vision and Pattern Recognition, Statistics - Machine Learning

Access URL: http://arxiv.org/abs/2204.13650

Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error

Authors: Fort, Stanislav, Brock, Andrew, Pascanu, Razvan, De, Soham, Smith, Samuel L.

Subject Terms: Computer Science - Machine Learning, Computer Science - Computer Vision and Pattern Recognition

Access URL: http://arxiv.org/abs/2105.13343

High-Performance Large-Scale Image Recognition Without Normalization

Authors: Brock, Andrew, De, Soham, Smith, Samuel L., Simonyan, Karen

Subject Terms: Computer Science - Computer Vision and Pattern Recognition, Computer Science - Machine Learning, Statistics - Machine Learning

Access URL: http://arxiv.org/abs/2102.06171

On the Origin of Implicit Regularization in Stochastic Gradient Descent

Authors: Smith, Samuel L., Dherin, Benoit, Barrett, David G. T., De, Soham

Subject Terms: Computer Science - Machine Learning, Statistics - Machine Learning

Access URL: http://arxiv.org/abs/2101.12176

Characterizing signal propagation to close the performance gap in unnormalized ResNets

Authors: Brock, Andrew, De, Soham, Smith, Samuel L.

Subject Terms: Computer Science - Machine Learning, Computer Science - Computer Vision and Pattern Recognition, Statistics - Machine Learning

Access URL: http://arxiv.org/abs/2101.08692

Cold Posteriors and Aleatoric Uncertainty

Authors: Adlam, Ben, Snoek, Jasper, Smith, Samuel L.

Source: ICML workshop on Uncertainty and Robustness in Deep Learning (2020)

Subject Terms: Statistics - Machine Learning, Computer Science - Machine Learning

Access URL: http://arxiv.org/abs/2008.00029

On the Generalization Benefit of Noise in Stochastic Gradient Descent

Authors: Smith, Samuel L., Elsen, Erich, De, Soham

Subject Terms: Computer Science - Machine Learning, Statistics - Machine Learning

Access URL: http://arxiv.org/abs/2006.15081

Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks

Authors: De, Soham, Smith, Samuel L.

Subject Terms: Computer Science - Machine Learning, Computer Science - Computer Vision and Pattern Recognition, Statistics - Machine Learning

Access URL: http://arxiv.org/abs/2002.10444

The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study

Authors: Park, Daniel S., Sohl-Dickstein, Jascha, Le, Quoc V., Smith, Samuel L.

Subject Terms: Computer Science - Machine Learning, Computer Science - Artificial Intelligence, Computer Science - Computer Vision and Pattern Recognition, Statistics - Machine Learning

Access URL: http://arxiv.org/abs/1905.03776

Stochastic natural gradient descent draws posterior samples in function space

Authors: Smith, Samuel L., Duckworth, Daniel, Rezchikov, Semon, Le, Quoc V., Sohl-Dickstein, Jascha

Subject Terms: Computer Science - Machine Learning, Computer Science - Artificial Intelligence, Statistics - Machine Learning

Access URL: http://arxiv.org/abs/1806.09597

Decoding Decoders: Finding Optimal Representation Spaces for Unsupervised Similarity Tasks

Authors: Zhelezniak, Vitalii, Busbridge, Dan, Shen, April, Smith, Samuel L., Hammerla, Nils Y.

Subject Terms: Computer Science - Artificial Intelligence, Computer Science - Computation and Language, Computer Science - Learning

Access URL: http://arxiv.org/abs/1805.03435