Home
Technology
Gaming
Entertainment
World & Business
Science
Sports
AI

World & Business

World & Business

Home
Technology
Gaming
Entertainment
World & Business
Science
Sports
AI

Extreme activation sparsification is claimed to use fewer FLOPs · Digg

Extreme activation sparsification is claimed to use fewer FLOPs

A post highlights fewer floating-point operations as a benefit of extreme activation sparsification.

TLDR

A post claims extreme activation sparsification uses fewer FLOPs, or floating-point operations, but does not say how many fewer.

Combined views

2.7K

3 Sources, first seen 2h ago

19 likes

Combined views

2.7K

3 Sources, first seen 2h ago

19 likes

François Fleuret

2h ago

first seen 2h ago

2 comments

4 saves

12 reposts

Featured Source

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

2 comments

4 saves

12 reposts

3 Sources

@MariaLomeli_🚨Our new paper is out: Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs We present model casting, a mid-training recipe that drastically sparsifies the activations within the FeedForward Network layer of a transformer. Model casting consists of switching to an activation that outputs zeros for all negative inputs (R-S+) and using an L1 loss to set the required sparsity as a target at training time.

@francoisfleuretLess flops with extreme activation sparsification!

@WeijiaShi2RT @MariaLomeli_: 🚨Our new paper is out: Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs We present model cas…

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

2601234567890123456789

2601234567890123456789

Today's Rank

#5

Today's Rank

#5

3 Sources

@MariaLomeli_🚨Our new paper is out: Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs We present model casting, a mid-training recipe that drastically sparsifies the activations within the FeedForward Network layer of a transformer. Model casting consists of switching to an activation that outputs zeros for all negative inputs (R-S+) and using an L1 loss to set the required sparsity as a target at training time.

@francoisfleuretLess flops with extreme activation sparsification!

@WeijiaShi2RT @MariaLomeli_: 🚨Our new paper is out: Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs We present model cas…

Related

A year in isolation sparks a dog-and-AI joke about companionship

A post asks readers to choose a companion for a year with necessities provided but no contact with the outside world. A quote-post answers with a pun about a foundation model and a dog.

Researcher Shares YouTube Video on Machine Learning