{"id":3100,"date":"2020-10-02T22:10:51","date_gmt":"2020-10-02T22:10:51","guid":{"rendered":"https:\/\/techclot.com\/index.php\/2020\/10\/02\/google-cambridge-deepmind-alan-turing-institutes-performer-transformer-slashes-compute\/"},"modified":"2020-10-02T22:10:51","modified_gmt":"2020-10-02T22:10:51","slug":"google-cambridge-deepmind-alan-turing-institutes-performer-transformer-slashes-compute","status":"publish","type":"post","link":"https:\/\/techclot.com\/index.php\/2020\/10\/02\/google-cambridge-deepmind-alan-turing-institutes-performer-transformer-slashes-compute\/","title":{"rendered":"Google, Cambridge, DeepMind & Alan Turing Institute’s ‘Performer’ Transformer Slashes Compute …"},"content":{"rendered":"

Google, Cambridge, DeepMind & Alan Turing Institute’s ‘Performer’ Transformer Slashes Compute …<\/a><\/p>\n

It\u2019s no coincidence that Transformer neural network architecture is gaining popularity across so many machine learning research fields. Best known for natural language processing (NLP) tasks, Transformers not only enabled OpenAI\u2019s 175 billion parameter language model GPT-3<\/a> to deliver SOTA performance, the power- and potential-packed architecture also helped DeepMind\u2019s AlphaStar<\/a> bot defeat professional StarCraft players. Researchers have now introduced a way to make Transformers more compute-efficient, scalable and accessible.<\/p>\n

While previous learning approaches such as RNNs suffered from vanishing gradient problems, Transformers\u2019 game-changing self-attention mechanism eliminated such issues. As explained in the paper introducing Transformers \u2014 Attention Is All You Need<\/em><\/a>, the novel architecture is based on a trainable attention mechanism<\/strong> that identifies complex dependencies between input sequence elements.<\/p>\n

Transformers however scale quadratically when the number of tokens in an input sequence increases, making their use prohibitively<\/strong>expensive<\/strong> for large numbers of tokens. Even when fed with moderate token inputs, Transformers\u2019 gluttonous appetite for computational resources can be difficult for many researchers to satisfy.<\/p>\n

A team from Google, University of Cambridge, DeepMind, and Alan Turing Institute <\/strong>have proposed a new type of Transformer dubbed Performer<\/strong>, based on a F<\/strong>ast A<\/strong>ttention V<\/strong>ia positive O<\/strong>rthogonal R<\/strong>andom features (FAVOR+<\/strong>) backbone mechanism. The team designed Performer to be \u201ccapable of provably accurate and practical estimation of regular (softmax) full rank attention, but of only linear space and timely complexity and not relying on any priors such as sparsity or low-rankness.\u201d<\/p>\n

\n
\"image.png\"<\/figure>\n<\/div>\n

Softmax has been a bottleneck burdening attention-based Transformers computation. Transformers typically use a learned linear transformation and softmax function to convert decoder output to predicted next-token probabilities. The proposed method instead estimates softmax and Gaussian kernels with positive orthogonal random features for a robust and unbiased estimation of regular softmax attention in the FAVOR+ mechanism. The research confirms that using positive features can efficiently train softmax-based linear Transformers.<\/strong><\/p>\n

\n
\"image.png\"<\/figure>\n<\/div>\n
\n
\"image.png\"<\/figure>\n<\/div>\n
\n
\"image.png\"<\/figure>\n<\/div>\n

Leveraging detailed mathematical theorems, the paper demonstrates that rather than relying solely on computational resources to boost performance, it is also possible to develop improved and efficient Transformer architectures that have significantly lower energy consumption. Also, because Performers use the same training hyperparameters as Transformers, the FAVOR+ mechanism can function as a simple drop-in without much tuning.<\/p>\n

The team tested Performers on a rich set of tasks ranging from pixel-prediction to protein sequence modelling. In their experimental setup, a Performer only replaced a regular Transformer\u2019s attention component with the FAVOR+ mechanism. On the challenging task of training a 36-layer model using protein sequences, the Performer-based model (Performer-RELU) achieved better performance than the baseline Transformer models Reformer and Linformer, which showed significant drops in accuracy. On the standard ImageNet64 benchmark, a Performer with six layers matched the accuracy of a Reformer with 12 layers. After optimizations, Performer was also twice as fast as Reformer.<\/p>\n

Because Performer-enabled scalable Transformer architectures can handle much longer sequences without constraints on attention mechanism structure while remaining accurate and robust, it is believed they could lead to breakthroughs in bioinformatics, where technologies such as such as language modelling for proteins have already shown strong potential.<\/p>\n

The paper Rethinking Attention With Performers <\/em>is on arXiv<\/a>.<\/p>\n

\n

Reporter<\/strong>: Fangyu Cai | Editor<\/strong>: Michael Sarazen<\/p>\n

\n
\n
\"B4.png\"<\/figure>\n<\/div>\n

Synced Report | <\/strong>A Survey of China\u2019s Artificial Intelligence Solutions in Response to the COVID-19 Pandemic \u2014 87 Case Studies from 700+ AI Vendors<\/strong><\/a><\/p>\n

This report offers a look at how China has leveraged artificial intelligence technologies in the battle against COVID-19. It is also available on Amazon Kindle<\/a>. Along with this report, we also introduced a <\/strong>database<\/strong><\/a> covering additional 1428 artificial intelligence solutions from 12 pandemic scenarios.<\/strong><\/p>\n

Click here<\/a> to find more reports from us.<\/p>\n

\n
\n
\"AI<\/figure>\n<\/div>\n

We know you don\u2019t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly<\/a><\/em> to get weekly AI updates.<\/strong><\/p>\n

\n
\n

Share this:<\/h3>\n<\/div>\n<\/div>\n
\n

Like this:<\/h3>\n
Like<\/span><\/span> Loading…<\/span><\/div>\n

<\/span><\/div>\n<\/p>\n

Published at Fri, 02 Oct 2020 20:03:45 +0000<\/p>\n

Amesite CEO Dr. Ann Marie Sastry Scheduled to Appear on Fox Business Network’s Mornings With …<\/a><\/p>\n

ANN ARBOR, Mich.<\/span>, Oct. 2, 2020<\/span> \/PRNewswire\/ — Amesite<\/b> Inc<\/b>. (Nasdaq: AMST<\/a>), an artificial intelligence software company providing online learning ecosystems for business, higher education, and K-12, announced today its CEO, Dr. Ann Marie Sastry<\/span>, is scheduled to appear on Fox Business Network’s Mornings With Maria<\/i> Monday morning at 6:30 a.m. ET<\/span>.  <\/p>\n

Dr. Sastry will discuss the latest in remote learning and how it is impacting students, parents, teachers, and administrators. She will also explain how artificial intelligence is helping to power Amesite’s next generation online learning platform and why innovation in the EdTech space has become so crucial. <\/p>\n

About Amesite Inc.<\/b><\/p>\n

Amesite is a high tech artificial intelligence software company offering a cloud-based platform and content creation services for K-12, college, university and business education and upskilling. Amesite-offered courses and programs are branded to our customers.  Amesite uses artificial intelligence technologies to provide customized environments for learners, easy-to-manage interfaces for instructors, and greater accessibility for learners in the US education market and beyond.  The Company leverages existing institutional infrastructures, adding mass customization and cutting-edge technology to provide cost-effective, scalable and engaging experiences for learners anywhere.  For more information, visit https:\/\/amesite.com<\/a><\/b>.<\/p>\n

Forward Looking Statements<\/u><\/b><\/p>\n

This communication contains forward-looking statements (including within the meaning of Section 21E of the Securities Exchange Act of 1934, as amended, and Section 27A of the Securities Act of 1933, as amended) concerning the Company, the Company’s planned online machine learning platform, the Company’s business plans, any future commercialization of the Company’s online learning solutions, potential customers, business objectives and other matters. Forward-looking statements generally include statements that are predictive in nature and depend upon or refer to future events or conditions, and include words such as “may,” “will,” “should,” “would,” “expect,” “plan,” “believe,” “intend,” “look forward,” and other similar expressions among others. Statements that are not historical facts are forward-looking statements. Forward-looking statements are based on current beliefs and assumptions that are subject to risks and uncertainties and are not guarantees of future performance. Actual results could differ materially from those contained in any forward-looking statement. Risks facing the Company and its planned platform are set forth in the Company’s filings with the SEC. Except as required by applicable law, the Company undertakes no obligation to revise or update any forward-looking statement, or to make any other forward-looking statements, whether as a result of new information, future events or otherwise.<\/p>\n

Media Contact \u2013 Robert Busweiler<\/span> \u2013 [email protected]<\/span><\/a> \u2013 631.379.6454<\/p>\n

SOURCE Amesite Inc.<\/p>\n

<\/p>\n

Related Links<\/h4>\n

https:\/\/amesite.com<\/a><\/p>\n<\/p>\n

Published at Fri, 02 Oct 2020 18:33:45 +0000<\/p>\n","protected":false},"excerpt":{"rendered":"

Google, Cambridge, DeepMind & Alan Turing Institute’s ‘Performer’ Transformer Slashes Compute … It\u2019s no coincidence…<\/p>\n","protected":false},"author":3,"featured_media":3101,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[3],"tags":[],"class_list":["post-3100","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artificial-intelligence"],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/techclot.com\/wp-content\/uploads\/2020\/10\/image-1.png?fit=1142%2C302&ssl=1","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p3orZX-O0","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/techclot.com\/index.php\/wp-json\/wp\/v2\/posts\/3100","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techclot.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techclot.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techclot.com\/index.php\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/techclot.com\/index.php\/wp-json\/wp\/v2\/comments?post=3100"}],"version-history":[{"count":0,"href":"https:\/\/techclot.com\/index.php\/wp-json\/wp\/v2\/posts\/3100\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techclot.com\/index.php\/wp-json\/wp\/v2\/media\/3101"}],"wp:attachment":[{"href":"https:\/\/techclot.com\/index.php\/wp-json\/wp\/v2\/media?parent=3100"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techclot.com\/index.php\/wp-json\/wp\/v2\/categories?post=3100"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techclot.com\/index.php\/wp-json\/wp\/v2\/tags?post=3100"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}