Thursday, June 22, 2017

Videos: Structured Regularization for High-Dimensional Data Analysis: Submodular Functions,computing the Non-computable via sparsity, the SDP approach to graph clustering, the hidden clique problem, robust deconvolution




On foundational computation problems in l1 and TV regularization
A.Hansen - 3/4 - 20/06/2017



Computing the Non-computable via sparsity
On foundational computation barriers in l1 and TV regularization
A.Hansen - 4/4 - 20/06/2017


Submodular Functions: from Discrete to Continuous Domains , Francis Bach (INRIA)

Abstract: Submodular set-functions have many applications in combinatorial optimization, as they can be minimized and approximately maximized in polynomial time. A key element in many of the algorithms and analyses is the possibility of extending the submodular set-function to a convex function, which opens up tools from convex optimization. Submodularity goes beyond set-functions and has naturally been considered for problems with multiple labels or for functions defined on continuous domains, where it corresponds essentially to cross second-derivatives being nonpositive. In this talk, I will show that most results relating submodularity and convexity for set-functions can be extended to all submodular functions. In particular, (a) I will naturally define a continuous extension in a set of probability measures, (b) show that the extension is convex if and only if the original function is submodular, (c) prove that the problem of minimizing a submodular function is equivalent to a typically non-smooth convex optimization problem. Most of these extensions from the set-function situation are obtained by drawing links with the theory of multi-marginal optimal transport, which provides also a new interpretation of existing results for set-functions. I will then provide practical algorithms to minimize generic submodular functions on discrete domains, with associated convergence rates, and an application to proximal operators for non-convex penalty functions. Preprint available here.



Andrea Montanari (Stanford): The semidefinite programming approach to graph clustering.

Andrea Montanari (Stanford): Local algorithms and graphical models. The hidden clique problem.

Carlos Fernandez-Granda (NYU): A sampling theorem for robust deconvolution

Abstract: In the 70s and 80s geophysicists proposed using l1-norm regularization for deconvolution problem in the context of reflection seismology. Since then such methods have had a great impact in high-dimensional statistics and in signal-processing applications, but until recently their performance on the original deconvolution problem was not well understood theoretically. In this talk we provide an analysis of optimization-based methods for the deconvolution problem, including results on irregular sampling and sparse corruptions that highlight the modeling flexibility of these techniques.


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Wednesday, June 21, 2017

Videos: Structured Regularization for High-Dimensional Data Analysis: Compressed Sensing: Structure and Imaging & Matrix and graph estimation



Lectures 1: Compressed Sensing: Structure and Imaging
   
 Lectures 2: Compressed Sensing: Structure and Imaging Anders Hansen (Cambridge) 

Lectures 1 and 2: Compressed Sensing: Structure and Imaging Abstract: The above heading is the title of a new book to be published by Cambridge University Press. In these lectures I will cover some of the main issues discussed in this monograph/textbook. In particular, we will discuss how the key to the success of compressed sensing applied in imaging lies in the structure. For example images are not just sparse in an X-let expansion, they have a very specific sparsity structure in levels according to the X-let scales. Similarly, when considering Total Variation, the gradient coefficients are also highly structured. Moreover, in most realistic sampling scenarios, the sampling operator combined with any X-let transform yields a matrix with a very specific coherence structure. The key to successfully use compressed sensing is therefore to understand how to utilise these structures in an optimal way, in particular in the sampling procedure. In addition, as the coherence and sparsity structures have very particular asymptotic behaviour, the performance of compressed sensing varies greatly with dimension, and so does the optimal way of sampling. Fortunately, there is now a developed theory that can guide the user in detail on how to optimise the use of compressed sensing in inverse and imaging problems. I will cover several of the key aspects of the theory accompanied with real-world examples from Magnetic Resonance Imaging (MRI), Nuclear Magnetic Resonance (NMR), Surface Scattering, Electron Microscopy, Fluorescence Microscopy etc. Recommended readings: (lectures 1 and 2) Chapter 4, 6 and 12 in “A mathematical introduction to compressed sensing” (Foucard/Rauhut) Breaking the coherence barrier: A new theory for compressed sensing On asymptotic structure in compressed sensing Structure dependent sampling in compressed sensing: theoretical guarantees for tight frames
   

Andrea Montanari (Stanford): Matrix and graph estimation 

 Abstract: Many statistics and unsupervised learning problems can be formalized as estimating a structured matrix or a graph from noisy or incomplete observations. These problems present a large variety of challenges, and an intriguing interplay between computational and statistical barriers. I will provide an introduction to recent work in the area, with an emphasis on general methods and unifying themes. 1) Random matrix theory and spectral methods. 2) The semidefinite programming approach to graph clustering. 3) Local algorithms and graphical models. The hidden clique problem. 4) Non-negative matrix factorization.






Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Tuesday, June 20, 2017

Learning Deep ResNet Blocks Sequentially using Boosting Theory

Here is another way of cutting down Deep Neural Nets training time.

Learning Deep ResNet Blocks Sequentially using Boosting Theory by Furong Huang, Jordan Ash, John Langford, Robert Schapire
Deep neural networks are known to be difficult to train due to the instability of back-propagation. A deep \emph{residual network} (ResNet) with identity loops remedies this by stabilizing gradient computations. We prove a boosting theory for the ResNet architecture. We construct T weak module classifiers, each contains two of the T layers, such that the combined strong learner is a ResNet. Therefore, we introduce an alternative Deep ResNet training algorithm, \emph{BoostResNet}, which is particularly suitable in non-differentiable architectures. Our proposed algorithm merely requires a sequential training of T "shallow ResNets" which are inexpensive. We prove that the training error decays exponentially with the depth T if the \emph{weak module classifiers} that we train perform slightly better than some weak baseline. In other words, we propose a weak learning condition and prove a boosting theory for ResNet under the weak learning condition. Our results apply to general multi-class ResNets. A generalization error bound based on margin theory is proved and suggests ResNet's resistant to overfitting under network with l1 norm bounded weights.  






Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Monday, June 19, 2017

Job; Postdoc Opening in Statistical Learning and Optimization, USC

Jason just sent me the following last week:


Dear Igor, 
I am looking for a postdoc. Could you please post the following on Nuit Blanche? BTW, I greatly enjoy reading your blog.

Postdoc Opening in Statistical Learning and Optimization 
Applications are invited for a postdoc position to work with Jason Lee at the University of Southern California. Prospective applicants should have a PhD in statistics, machine learning, signal processing, or applied mathematics. The successful candidate will have the flexibility to choose a focus within non-convex optimization, deep learning, and statistical inference and learning theory. Applications from candidates with a strong background in Optimization, Statistics, and Machine Learning are particularly welcome. 
Applicants are requested to send a CV, 3 representative papers, and the contact details of three references. 
Please send applications and informal inquiries to jasonlee(AT)marshall(DOT)usc(DOT)edu.

Best,
Jason


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Job: Lecturer / Senior Lecturer in Machine Learning and Computational Intelligence, University of Surrey, UK

Mark just me the folloing last week:

Dear Igor,
This job opportunity may interest some readers of Nuit Blanche?
Best wishes,
Mark
---
Lecturer / Senior Lecturer in Machine Learning and Computational Intelligence
Department of Computer Science, University of Surrey
Salary: GBP 39,324 to GBP 57,674 per annum
Closing Date: Wednesday 28 June 2017
The Department of Computer Science at the University of Surrey invites applications for a post of Lecturer/Senior Lecturer (similar to Assistant/Associate Professor) in Machine Learning and Computational Intelligence. We aim to attract outstanding candidates to the Nature Inspired Computing and Engineering (NICE) Group, who will have strong visions for research, a growing international research profile, a passion for teaching, and who value collaborative research and working in a team. This is an exciting opportunity in a department that is growing its reputation for delivering quality interdisciplinary and applied research based on strong foundations.
The post-holder will enhance or complement one or more of the following areas: evolutionary computation, computational intelligence, machine learning, and computational neuroscience, with applications to data-driven optimisation, data analytics and big data, secure machine learning, self-organising and autonomous systems, healthcare and bioinformatics. It is expected that the post-holder will also contribute to high quality teaching at undergraduate and post-graduate level, for example in data science.
Applicants to the post should have a PhD in a relevant subject or equivalent professional experience. An ability to produce high quality outputs is also required. The appointed candidate will be expected to contribute to all aspects of the Department's activities.
We are looking for individuals that can inspire students through their curiosity for leading-edge aspects of technology. In particular, the teaching duties of the role include delivering high quality teaching to all levels of students, supervising undergraduate project students and postgraduate dissertations and contributing to the teaching of computational intelligence, machine learning, data science, as well as other practical areas of Computer Science, such as object-oriented programming and advanced algorithms.
The Department of Computer Science embodies the ethos of "applying theory into practice" across its research and teaching activities. Its research activities are focused into two research groups: Nature Inspired Computing and Engineering (NICE) and Secure Systems. The research on evolutionary optimization and computational intelligence is internationally recognized and has a close collaboration across the University and internationally with academia and industry. The department has excellent links with departments across the University, as well as with industry, such as Honda, The Pirbright Institute, National Physical Laboratory, and Moorfields Eye Hospital.
This is a full-time and permanent position. The post is available from September 2017 but there is flexibility in the start date. The University is located close to the Surrey Research Park in Guildford and within easy commuting distance of London.
For further details and an informal discussion please contact Professor Yaochu Jin, Head of the NICE Group, yaochu.jin@surrey.ac.uk, or Professor Mark Plumbley, Head of Department at m.plumbley@surrey.ac.uk.
The University and the Department specifically are committed to building a culturally diverse organisation and strongly encourages applications from female and minority candidates. The Department shares the Athena SWAN ideals with respect to the equality and diversity agenda.
Presentations and interviews will take place on 13 July 2017. During the interview process there will be an opportunity to meet members of the NICE group in the evening of 12 July 2017 for an informal dinner and to meet with other colleagues within the Department on an informal basis.
For further details and information on how to apply, visit https://jobs.surrey.ac.uk/039217
--
Prof Mark D Plumbley
Interim Head of Department of Computer Science
Professor of Signal Processing
Centre for Vision, Speech and Signal Processing (CVSSP)
University of Surrey - Celebrating 50 years in Guildford
Email (Head of Department matters): cs-hod@list.surrey.ac.uk
Email (Other matters): m.plumbley@surrey.ac.uk
University of Surrey, Guildford, Surrey, GU2 7XH, UK



Sunday, June 18, 2017

Sunday Morning Videos: NIPS 2016 workshop on nonconvex optimizations.



Hossein just mentioned this on his twitter:

Here is the list of talks and videos:
  • Nando de Freitas (Learning to Optimize) [Slides] [Video]
  • Morning Poster Spotlight (papers #1 to #8) [Slides] [Video]
  • Jean Lasserre (Moment-LP and Moment-SOS Approaches in Optimization) [Slides] [Video]
  • Surya Ganguli (Non-convexity in the error landscape and the expressive capacity of deep neural networks) [Slides] [Video
  • Ryan Adams (Leveraging Structure in Bayesian Optimization) [Slides] [Video]
  • Stefanie Jegelka (Submodular Optimization and Nonconvexity) [Slides] [Video]
  • Suvrit Sra (Taming Non-Convexity via Geometry) [Slides] [Video]
  • Francis Bach (Submodular Functions: from Discrete to Continuous Domains) [Slides] [Video]
  • Panel Discussion [Video]
  • Afternoon Poster Spotlight (papers #9 to #16) [Slides] [Video]





Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Friday, June 16, 2017

FreezeOut: Accelerate Training by Progressively Freezing Layers - implementation -

What initially looks like playing with hyperparameters brings new life to a somewhat older approach. From Alex's tweet:







The early layers of a deep neural net have the fewest parameters, but take up the most computation. In this extended abstract, we propose to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass. We empirically demonstrate that FreezeOut yields savings of up to 20% wall-clock time during training with 3% loss in accuracy for DenseNets on CIFAR.


DenseNet is at: http://github.com/bamos/densenet.pytorch 
while FreezeOut is here: http://github.com/ajbrock/FreezeOut 




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Thursday, June 15, 2017

Self-Normalizing Neural Networks - implementation -

There is some commotion on the interweb about this new nonlinear activation function that removes the need for tricks like Batch Normalization and seem to beat recent architectures like Resnets ...and there is the long appendix thingy too. Wow !




Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -- even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs and other machine learning methods such as random forests and support vector machines. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep. Implementations are available at: github.com/bioinf-jku/SNNs.
Some mention on the web:


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Wednesday, June 14, 2017

CSjob: Postdoc, Optimisation for Matrix Factorisation, Toulouse, France

Cedric came to visit us yesterday at LightOn and he let us know about his search for a postdoc, here is the announcement:

Postdoc opening Optimisation for matrix factorisation
Project FACTORY 
 

New paradigms for latent factor estimation Announcement Applications are invited for a 2-year postdoc position to work with Cedric Fevotte (CNRS senior scientist) on matrix factorisation techniques for data processing. The position is part of project FACTORY (New paradigms for latent factor estimation), funded by the European Research Council under a Consolidator Grant (2016-2021). The successful candidate will be based in Toulouse, France.  
Project description  
The project concerns matrix factorisation and dictionary learning for data analysis at large, with an emphasis on statistical estimation in mean-parametrised exponential models, non-convex optimisation, stochastic algorithms & approximate inference, representation learning, and applications to audio signal processing, remote sensing & data mining.
The European Research Council offers highly competitive funding for scientific excellence. The successful candidate will enjoy an inspiring and resourceful environment, with the possibility of travelling to conferences and visiting other national or international labs.
More information at http://www.irit.fr/~Cedric.Fevotte/factory/  
Host institution and place of work 
The successful candidate will be employed by the Centre National de la Recherche Scientifique (CNRS, the National Center for Scientific Research). CNRS is the largest state-funded research organisation in France, involved in all scientific fields. FACTORY is hosted by the Institut de Recherche en Informatique de Toulouse (IRIT), a joint laboratory of CNRS and Toulouse universities & engineering schools. IRIT is among the largest computer & information sciences labs in France. Toulouse is the fourth largest city in France, the capital of the Midi-Pyr´en´ees region in the South-West of France, and is praised for its high quality of living. The physical location for the project is the ENSEEIHT campus (Signal & Communications group), in a lively neighbourhood of the city center.  
Candidate profile and application  

Prospective applicants should have a PhD in machine learning, signal processing, applied mathematics, statistics, or a related discipline, good programming skills, and good communication skills in English, both written and oral. The successful candidate will have the flexibility to choose a topic within the range of the project, according to his/her experience and preferences. Applications from candidates with a good background in optimisation or stochastic simulation are particularly encouraged. The net monthly salary is 2300e for researchers with less than 2 years of professional experience after the PhD, and starts from 2700e in other cases. The position comes with health insurance & other social benefits. Applicants are requested to send a CV, a brief statement of research interests and the contact details of two referees in a single PDF file. Applications and informal enquiries are to be emailed to cedric(dot)fevotte(at)irit(dot)fr




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Tuesday, June 13, 2017

Paris Machine Learning meetup "Hors Série" #13: Automatic Machine Learning



So tonight is going to be the 13th Hors Série of Season 4 of the Paris Machine Learning meetup. This will bring about to a record 23 the number of meetups Franck and I  have made happen this season!

Tonight will be a presentation of two toolboxes around the issue of Automated Machine Learning. One toolbox is by Axel de Romblay and the other is by Alexis Bondu.

There is limited seating however we will have a streaming (see below).

Thank to Sewan for hosting us. Here is the video for the streaming:

Program:
  • 0 - Preview of the slides
    • A preview of the slides is now available:
    • - French version (link)
    • - English version (link)
  • 1  - Introduction
    • How to ?
    • What's important ?
  • 2 - Theories and Baseline to automate Machine Learning
    • Overview of the different approaches to automate Machine Learning (bayesian, ...)
  • 3 - Demo & Coding
    • Edge ML &  MLBox
    • Comparison of these tools with the same dataset
  • 3-1 - MLBox (Machine Learning Box)
    • MLBox is a powerful Automated Machine Learning python library. It provides the following functionalities:
      • - Fast reading and distributed data preprocessing / cleaning / formatting
      • - Highly robust feature selection and leak detection
      • - Accurate hyper-parameter optimization in high-dimensional space
      • - State-of-the art predictive models for classification and regression (Deep Learning, Stacking, LightGBM,...)
    • - Prediction with models interpretation
    • To learn more about this tool and how to get it installed, please refer to:
    • https://github.com/AxeldeRomblay/MLBox
  • 3.2 - Edge ML
    • Edge ML is an Automated Machine Learning software which implements the MODL approach, which is unique in two respects :
      • i) it is highly scalable and avoids empirical optimization of the models by grid-search;
      • ii) it provides accurate and very robust models.
    • To participate in the interactive demonstration, you can install Edge ML :
      • 1 - Install the shareware version: link
      • 2 - Create and download your licence file: link
      • 3 - Copy / Paste your licence file to your home directory
    • 4 - Download the Python wrapper (with demo example): link


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Monday, June 12, 2017

CATERPILLAR: Coarse Grain Reconfigurable Architecture for Accelerating the Training of Deep Neural Networks

Learning to learn with efficiency in mind.



Accelerating the inference of a trained DNN is a well studied subject. In this paper we switch the focus to the training of DNNs. The training phase is compute intensive, demands complicated data communication, and contains multiple levels of data dependencies and parallelism. This paper presents an algorithm/architecture space exploration of efficient accelerators to achieve better network convergence rates and higher energy efficiency for training DNNs. We further demonstrate that an architecture with hierarchical support for collective communication semantics provides flexibility in training various networks performing both stochastic and batched gradient descent based techniques. Our results suggest that smaller networks favor non-batched techniques while performance for larger networks is higher using batched operations. At 45nm technology, CATERPILLAR achieves performance efficiencies of 177 GFLOPS/W at over 80% utilization for SGD training on small networks and 211 GFLOPS/W at over 90% utilization for pipelined SGD/CP training on larger networks using a total area of 103.2 mm2 and 178.9 mm2 respectively.



h/t Iacopo




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Meeting: Medical imaging in the era of AI, Yonsei University, Korea


Jong just sent me the following:

Dear Igor,

I would like to bring your attention to upcoming workshop in Korea: "Medical Imaging in the era of AI." This workshop is organized in the form of "Imaging Summit" by inviting many leading researchers in the medical imaging and inverse problems.


Date: July 10th, 2017
Place: Grand Ballroom in the Commons , Yonsei University, Korea 
As you know, machine learning techniques have been investigated for various biomedical image reconstruction and inverse problems with encouraging preliminary results. Given the importance of new opportunity of machine learning for image reconstruction, the summit titled "Medical Imaging in the era of AI” is organized to devote to this great convergence between image reconstruction and machine learning.


I believe that many Nuit Blanche readers may be interested in this topic. I appreciate it if you can announce this workshop in Nuit Blanche.

Best,-Jong

Great Jong ! Here is the program.




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Saturday, June 10, 2017

Saturday Morning Video: Deep Learning Meets Sparse Coding, Chandra Sekhar Seelamantula

Chandra just sent me the following:

Hi Igor, 
I would like to draw your attention to the following work of ours: 
The lecture video related to the paper is up on youtube: 
I request you to kindly share these with Nuit Blanche readers, if possible.
Hope you find it interesting.
Thank you so much.
Best regards,
Chandra
==
Chandra, this is great ! Here is the video:






We address the problem of reconstructing sparse signals from noisy and compressive measurements using a feed-forward deep neural network (DNN) with an architecture motivated by the iterative shrinkage-thresholding algorithm (ISTA). We maintain the weights and biases of the network links as prescribed by ISTA and model the nonlinear activation function using a linear expansion of thresholds (LET), which has been very successful in image denoising and deconvolution. The optimal set of coefficients of the parametrized activation is learned over a training dataset containing measurement-sparse signal pairs, corresponding to a fixed sensing matrix. For training, we develop an efficient second-order algorithm, which requires only matrix-vector product computations in every training epoch (Hessian-free optimization) and offers superior convergence performance than gradient-descent optimization. Subsequently, we derive an improved network architecture inspired by FISTA, a faster version of ISTA, to achieve similar signal estimation performance with about 50% of the number of layers. The resulting architecture turns out to be a deep residual network, which has recently been shown to exhibit superior performance in several visual recognition tasks. Numerical experiments demonstrate that the proposed DNN architectures lead to 3 to 4 dB improvement in the reconstruction signal-to-noise ratio (SNR), compared with the state-of-the-art sparse coding algorithms.



Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Friday, June 09, 2017

CSJob: PhD position “Tensors for System Identification” at ELEC, Vrije Universiteit Brussel (VUB)

Philippe just sent me the following: 

Dear Igor,

We have an open PhD position at Vrije Universiteit Brussel, Belgium, see below. We believe the topic is relevant to your readers, as it is in the context of advanced matrix and tensor factorizations. Could you please consider posting it to Nuit Blanche?

Kind regards,

Philippe


Sure Philippe !, Here is the announcement:




PhD position “Tensors for System Identification” at ELEC, Vrije Universiteit Brussel (VUB)




A fully funded PhD position is now available at Department ELEC, Vrije Universiteit Brussel (VUB). The goal is to develop novel methods for nonlinear system identification using tensors. We will exploit the Volterra representation while aiming for interpretable block-oriented models. Tensors play a central role in understanding and taking advantage of the Volterra kernels. See our paper http://homepages.vub.ac.be/~mishteva/papers/voltpWH.pdf





We offer an attractive salary, a job in the heart of Europe, and support from a renowned research group. This is a four year PhD project, with a yearly evaluated and renewable contract. The preferred starting date is as soon as possible and no later than Oct 1, 2017.


The specific PhD topic can be adapted to the interests of the applicant. It is especially suitable for mathematicians interested in engineering applications, and for engineers interested in mathematics.


Requirements: Master’s degree in (Applied) Mathematics, Electrical Engineering, Physics, Computer Science, or a related domain. Further requirements include excellent programming skills (e.g., MATLAB) and excellent English language skills. Experience in tensor methods and system identification is an advantage, but is not required.


Please send a two-page CV and a one-page personal statement (motivation and background knowledge) as a single PDF document to philippe.dreesen@vub.ac.be. Mention “VOLT-FWO-2017N” in the email subject line. Applications received before July 15, 2017 will be given full consideration. Informal inquiries can be sent to the same email address.





Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Training Quantized Nets: A Deeper Understanding

Ah, here is some insight Christoph, Tom et al. !



Currently, deep neural networks are deployed on low-power embedded devices by first training a full-precision model using powerful computing hardware, and then deriving a corresponding low-precision model for efficient inference on such systems. However, training models directly with coarsely quantized weights is a key step towards learning on embedded platforms that have limited computing resources, memory capacity, and power consumption. Numerous recent publications have studied methods for training quantized network, but these studies have mostly been empirical. In this work, we investigate training methods for quantized neural networks from a theoretical viewpoint. We first explore accuracy guarantees for training methods under convexity assumptions. We then look at the behavior of algorithms for non-convex problems, and we show that training algorithms that exploit high-precision representations have an important annealing property that purely quantized training methods lack, which explains many of the observed empirical differences between these types of algorithms.





Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Thursday, June 08, 2017

Coreset Construction via Randomized Matrix Multiplication




Coresets are small sets of points that approximate the properties of a larger point-set. For example, given a compact set \mathcal{S} \subseteq \mathbb{R}^d, a coreset could be defined as a (weighted) subset of \mathcal{S} that approximates the sum of squared distances from \mathcal{S} to every linear subspace of \mathbb{R}^d. As such, coresets can be used as a proxy to the full dataset and provide an important technique to speed up algorithms for solving problems including principal component analysis, latent semantic indexing, etc. In this paper, we provide a structural result that connects the construction of such coresets to approximating matrix products. This structural result implies a simple, randomized algorithm that constructs coresets whose sizes are independent of the number and dimensionality of the input points. The expected size of the resulting coresets yields an improvement over the state-of-the-art deterministic approach. Finally, we evaluate the proposed randomized algorithm on synthetic and real data, and demonstrate its effective performance relative to its deterministic counterpart.




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Tuesday, June 06, 2017

Hyperparameter Optimization: A Spectral Approach




From the introduction

In this paper we introduce a new spectral approach to hyperparameter optimization based on harmonic analysis of Boolean functions. At a high level, the idea is to fit a sparse polynomial function to the discrete, high-dimensional function mapping hyperparameters to loss, and then optimize the resulting sparse polynomial. Using ideas from discrete Fourier analysis and compressed sensing, we can give provable guarantees for a sparse-recovery algorithm that admits an efficient, paralellizable implementation. Here we are concerned with the tradeoff between running time and sample complexity for learning Boolean functions f where sampling uniformly from f is very expensive. This approach appears to be new and allows us to give uniform-distribution learning algorithms for Boolean concept classes such as decision trees that match the state-of-the-art in running time and save dramatically in sample complexity.


We give a simple, fast algorithm for hyperparameter optimization inspired by techniques from the analysis of Boolean functions. We focus on the high-dimensional regime where the canonical example is training a neural network with a large number of hyperparameters. The algorithm - an iterative application of compressed sensing techniques for orthogonal polynomials - requires only uniform sampling of the hyperparameters and is thus easily parallelizable. Experiments for training deep nets on Cifar-10 show that compared to state-of-the-art tools (e.g., Hyperband and Spearmint), our algorithm finds significantly improved solutions, in some cases matching what is attainable by hand-tuning. In terms of overall running time (i.e., time required to sample various settings of hyperparameters plus additional computation time), we are at least an order of magnitude faster than Hyperband and even more so compared to Bayesian Optimization. We also outperform Random Search 5X. Additionally, our method comes with provable guarantees and yields the first quasi-polynomial time algorithm for learning decision trees under the uniform distribution with polynomial sample complexity, the first improvement in over two decades.

h/t François on Twitter



Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Monday, June 05, 2017

Kronecker Recurrent Units




Our work addresses two important issues with recurrent neural networks: (1) they are over-parameterized, and (2) the recurrence matrix is ill-conditioned. The former increases the sample complexity of learning and the training time. The latter causes the vanishing and exploding gradient problem. We present a flexible recurrent neural network model called Kronecker Recurrent Units (KRU). KRU achieves parameter efficiency in RNNs through a Kronecker factored recurrent matrix. It overcomes the ill-conditioning of the recurrent matrix by enforcing soft unitary constraints on the factors. Thanks to the small dimensionality of the factors, maintaining these constraints is computationally efficient. Our experimental results on five standard data-sets reveal that KRU can reduce the number of parameters by three orders of magnitude in the recurrent weight matrix compared to the existing recurrent models, without trading the statistical performance. These results in particular show that while there are advantages in having a high dimensional recurrent space, the capacity of the recurrent part of the model can be dramatically reduced.



Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Sunday, June 04, 2017

Book:Artificial Intelligence and Games by Georgios N. Yannakakis and Julian Togelius

From the website:

Welcome to the Artificial Intelligence and Games book. This book aims to be the first comprehensive textbook on the application and use of artificial intelligence (AI) in, and for, games. Our hope is that the book will be used by educators and students of graduate or advanced undergraduate courses on game AI as well as game AI practitioners at large.
First Public Draft
The first draft of the book is available here!
If you spot any typos or inaccurate information, disagree with parts of the text or you have suggestions for papers we should discuss or exercises (and readings) we should include please contact us via email at gameaibook [ at ] gmail [ dot ] com.
We would appreciate your feedback on this first draft by no later than June 20, 2017 so that we meet the publication deadlines.

Here is the introduction of Artificial Intelligence and Games by Georgios N. Yannakakis and Julian Togelius

Introduction 

Artificial Intelligence (AI) has seen an immense progress in recent years. This progress is a result of a vibrant and thriving research field that features an increasing number of important research areas. The success stories of AI can be experienced in our daily lives and also evidenced though its many practical applications. AI nowadays can understand images and speech, detect emotion, drive cars, search the web, support creative design, and play games, among many other tasks; for some of these tasks machines have reached human-level status. In addition to the algorithmic innovation, the progress is often attributed to increasing computational power or to hardware advancements. There is, however, a difference between what machines can do well and what humans are good at. In the early days of AI we envisaged computational systems that deliver aspects of human intelligence and achieve humanlevel problem solving or decision making skills. While these problems can be difficult for most of us they were presented to machines as a set of formal mathematical notions within rather narrow and controlled spaces. The properties of these domains collectively allowed AI to succeed. Naturally, games—especially board games—have been a popular domain for early AI attempts as they are formal and highly constrained yet complex decision making environments. Over the years the focus of much AI research has shifted to tasks that appear simple for us to do, such as remembering a face or recognizing our friend’s voice over the phone. AI researchers have been asking questions such as: How can AI detect and express emotion? How can AI educate people, be creative or artistically novel? How can AI play a game it has not seen before? How can AI learn from minimal amount of trials? How can AI feel guilt?. All these questions pose serious challenges to AI and correspond to tasks that are not easy for us to formalize or define objectively. Unsurprisingly, tasks that require relatively low cognitive effort from us often turn out to be much harder for machines to tackle. Again, games have provided a popular domain to tackle such tasks as they feature aspects of subjective nature that cannot be formalized easily. These include, for instance, the experience of play or the creative process of game design. Games have been helping AI to grow and advance since its birth. Games not only pose interesting and complex problems for AI to solve—e.g. playing a game well; they also offer a canvas for creativity and expression which is experienced by users (people or even machines!). Thus, arguably, games is a rare domain where science (problem solving) meets art and interaction: these ingredients traditionally made games a unique and favorite domain for the study of AI. But it is not only AI that is advanced through games; it is also games that are advanced through AI research. We argue that AI has been helping games to get better in several fronts: in the way we play them, in the way we understand their inner functionalities, in the way we design them, in the way we understand play, interaction and creativity. This book is dedicated to the healthy relationship between games and AI and the numerous ways both games and AI have been challenged, but nevertheless, advanced through this relationship.






Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Printfriendly