The central question this paper asks is “how should we distribute our ML programs over a cluster?“. For some of us, the answer is “Spark”, and we call it a day. But if we aren’t satisified with the current state of performance, and would like to, instead, lean into the performance edge between ML models and systems, and explore the possibilities of improved performance, then we need to dig a little further.

DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning by Min Du, Feifei Li, Guineng Zheng, & Vivek Srikumar (2017). Overview Inspired by natural language modeling, the authors present a deep learning approach to anomaly detection in system logs. Key concepts system logs: A file that records the system states and significant events for debugging software. syslog: The system log standard maintained by Internet Engineering Task Force (IETF).

Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch. Machine Intelligence Research Institute. Alignment in this context means making sure agents arrive at and optimize objective functions that are in the spirit of what was intended; that is that goals are reached while making sure no one gets hurt. One of the key takeaways from this overview is that our solutions must scale with intelligence, so for any new discovery, how long will it “hold” in lock step with advances in intelligence?

Adam: A Method for Stochastic Optimization (2015). Diederik P. Kingma and Jimmy Lei Ba. Conference paper at ICLR 2015. Overview This method computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients. Combines AdaGrad (which works well with sparse gradients) and RMSProp (works well in on-line and non-stationary settings.

Introduction to Gaussian Processes (1998) - David Mackay Overview “From a Bayesian perspective, a choice of a neural network model can be viewed as defining a prior probability distribution over non-linear functions, and the neural network’s learning process can be interpreted in terms of the posterior probability distribution over the unknown function. (Some learning algorithms search for the function with maximum posterior probability, and other Monte Carlo methods draw samples from this posterior probability).

Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016). Shahriari et al. Proceedings of the IEEE Introduction “Mathematically we are considering the problem of finding a global maximizer (or minimizer) of an unknown objective function f, where X is some design space of interest; …” ”…in global optimization, X is often a compact subset of R^d but the Bayesian optimization framework can be applied to more unusual search spaces that involve categorical or conditional inputs.

Dušan Teodorovic. Swarm Intelligence systems for transportation engineering: principles and applications. Transportation Research, 2008. This is one of the most comprehensive articles I have found that attempts to connect swarm intelligence heuristics to transportation systems. Most of the article is dedicated to introducing four multi-agent systems that leverage information sharing models to optimize search techniques, and I will recap those here. Ant Colony Optimization (ACO) Ants leave pheromone trails, and an ant will use the strength of the signal to weight their choice of path, as well-trod paths traveled by ants heading to a food source have a stronger pheromone signal.

This paper applies models of bounded and biased cognition to the creation of a generative model for human choices in decision problems. Perhaps more importantly, the authors attempt to infer preferences (not beliefs) from this model. They focus their attention on four types of agents: Hyperbolic-discounting (i.e. my favorite new way of saying procrastinating) Agents using Monte Carlo approximations of expected utility “Myopic” agents Bounded value-of-information agents Choice is made in proportion to a softmax function of expected utility:

“Mathematics - Application and Applicability”, by Mark Steiner. Included in The Nature of Nature: Examining the Role of Naturalism in Science (2011). Edited by Bruce L. Gordon and William A. Dembski. ISI Books: Wilmington, Deleware. Key ideas Canonical applications (theories developed to describe an application) vs. non-canonical (applying mathematics in situations other than those that created them) Distinguishing applications of mathematics from mathematics itself Exploration of individual thinkers and their attempts to reconcile mathematics and the empirical world (including Gottlob Frege, Hartry Field, Eugene Wigner), with the group-theoretic leading the charge

What comes after minds? by Marvin Minsky. From The New Humanists: Science at the Edge (2003). Edited by John Brockman. Barnes & Noble Books: New York. Key quotes “No uniform scheme will lead to machines as resourceful as the human brain. Instead, I’m convinced that this will require many different ‘ways to think’ - along with bodies of knowledge about how and when to use them”. “Computer science has helped us envision a far wider range of ways to represent different types and forms of knowledge,…” “I see each emotional state as a distinctly different way to think”.