I’ve recently become involved with the new Linux Foundation Open Source Climate Finance organization (OS-Climate). I’m helping manage AWS contributions to the project, as we build an open source data lake and analysis service that can be used to model climate related asset risks for investors. This includes physical asset risk from things like sea level rise and extreme weather events, as well as market changes from customers deciding to buy more sustainable products, and economic risk as the world economy is impacted by climate change. …

One of the impacts of COVID-19 has been a large rapid unplanned change in business activity for most of the global economy. There are several outcomes, some markets like work from home tools and services, healthcare and grocery retail are extra busy, social media applications are extra busy but their revenue from advertising may be impacted, and many businesses in manufacturing, travel and entertainment are have largely been shut down because their workforce and customers are in lockdown. While datacenter based IT costs are largely fixed, in a three year depreciation schedule, cloud costs are inherently variable, and can be sized to fit the need. …

When I joined Netflix in 2007 I was managing a team that built the personalized home page for the DVD shipping web site. The first thing I found was that every user visible change we made went through an A/B test. This was institutionalized across Netflix from the beginning, and it’s part of the formation story told in Marc Randolph’s book That Will Never Work. As an experienced product manager and the founding CEO, he wanted to be able to measure what was really working. An oft-quoted study by Microsoft Bing found that of the changes they test, one-third prove effective, one-third have neutral results, and one-third have negative results. …

March 17, 2020

A resilient system continues to operate successfully in the presence of failures. There are many possible failure modes, and each exercises a different aspect of resilience. The system needs to maintain a safety margin that is capable of absorbing failure via defense in depth, and failure modes need to be prioritized to take care of the most likely and highest impact risks. In addition to the common financial calculation of risk as the product of probability and severity, engineering risk includes detectability. Failing silently represents a much bigger risk than when the same failure is clearly and promptly reported as an incident. Hence, one way to reduce risk is to make systems more observable. Another problem is that a design control, intended to mitigate a failure mode, may not work as intended. Infrequent failures exercise poorly tested capabilities that tend to amplify problems in unexpected ways rather than mitigate them, so it’s important to carefully exercise the system to ensure that design controls are well tested and operating correctly. Staff should be familiar with recovery processes and the behavior of the system when it’s working hard to mitigate failures. …

We’re excited to let you know that we have an Open Source track at re:Invent this year! Learn about what’s in it:
https://aws.amazon.com/blogs/opensource/open-source-at-aws-reinvent-2019/

Some selected sessions:

OPN201 Change is coming to robotics development: The shift to ROS 2
🤖 Alejandra Olvera-Novack — AWS Developer Relations on how the shift from Robot Operating System (ROS) 1 to ROS 2 will change the landscape for all robot lovers.

OPN220 Build robotic cloud simulations with ROS and AWS RoboMaker
Join Camilo Buscaron, AWS Principal Open Source Technologist, and Katherine Scott, Developer Advocate, Open Robotics in this workshop to use Gazebo, a 3D simulator, and Robot Operating System (ROS) on AWS RoboMaker and learn how to spin up robotic simulations.

OPN204 Secure your Open Distro for Elasticsearch cluster
Dive deep on Open Distro Security: secure your cluster with your own SSL certs, set up fine-grained access control for your documents and fields, assign users to roles, create multi-tenant Kibana visualizations and dashboards, and integrate with federated identity providers like LDAP, Active Directory, and Okta.

OPN303 BPF performance analysis (repeats)
Learn how Netflix uses eBPF to create powerful performance analysis tools. Brendan Gregg tours BPF tracing, with open source tools & examples for EC2 instance analysis.

OPN205-R Contributing to the AWS Construct Library (repeats)
Using and loving the AWS Cloud Development Kit and want to help make it better? Join Lee Packham, AWS Solutions Architect and Enrico Huijbers, AWS Software Development Engineer to find out how easy it is.

OPN304 Learnings from migrating a service from JDK 8 to JDK 11
AWS Lambda improved latency by migrating to JDK 11 with Amazon Corretto. Learn about how Lambda works behind the scenes, and how you can follow these steps to migrate your application to Corretto with Niall Connaughton — Software Engineer, AWS Serverless Applications, and Yishai Galatzer, Senior Manager Software Development.

OPN207 PartiQL: One query language for all of your data
PartiQL is a SQL-compatible query language that makes it easy and efficient to process both structured (relational) data, as well as semi-structured, schemaless, and nested data, as typically found in open data formats in the Amazon S3 data lake and document-oriented databases. PartiQL is a unifying query language, enabling single-query-language access to multiple AWS services. Almann Goo, Principal Engineer, and Yannis Papakonstantinou, Senior Principal Scientist discuss PartiQL’s core tenets and how they served AWS services and Amazon projects, and how PartiQL can benefit your applications and tools in need of unifying query access to the data lake.

OPN308-R1 PartiQL: Solution integration and joining the community (repeats)
In this chalk talk, Almann Goo, Principal Engineer, and Yannis Papakonstantinou, Senior Principal Scientist, address how you can use the PartiQL open source in the development of your application, BI tool, and/or database service, and how to join the PartiQL community and contribute to the next steps in its evolution.

OPN402 Firecracker open-source innovation
Since Firecracker’s release at re:Invent 2018, several open-source teams have built on it, while AWS has continued investing in Firecracker’s speed. Radu Weiss, Software Development Manager AWS, and Sebastian Bernheim,Customer Success Engineer Weaveworks, discuss Weave Ignite from Weaveworks, which unifies containers and VMs using Firecracker microVMs with OCI images, containerd, and CNI. Leigh Capili (DevAdv) demos how to launch apps from Git using Kubernetes orchestration, Ignite virtualization, and GitOps management. Alexandra Iordache (Firecracker maintainer) showcases Firecracker’s new snapshotting capability.

OPN211 How Zalando runs Kubernetes clusters at scale on AWS
Henning Jacobs, Senior Principal Zalando SE, shows Zalando’s approach for running 140+ clusters on AWS, how it does continuous delivery for its cluster infrastructure, and how it created open-source tooling to manage cost efficiency and improve developer experience, openly sharing failures and learnings from three years of Kubernetes in production.

OPN305 How Optum manages transient developer accounts at scale
Optum uses Disposable Cloud Environment (an open-source account management product to facilitate public cloud experimentation in isolated sandbox environments) to empower its tens of thousands of engineers to explore public cloud services and resources while mitigating business risk. …

A resilient system continues to operate successfully in the presence of failures. There are many possible failure modes, and each exercises a different aspect of resilience. The system needs to maintain a safety margin that is capable of absorbing failure via defense in depth, and failure modes need to be prioritized to take care of the most likely and highest impact risks.

In addition to the common financial calculation of risk as the product of probability and severity, engineering risk includes detectability. Failing silently represents a much bigger risk than the same failure that is clearly and promptly reported as an incident. Hence, one way to reduce risk is to make systems more observable. Another problem is that a design control, intended to mitigate a failure mode, may not work as intended. Infrequent failures exercise poorly tested capabilities that tend to amplify problems in unexpected ways rather than mitigate them, so it’s important to carefully exercise the system to ensure that design controls are well tested and operating correctly. In the same way that we have moved from a few big software releases a year to continuous delivery of many small changes, we need to move from annual disaster recover tests or suffering when things actually break, to continuously tested resilience. Staff should be familiar with recovery processes and the behavior of the system when it’s working hard to mitigate failures. …

I recently read Annaka Harris’ new book Conscious about the mysteries and nature of consciousness and had an epiphany. What we do to make our computer systems observable is actually closely related to consciousness. The essential quality of conciseness is a model of the system. That model is fed with information and interrogated about the overall health and behavior of the system. If the system is ourselves then we can ask “how do you feel?”, “do you need anything?”, “what did you learn today?”, “what do you think is going to happen next?”, “What is the solution to this problem?”, “what do you think about this new information?” etc. We have a verbal query engine for our model. The model isn’t really in charge of us in the sense that we feel it is, it’s monitoring and making sense of what is happening around it, after the event. The illusion that we consciously decide in advance is discussed and deconstructed in Sam Harris’ book Free Will. (Yes, Sam and Annaka are a couple). …

[I found this in an archive and thought it was worth sharing 10 years on. This is one of the first events I attended as we were starting to figure out our cloud transition at Netflix, and was where I first met Dave Nielsen, Reuven Cohen, Sam Charrington, and Sara Dornsife]

From cloudcamp 9/30/08

Hello CloudCampers,

As you probably know, CloudCampSV is taking place tonite at Sun’s Executive Briefing Center in Menlo Park. And it’s sold out. We have an excellent lineup of attendees and we expect the unconference to be quite informative. See below for the schedule.

We are also happy to partner with SDForum’s Cloud Computing and Beyond conference which takes place tomorrow in Santa Clara. …