Upgrade Orchestrations are essential! We're both delighted and frustrated by OpenStack's pace of innovation because by the time we get the current release working then new hotness arrives. Last year, it was enough to just install OpenStack, but now we think it's required to have an upgrade plan. As the founders of Crowbar, we are leaders in the cookbook design for OpenStack and have a lot of experience with orchestration for OpenStack deployments. This community discussion about our proposed upgrade pattern reviews our devops recommendations (do NOT mix cookbooks for multiple releases) and orchestration design (dedicated cookbooks for orchestration). If you're interested in cookbooks that are testable and minimize complexity then this session is for you! We want orchestrations between versions that can focus on the specific use-cases around the migration scenarios like incremental, fastest-possible, change of operating system, or VM migration. If you agree that migrations between versions are also very important then look no farther!
Typical cloud deployments - be it Openstack, Eucalyptus etc - have a separate control layer installed and upgraded using separate tools (which might be hand-configured PXE + preseeding, Crowbar, Cobbler, Orchestra/MAAS, FAI etc). As a result you have two distinct provisioning systems in play, which allows for more operator error and requires increased special cases in automation.
Using Openstack’s bare-metal hypervisor, we are building a fully self contained cloud, where the control layer for the cloud is itself deployed and upgraded via the same cloud API.
Come hear me talk about the challenges involved in bootstrapping and operating such an environment, the benefits it can bring and what you can do with it!
A predecessor of this talk was given at Linux.conf.au in January of this year where it was well received.
OpenStack and KVM/QEMU support a cornucopia of image and disk formats. With so many options it can be difficult to understand all the various trade-offs of these formats.
We will explore some of the more common image formats and disk formats and their trade-offs. We will cover tips and tricks for converting image formats, and working with these images directly. Additionally we will dive into how nova, libvirt, and KVM interact with these formats when doing various operations such as snapshoting and image resizing. We will look at best practices for configuring the guest operating systems such as default login credentials, network configuration, and device/performance optimization. Finally we will look at how to pre-install tools to assist in configuring the OS at VM creation as well as how to interact with the metadata API.
Synopsis: There is no easy answer or magic solution when architecting your private cloud. OpenStack is flexible and can be designed in many ways which can be a blessing or a curse. The goal of this talk is to provide guidance on how to start thinking about your private cloud architecture.
I am continuing this series from my operational talks at the Grizzly Summit and would like to make this a standard talk at every summit. We've been working with Folsom for 6 months and will be updated as such.
Overview:
1. Build with the end in mind (don't paint yourself into an architectural corner)
2. Images and Storage
3. Architecture examples and thoughts for the following environment sizes: a. 1-20 physical nodes
b. 20-100 physical nodes
4. Performance Considerations and Bottlenecks
5. Lessons Learned
6. Operational updates
7. Q/A and Community Input
OpenStack is now installed. Now how do you operate it? How do you perform upgrades? Join Director of Software Development Jason Cannavale, as he demonstrates and discusses the Rackspace Private Cloud approach to operating an OpenStack powered cloud and simplifying the operator experience.
Ceilometer is now one year old, and we just delivered our first synchronized release with OpenStack, our second official release.
During the past 6 months, and as a follow up to the intense discussions we had at the last summit, we delivered a much more robust
solution which perimeter and architecture has been extended from just metering to metric gathering at large accross all OpenStack projects.
This talk will first shortly go back on the project history, then explain the architecture evolution and uses cases it now permit and will close by explaining how you can put Ceilometer into action on your own projects.
Boris from Russia here... We do much OpenStack at Mirantis. Much customer ask us to make cloud controller is highly available. Also much customer is cheap and ask only free, open source stuff in their cloud. At Mirantis we like make customer happy, so we make puppet recipe to make very highly available OpenStack for free. In this talk I make simple demonstration that even a goat that had a lot of vodka can understand how is use open puppet recipes to make highly available OpenStack and pay zero rubles to anyone. Also, a goat.
The open source configuration management and automation framework Chef is used to deploy and manage many large public and private installations of OpenStack and supports a wide variety of deployment scenarios. Chef for OpenStack is a project based on the healthy exchange of code, ideas and documentation for deploying and operating OpenStack with Chef. With involvement from Intel, AT&T, Dell, HP,
Rackspace and many others there is a community of collaboration between users, developers and operators. This session will discuss the currently available resources and documentation, the evolution and layout of the project and the roadmap going forward.
A lot of effort has gone into cloud storage peformance benchmarking, both of swift and other cloud stacks and part of the result is a lot of confusion in the numbers, in large part because there is no standard. This is further complicated because some implementations are written in java, some in python and some in raw curl. Furthermore, the underlying libraries themselves can cause variances as they do not all use the same buffer sizes, enable/disable ssl-compression and probably other parameters as well.
I would like to talk about our benchmarking methodologies at HP as well as describe a tool suite I've developed that implements them and share some results of benchmarking our own OpenStack implementation. One thing I've discovered over previous months of testing is that both latency and cpu overhead can have a major impact on performance and those are captured as well, something most tools typically don't report.
The tools are written in python and use the OpenStack python-swiftclient library.
The OpenStack project does an insane amount of automated testing as part of the development cycle, but up until now there has been no corresponding testing that can be performed against running public clouds. While we want to do that, before we can test other people's clouds for compatibility, we need to be able to express what it is they need to be compatible with.
Enter RefStack
It turns out that OpenStack is rich enough now to express a reference implementation in terms of itself, using heat templates. Some people think that's a great end to itself - deploy your OpenStack using OpenStack - but others are not quite as sure about that yet, and have significant investment in things like chef, puppet, crowbar or cobbler. To meet the needs of expressing a useful set of testable information and not leave that specification as an academic exercise, or as the recipient of more tool wars - we've come up with a plan to have the heat templates describe the state, the "what" if you will, and to describe a clear boundary line across which metadata is passed to the tools on the individual nodes that will turn that metadata into configuration.
Over the course of the talk, we'll discuss:
Let’s discuss Hadoop for OpenStack log analysis! Hadoop can support operational monitoring, troubleshooting, and capacity planning in a consistent and open way. We’ll share the work we’ve started, and lead an interactive discussion of different approaches already in play. Our goal is to collaborate on the best patterns for different deployment environments.
Even with Quantum lingering right around the corner, nova-network still has its place in existing OpenStack clouds and will be used in the immediate future for many deployments. The goal of this talk is to provide in depth information about nova-network and items to consider when architecting your cloud.
OpenStack Expertise Level: Beginner - Intermediate with good working knowledge of networking, linux networking and iptables.
Overview:
1) nova-network overview
2) nova-network options
3) iptables and ebtables
4) Floating IPs
5) Considerations for integrating into your existing network
6) Example architectures
7) Q/A
This discussion will cover how to use existing tool chains (oz, boxgrinder, veewee) to automate building homogenous, patched OS images for cloud consumers, as well as functionally testing and deploying your images using continuous integration.
One of the great challenges of of monitoring any large cluster is how much data to collect and how often to collect it. Those responsible for managing the cloud infrastructure want to see everything collected centrally which places limits on how much and how often. Developers on the other hand want to see as much detail as they can at as high a frequency as reasonable without impacting the overall cloud performance.
To address what seems to be conflicting requirements, we've chosen a hybrid model at HP. Like many others, we have a centralized monitoring system that records a set of key system metrics for all servers at the granularity of 1 minute, but at the same time we do fine-grained local monitoring on each server of hundreds of metrics every second so when there are problems that need more details than are available centrally, one can go to the servers in question to see exactly what was going on at any specific time.
The tool of choice for this fine-grained monitoring is the open source tool collectl, which additionally has an extensible api. It is through this api that we've developed a swift monitoring capability to not only capture the number of gets, put, etc every second, but using collectl's colmux utility, we can also display these in a top-like formact to see exactly what all the object and/or proxy servers are doing in real-time.
We've also developer a second cability that allows one to see what the Virtual Machines are doing on each compute node in terms of CPU, disk and network traffic. This data can also be displayed in real-time with colmux.
This talk will briefly introduce the audience to collectl's capabilities but more importantly show how it's used to augment any existing centralized monitoring infrastructure.
Join this panel of infrastructure and cloud hardware experts in a spirited discussion about what works (and what does not) for OpenStack deployments. We’ve assembled hardware and solution vendors together in a panel so that operators can learn from their field experience. We’ll also be hearing about what makes individual offerings advantaged for OpenStack and how to build a cloud that can scale.
How different are servers from VMs? Do we need special tools to manage servers, or can we adapt a more cloud-like pattern in managing them at scale? Heat has been designed to deploy cloud applications on top of OpenStack. But with Nova Baremetal, the line blurs between cloud and real server. As part of the OpenStack - on OpenStack, or "TripleO" project, we're excited to use Heat to manage a complete deployment of OpenStack. We'll be sharing the various techniques we make use of in Heat to leverage its orchestration capabilities in fully automating the deployment and management of OpenStack.
From a public cloud big enough to make Jeff Bezos crap his pants, down to a single-node DevStack environment under VirtualBox, this talk will cover why scale matters and what you must take into account when planning an OpenStack deployment. Scalability details of specific OpenStack components (compute, block and object storage, and networking), their inherent limits, and effective workarounds will be discussed along with a review of a few deployments that worked and others that didn’t.
As the main sponsors of Ubuntu, Canonical is deeply experienced with running instances in all the major public clouds, and as one of the first members of the OpenStack project, also has organizational expertise with the private cloud.
With this all in mind, we asked our IS team a question:
Would it be possible for us to move to a cloud-centric workflow across the entire company? Supporting not only the internal systems that keep Canonical running, but also parts of the widely popular and globally used Ubuntu project?
The answer was "Yes"....but we learned a ton and would like to share some of the things we learned around the following topics:
Organizational needs of moving from a traditional "IS over here and developers over there” to "DevOps".
Software
Hardware
Workflows - How code gets from a laptop to production quickly and tested?
What happens when things fail? How do you roll back?
User/Developer education
Inktank Ceph is a transformational open source storage solution fully integrated into OpenStack providing scalable object and block storage (via Cinder) using commodity servers. The Ceph solution is resilient to failures, uses storage efficiently, and performs well under a variety of VM Workloads.
Dell Crowbar is an open source software framework that can automatically deploy Ceph and OpenStack on bare metal servers in a matter of hours. The Ceph team worked with Dell to create a Ceph barclamp (a crowbar extention) that integrates Glance, Cinder, and Nova-Volume. As a result, it is lot faster and easier to install, configure, and manage a sizable OpenStack and Ceph cluster that is tightly integrated and cost- optimized.
Hear how OpenStack users can address their storage deployment challenges:
Considerations when selecting a cloud storage system
Overview of the Ceph architecture with unique features and benefits
Overview of Dell Crowbar and how it can automate and simplify Ceph/OpenStack deployments Best practices in deploying cloud storage with Ceph and OpenStack
Co-presented by Kamesh Pemmaraju, Product Manager from Dell and Miroslav Klivansky, Technical Marketing Engineer from Inktank.
According to Wikipedia, Disaster Recovery (DR) is "the process, policies and procedures . . for recovery . . . of technology infrastructure . . . after a natural or human-induced disaster." The ability to recover quickly with minimal data loss after a disaster such as a fire, hurricane, etc., can make the difference between an organization staying in business or vanishing. In an OpenStack environment there are multiple approaches of realizing this recovery which differ in how much work is lost (the recovery point objective - RPO) and how long it takes to recover (the recovery time objective - RTO). These approaches trade-off up-front effort and cost (when there is no disaster) against greater data loss (RPO) and much longer recovery times (RTO) after a disaster. The appropriate approach depends upon the organization's objectives.
In this presentation, after a brief background on DR concepts, we will survey the various approaches that can be used to provide DR for an OpenStack cloud, showing how the up-front investment impacts RPO and RTO. We will start by considering solutions that work in any OpenStack environment, independent of the underlying physical infrastructure; while these solutions are relatively simple, they lead to long recovery times and significant data loss. We will also consider solutions integrated with the application, i.e., provided from within the guest; these solutions typically provide higher quality of service but at the drawback of being application specific. Finally, we will consider approaches which take advantage of advanced functions seen in storage controllers; these approaches can avoid all (or most) data loss and often can recover quickly, but require up front investment.
This panel will bring together the co-authors of the missing OpenStack Operations Guide, a book written in 5 days for the benefit of the entire OpenStack community. They will discuss the trials and tribulations of writing a book in such a short period of time, from the conception and proposal to the process and publication. The importance of documentation cannot be understated and this panel will highlight the efforts that the doc team are going to in order to produce the documentation necessary for a successful OpenStack ecosystem.
And bring your questions. No subjects are taboo. Ask about the stress, logisitics, collaboration, and whatever else you can think of. Tell us what you thought was missing from the book or what could be improved. Your willing panelists will answer anything!
The first 60 attendees to the panel will get a printed copy of the OpenStack Operations Guide courtesy of Rackspace.