2017年8月10日 JAY CHRISTOPHERSON
在我们确定如何设计计算基础结构以支持业务应用时, 我们面临两个主要挑战: 我们应该如何处理分布式运行时更改和服务发现。我们需要在部署世界各地的远程卫星通信地面站 (spokes) 时进行分布式更改, 但它们都是通过中心位置 (hub) 来管理的。从中心位置所做的更改需要以自动方式分发到一个或多个远程地面站。至于服务发现, 我们经常构建和部署, 我们需要确保在没有任何手动更新的情况下, 尽快检测和更新我们基础结构中的服务更改。这就是我们关注HashiCorp Consul的原因。
我们的所有应用在部署时都注册到Consul，所以我们大量使用基于Consul的健康检查, 标签, 外部服务注册和负载平衡 (对于某些工具, 可以注册一个 "活动" 组件)功能。此外, DNS 允许我们的各种应用和服务动态更新并及时发现其他所需的服务。对于运行时配置, 我们投资了诸如Consul模板这样的工具来构建可根据触发器更新的动态配置文件, 例如Consul中的value更改。最后, 对我们部署的服务的更改通常变得简单, 就像对源存储库文件中保存的参数进行更改一样。构建在更改时触发, 更新被推送到Consul。现在, 我们部署的服务的更改可以自然而然进行。
我们基于Consul的系统构建以及其他的 HashiCorp 产品
在Spaceflight Industries中的任何部署的核心包括四部分: HashiCorp Consul, HashiCorp Vault, HashiCorp Nomad, 和HashiCorp Terraform。专注于Consul, 它是我们所做一切的基本组成部分, 包括关键/价值管理、服务注册、动态 DNS 和外部服务。我们设计的任何新服务都是围绕如何与Consul进行交互和配置而设计的。它还充当我们的 Vault 部署的后备存储。对于任何生产级服务, 我们都在群集中部署Consul。
我们在一个hub-and-spoke架构中设计了我们的 setup。我们有一个中央 hub, 所有地面站的所有变化都同步到远程站点。我们只选择在中央hub群集中保留本地的一些关键内容。以前, 我们使每个远程地面站成为hub集群的只读副本。但是, 我们发现将某些项目标记为本地 (即不同步) 的功能在最新的Consul 版本中可用, 这使我们更容易获得某些内容, 我们更希望采用这种更混合的同步方法。
我们对 Consul 的使用在整个过程中为我们提供了重要的经验。我们过去使用脚本填充了 K/V 储存, 但发现这实际上不是长期可持续的。我们真正想要的, 是让开发人员在源代码控制中 check in 一个更改, 并能够 "自动" 生成/测试/部署, 以便跟踪所有更改, 而无需对 Consul 或我们的 DevOps 基础结构有深入了解。使用 Terraform 资源来用 K/V 或外部服务来填充 Consul 意味着我们可以很容易地实现这个目标。在源代码中进行更改、提交、触发构建/测试、触发Terraform 计划/部署, 而这一切都在极少的时间内在后台发生, 并且在初始提交之后无需进行交互。
我们在这里的最佳实践建议 (这可能是最明显的), 是构建您的系统以便可以轻松地从单个来源跟踪更改。另外, 设置您的系统, 以便只能从该单一来源进行更改。实施 break-glass procedure, 这将允许某人升级他们的特权, 以便他们可以直接在紧急情况下进行更改。
提高开发人员工作效率, 减少管理开销和瓶颈
我们的实施方案有几个主要好处:
- 我们需要很少的管理开销来管理全球基础设施。
- 没有一个人 (或组) 需要对系统进行更改-没有 "如果他被公共汽车撞到" 的情形。任何通过身份验证的具有源存储库访问权限的单个开发人员都可以进行更改, 并能通过系统实现从提交到部署的跟踪。
- 围绕 Hashicorp 服务 (特别是 Consul 构建我们的体系结构意味着我们的开发人员在基础结构的不同功能部分中有通用的配置语言, 有获得帮助的能力（如果他们需要的话，管是基于community的还是基于合同的), 并且能够轻松地将生产环境缩小到单个开发环境, 并具有一些参数变化。
所有这些都减少了我们构建系统和推出更改的时间。然后, 我们可以花更多的时间来构建我们的产品, 而不是交付基础设施。
【原文】Spaceflight uses HashiCorp Consul for Service Discovery and Runtime Configuration in their Hub-and-Spoke Network Architecture
AUG 10 2017 JAY CHRISTOPHERSON
We had two main challenges facing us as we determined how to design the computing infrastructure to support our business applications: how we should handle distributed runtime changes and service discovery. We need distributed changes as we deploy remote satellite communications ground stations (spokes) around the world, but which are all managed from a central location (hub). Changes made from a central location need to be distributed out to one or more remote ground stations in an automated fashion. As for service discovery, we build and deploy quite often and we needed to make sure that changes to services in our infrastructure are detected and updated as quickly as possible without any manual updates. These are the reasons we looked at HashiCorp Consul.
Our applications all register on deployment with Consul and from there, we make heavy use of Consul-based health checks, tags, external services registration, and load balancing (for certain tools that can register an “active” component). Also, DNS allows all of our various applications and services to dynamically update and discover other required services. For runtime configuration we have invested in tools like consul-template to build dynamic configuration files that can update based on triggers, such as a value change in Consul. In the end, a change to our deployed services often becomes as simple as making a change to a parameter held in a file in our source repository; builds are triggered on change, the updates are pushed to Consul. Now, changes to our deployed services “just happen.”
Our Architecture with Consul and other HashiCorp products
The core of any deployment at Spaceflight Industries is composed of four pieces: HashiCorp Consul, HashiCorp Vault, HashiCorp Nomad, and HashiCorp Terraform. Focusing on Consul, it’s the base component of everything we do which includes key/value management, service registration, dynamic DNS, and external services. Any new service we design is designed around how it is going to interact and be configured with Consul. It also serves as the backing store for our Vault deployment. For any production level services, we deploy Consul in a cluster.
We’ve designed our setup in a hub-and-spoke architecture. We have a central hub where any changes that are common across all ground stations are synchronized out to remote sites. There are a few key things that we choose to keep locally in the hub cluster only. Previously, we made every remote ground station a read-only copy of the hub cluster. However, we have found that the ability to mark some items as local-only (i.e., not sync’d out), available in the more recent versions of Consul, has made certain things easier for us and we prefer a more hybrid approach to synchronization now.
Our implementation of Consul provided us with key learnings throughout the process. We used to populate the K/V store using scripts but found that this really wasn’t sustainable long term. What we really wanted, was to have a developer check in a change to source control, and have that “automagically” build/test/deploy so that all changes are tracked, without requiring special knowledge of Consul or our DevOps infrastructure. Using a Terraform resource to populate Consul with K/V or external services means that we can easily achieve that goal. Make a change in source, commit, trigger a build/test, trigger a Terraform plan/deploy, and it all happens behind the scenes in very little time with zero interaction beyond the initial commit.
Our best practice recommendation here (which is probably obvious to most), is to build your system so that changes can be easily tracked from a single source. Also, set up your system so that changes can only be made from that single source. Implement a break-glass procedure which would allow someone to escalate their privileges such that they can make changes directly in an emergency.
Less management overhead and bottlenecks with increased developer productivity
There have been several key benefits from our implementation:
- We require very little administrative overhead to manage a global infrastructure.
- There’s not one single person (or group) that is required to make changes to the system - no “what if he got hit by a bus” scenario. Any single developer, with proper authenticated access to our source repo, can make a change that they can track through the system from commit to deploy.
- Building our architecture around Hashicorp services (Consul in particular) means that our developers have a common configuration language across different functional pieces of our infrastructure, the ability to receive help if they need it (both community- and contract-based), and the ability to easily scale down a production environment to a single development environment with a few parameter changes.
All of this reduces our time to build systems and roll out changes. We then get to spend more time on building our products rather than the process of delivering our infrastructure.