OpenStack_Swift源代码分析——创建Ring及加入?设备源代码算法具体分析
内容导读
互联网集市收集整理的这篇技术教程文章主要介绍了OpenStack_Swift源代码分析——创建Ring及加入?设备源代码算法具体分析,小编现在分享给大家,供广大互联网技能从业者学习和参考。文章包含6390字,纯文字阅读大概需要10分钟。
内容图文
1 创建Ring 代码具体分析
在OpenStack_Swift——Ring组织架构中我们具体分析了Ring的具体工作过程,以下就Ring中添加?设备,删除设备,已经又一次平衡的实现过程作具体的介绍。
首先看RingBuilder类
def __init__(self, part_power, replicas, min_part_hours): #why 最大 2**32 if part_power > 32: raise ValueError("part_power must be at most 32 (was %d)" % (part_power,)) if replicas < 1: raise ValueError("replicas must be at least 1 (was %.6f)" % (replicas,)) if min_part_hours < 0: raise ValueError("min_part_hours must be non-negative (was %d)" % (min_part_hours,)) self.part_power = part_power self.replicas = replicas self.min_part_hours = min_part_hours self.parts = 2 ** self.part_power #分区数量即虚拟节点数量 self.devs = [] #用来存放设备的加入?的设备都会加入到devs里 self.devs_changed = False self.version = 0
self._last_part_moves_epoch = None self._last_part_moves = None self._last_part_gather_start = 0 self._remove_devs = [] self._ring = None当中devs里面没一个dev含有的属性为: dev = {‘weight‘: 100.0, ‘zone‘: 0, ‘ip‘: ‘127.0.0.1‘, ‘region‘: 0, ‘parts‘: 0, ‘id‘: 0, ‘meta‘: ‘some meta data‘, ‘device‘: ‘sda1‘, ‘parts_wanted‘: 96, ‘port‘: 6000}
当中part_wanted 是在加入?设备后设备须要的虚拟节点数,parts为设备已经被分配的虚拟节点。
在swift-ring-builder object.builder create 18 3 1
会调用Command类的create方法,详细看此方法:
def create(self): """ swift-ring-builder object.builder create 18 3 1 swift-ring-builder <builder_file> create <part_power> <replicas> <min_part_hours> Creates <builder_file> with 2^<part_power> partitions and <replicas>. <min_part_hours> is number of hours to restrict(限制) moving a partition more than once(不止一次). """ if len(argv) < 6: print Commands.create.__doc__.strip() exit(EXIT_ERROR) builder = RingBuilder(int(argv[3]), float(argv[4]), int(argv[5])) backup_dir = pathjoin(dirname(argv[1]), 'backups') try: mkdir(backup_dir) except OSError as err: if err.errno != EEXIST: raise builder.save(pathjoin(backup_dir, '%d.' % time() + basename(argv[1]))) builder.save(argv[1]) #序列化 exit(EXIT_SUCCESS)在第一次创建时,首先会推断/etc/swift下是否有object.builder文件,没有须要创建此文件,并依据命令中个的 18 3 1 来实例化RingBuilder,最后将builder序列化。
2 为Ring加入?设备
创建了.builder文件后,就须要加入?设备,以下为加入?一个设备的命令:swift-ring-builder object.builder add z2-127.0.0.1:6020/sdb2 100
当中 add 指明为加入?设备 z2 为zone2,127.0.0.1:6020/sdb2为设备ip地址,绑定的port以及设备中存储文件的磁盘,100为设备的权重
以下看加入?设备的详细实现:
这种方法在main方法中药推断/etc/swift下是否有object.builder文件,在第一步create中我们已经创建了此文件,如今仅仅须要将他反序列化回来并用存的数据来实例化RingBuilder类,然后调用Commands类的add方法:
def add(self): """ swift-ring-builder <builder_file> add [r<region>]z<zone>-<ip>:<port>[R<r_ip>:<r_port>]/<device_name>_<meta> <weight> [[r<region>]z<zone>-<ip>:<port>[R<r_ip>:<r_port>]/<device_name>_<meta> <weight>] ... Where <r_ip> and <r_port> are replication ip and port. or swift-ring-builder <builder_file> add [--region <region>] --zone <zone> --ip <ip> --port <port> --replication-ip <r_ip> --replication-port <r_port> --device <device_name> --meta <meta> --weight <weight> Adds devices to the ring with the given information. No partitions will be assigned to the new device until after running 'rebalance'. This is so you can make multiple device changes and rebalance them all just once. """ if len(argv) < 5 or len(argv) % 2 != 1: print Commands.add.__doc__.strip() exit(EXIT_ERROR) for new_dev in _parse_add_values(argv[3:]): for dev in builder.devs: if dev is None: continue if dev['ip'] == new_dev['ip'] and dev['port'] == new_dev['port'] and dev['device'] == new_dev['device']: print 'Device %d already uses %s:%d/%s.' % (dev['id'], dev['ip'], dev['port'], dev['device']) print "The on-disk ring builder is unchanged.\n" exit(EXIT_ERROR) dev_id = builder.add_dev(new_dev) #调用RingBuilder类的add_dev方法 print('Device %s with %s weight got id %s' % (format_device(new_dev), new_dev['weight'], dev_id)) builder.save(argv[1]) exit(EXIT_SUCCESS)将新加入?的设备会加入到devs[]中,当中会调用add_dev,以下看此方法的详细实现
def add_dev(self, dev): """ Add a device to the ring. This device dict should have a minimum of the following keys: ====== =============================================================== id unique integer identifier amongst devices. Defaults to the next id if the 'id' key is not provided in the dict weight a float of the relative weight of this device as compared to others; this indicates how many partitions the builder will try to assign to this device region integer indicating which region the device is in zone integer indicating which zone the device is in; a given partition will not be assigned to multiple devices within the same (region, zone) pair if there is any alternative ip the ip address of the device port the tcp port of the device device the device's name on disk (sdb1, for example) meta general use 'extra' field; for example: the online date, the hardware description ====== =============================================================== .. note:: This will not rebalance the ring immediately as you may want to make multiple changes for a single rebalance. :param dev: device dict :returns: id of device dev = {'weight': 100.0, 'zone': 0, 'ip': '127.0.0.1', 'region': 0, 'parts': 0, 'id': 0, 'meta': 'some meta data', 'device': 'sda1', 'parts_wanted': 96, 'port': 6000} """ if 'id' not in dev: dev['id'] = 0 if self.devs: dev['id'] = max(d['id'] for d in self.devs if d) + 1 if dev['id'] < len(self.devs) and self.devs[dev['id']] is not None: raise exceptions.DuplicateDeviceError( 'Duplicate device id: %d' % dev['id']) # Add holes to self.devs to ensure self.devs[dev['id']] will be the dev while dev['id'] >= len(self.devs): self.devs.append(None) dev['weight'] = float(dev['weight']) dev['parts'] = 0 self.devs[dev['id']] = dev self._set_parts_wanted() #设置part_wanted self.devs_changed = True self.version += 1 return dev['id']此方法主要是获得weight,并为设备dev中的id赋值,以及part_wanted赋值。当中part_wanted值的计算为:
((self.parts * self.replicas / sum(d[‘weight‘] for d in self._iter_devs())) * dev[‘weight‘]) - dev[‘parts‘]
parts=2**power(此处在create方法中的18) replicas为备份数,此处为3
即part_wanted=((虚拟节点数*备份数/全部已加入?的节点的weight和)*当前设备的weight)-已被分配给此设备的虚拟节点数
至此 创建Ring和为Ring加入?设备解说完毕,再完毕这两个步骤后,就须要平衡环,并为replica2part2dev(备份到分区到设备的映射)赋值,平衡算法是Ring 的核心算法,在下一篇文章中详细解说。
因为本人水平有限,文中难免出现理解错误,敬请指正、交流,谢谢!
原文:http://www.cnblogs.com/hrhguanli/p/3819970.html
内容总结
以上是互联网集市为您收集整理的OpenStack_Swift源代码分析——创建Ring及加入?设备源代码算法具体分析全部内容,希望文章能够帮你解决OpenStack_Swift源代码分析——创建Ring及加入?设备源代码算法具体分析所遇到的程序开发问题。 如果觉得互联网集市技术教程内容还不错,欢迎将互联网集市网站推荐给程序员好友。
内容备注
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 gblab@vip.qq.com 举报,一经查实,本站将立刻删除。
内容手机端
扫描二维码推送至手机访问。