In order to contribute to Enough project, I need to execute OpenStack based tests. Enough project provides me OVH credentials allowing to execute these tests.
I wanted to be able to use another OpenStack provider because the OpenStack Orchestration API provided by OVH isn’t reliable. For example these issues were encountered several times:
INTERNALERROR> [packages-host]: UPDATE_IN_PROGRESS Stack UPDATE started INTERNALERROR> [packages-host]: UPDATE_FAILED Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. INTERNALERROR> <class 'oslo_db.exception.DBError'> (HTTP 500) (Request-ID: req-55b4b67a-10d9-4c26-8af9-0226bb78d8b3)
openstack.py 34 INFO ERROR: Property error: : resources.instance.properties.flavor: : Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. openstack.py 34 INFO <class 'sqlalchemy.exc.DBAPIError'> (HTTP 500) (Request-ID: req-5acadfdb-03b0-4c93-8e5f-85b039b2775c)
Besides the OpenStack Orchestration API provided by OVH is quite slow (see performance sections below).
The current code base requires:
- Orchestration API available (Heat, at least
- a public IPv4 per virtual machine (not a floating IP but a direct IP)
- In order to deploy every available services 15 virtual machines are required but only 5 virtual machines are required to execute the tests.
- available flavors:
- the flavors must be provided with an attached root disk by default (not an explicit block storage)
- most of the virtual machines use 2Go RAM, some hosts/tests require 4Go or 8Go RAM
Ideally, these characteristics are available:
- a private network with an IP per virtual machine
- on demand billing: per seconds/per minute
- Debian stable image
- location/region: on the same continent as the user
- the provider should be implicated in OpenStack development
OpenStack provider list (as of november 2020)
|Provider||Heat API||Public IPv4||default disk||billing||Europe region available||OpenStack commits|
|Elastx, details||floating IP||minute|
|Irideos I contacted them in order to get more information but they didn’t answered.||floating IP|
|Limestone networks||IPv6 only||minute||484|
|Open Telekom Cloud, details||floating IP||seconds||90|
|Orange, details||floating IP||minute||1242|
The blank fields means that i don’t know the values, if people have the information let me know !
My own manual tests
Besides OVH, I tested: Open Telekom Cloud and Fuga providers.
Open Telekom Cloud
This provider uses FusionSphere (Huawei’s commercial OpenStack release). The web interface is a bit disappointing compared to Horizon web interface, it doesn’t provide a link to
I didn’t encounter any technical issue with this provider: no stack stuck, no weird/unexpected API error.
By default, the quotas are: 10 Elastic cloud instances, 40 vCPU, 163Go RAM, 50 disks, 12To (disks), 10 virtual network, only 3 floating IPs.
The standard price model for the elastic instances is based on per-second consumption. I paid 3.19€ for 151 hours of s3.medium.2 (2 Go RAM, 1 vCPU, 20 Go SATA) instance (0.021€/hour), 1.66€ for 79 hours of s2.medium.2 (2 Go RAM, 1 vCPU, 20 Go SATA) instance (0.021€/hour) and 0.87€ for 19 hours of s2.medium.4 (4 Go RAM, 1 vCPU, 20 Go SATA) instance (0.046€/hour).
On average (12 runs)
tests/run-tests.sh tox -e icinga -- --enough-no-tests --enough-no-destroy playbooks/icinga/tests takes 9 minutes 30 seconds. Using the OVH provider, the same command takes 15 minutes 34 seconds (8 runs).
This provider doesn’t support
direct IP which is a requirement for Enough: that’s why I switched to Fuga.
--- clouds: production: auth: auth_url: "https://iam.eu-de.otc.t-systems.com/v3" project_name: "eu-de" # optional tenant_name: "eu-de" # optional user_domain_name: "OTC-EU-DE-000000000010000@@@@@" project_id: "8fd12e514a3a4ab5bb9565a67b9a6b03" username: "######## OTC-EU-DE-000000000010000@@@@@" password: "XXXXX" identity_api_version: 3 interface: public endpoint_type: public volume_api_version: 2 image_api_version: 2
By default, the quotas are: 20 instances, 50 vCPU, 20Go RAM, 10 volumes, 12To (disks).
I paid 0.6€ for 27 hours of
s3.small (2 Go RAM, 1 vCPU, 50 Go SSD) instance (0.02232€/hour).
On average (8 runs)
tests/run-tests.sh tox -e icinga -- --enough-no-tests --enough-no-destroy playbooks/icinga/tests takes 8 minutes 34 seconds. Using the OVH provider, the same command takes 15 minutes 34 seconds (8 runs).
I contacted the support three times:
- at the beginning I wasn’t able to create a stack using the provided credentials (the team credentials must be used in order to use Heat API). I contacted Fuga support using their web interface at 11pm (UTC), the workaround was provided by them the next day at 8:42 am.
- twice in order to delete stacks stuck in
DELETED_FAILEDstate. The first time I needed to reach them again: they didn’t acknowledge the issue at first and after pushing it was resolved. The 2nd time I contacted them a Saturday at 6:30 pm, they fixed this issue the following Monday around 1pm.
Heat API errors
The following errors have been encountered (at least once each, not many more):
INTERNALERROR> 2020-10-12 14:42:02Z [website-host]: UPDATE_IN_PROGRESS Stack UPDATE started INTERNALERROR> 2020-10-12 14:42:11Z [website-host]: UPDATE_FAILED (sqlalchemy.exc.ResourceClosedError) This Connection is closed (Background on this error at: http://sqlalche.me/e/dbapi)
INTERNALERROR> 2020-10-15 22:44:41Z [website-host]: UPDATE_IN_PROGRESS Stack UPDATE started INTERNALERROR> 2020-10-15 22:44:45Z [website-host]: UPDATE_FAILED (pymysql.err.OperationalError) (1213, u'Deadlock: wsrep aborted transaction') (Background on this error at: http://sqlalche.me/e/e3q8)
clouds: production: auth: auth_url: "https://identity.api.ams.fuga.cloud:443/v3" user_id: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" password: "YYYYYYYYYYYYYYYYYYYYYYYYY" user_domain_id: "ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ" project_domain_id: "ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ" project_id: "WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW" region_name: "ams" interface: "public" identity_api_version: 3
These tests don’t allow me to recommend Fuga for production services yet: more tests would be required but the provided service is good enough to execute tests requiring Heat Orchestration API.
I hope this work will ease the contributions of the community.
Don’t hesitate to share your feedback about OpenStack providers.
Since we will be able soon to use a more reliable provider, my next goal will be to enable OpenStack integration tests in the CI.
Thanks Misc for the proofreading !