pool/python-osprofiler
SHA256
2
0
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

- update to version 2.6.0

Browse Source 
- Switch to stestr
  - When shortening span-ids, check if they're already short
  - Do not insert osprofiler filter into Neutron api-paste pipeline
  - Change python3.5 job to python3.7 job on Stein+
  - import zuul job settings from project-config
  - [devstack] Add support for elasticsearch backend
  - Change openstack-dev to openstack-discuss
  - Update reno for stable/rocky
  - Update min tox version to 2.0
  - Don't quote {posargs} in tox.ini
  - add python 3.6 unit test job
  - Make tracing of SQL statements configurable in DevStack plugin
  - add lib-forward-testing-python3 test job
  - build universal wheels
  - Reload keystone to apply osprofiler config
  - Allow test path to be overridden
  - Add sqlalchemy collector
  - Configure Jaeger collector in DevStack
  - add password for connecting redis-sentinel
  - Add support for mongodb backend in devstack plugin
  - In DevStack install Redis client library via pip, not as system package
  - Use $STACK_USER variable in install_jaeger function
  - Change http to https in reference link
  - In case of an error, always add message
  - Use templates for cover and lower-constraints
- remove 0001-Add-sqlalchemy-collector.patch and 0001-Don-t-fail-if-sqlalchemy-driver-fails-to-initialize.patch

OBS-URL: https://build.opensuse.org/package/show/Cloud:OpenStack:Factory/python-osprofiler?expand=0&rev=23
This commit is contained in:
Dirk Mueller 2019-04-08 14:05:31 +00:00 committed by Git OBS Bridge
parent 7bc255e325
commit 43c46bc8b3
7 changed files with 44 additions and 317 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

207
0001-Add-sqlalchemy-collector.patch
View File

@ -1,207 +0,0 @@
From 032a21861854c5f63a039c997a58b4a979e62750 Mon Sep 17 00:00:00 2001
From: Thomas Bechtold <tbechtold@suse.com>
Date: Fri, 8 Feb 2019 12:37:38 +0100
Subject: [PATCH] Add sqlalchemy collector
Beside the already available collectors, add a sqlalchemy based
collector. This is useful if you don't want to maintain another DB
solution and just use the (usually) already available database.
The driver currently implements the notify() and get_report() methods
so it is possible to store trace points and to get a single trace.
Change-Id: If91b35d4b97862c0ecf6677f4c6b95a09d411195
---
doc/source/user/collectors.rst | 25 +++++
osprofiler/drivers/__init__.py | 1 +
osprofiler/drivers/base.py | 6 ++
osprofiler/drivers/sqlalchemy_driver.py | 119 ++++++++++++++++++++++++
4 files changed, 151 insertions(+)
create mode 100644 osprofiler/drivers/sqlalchemy_driver.py
diff --git a/doc/source/user/collectors.rst b/doc/source/user/collectors.rst
index e163d57..5d48caa 100644
--- a/doc/source/user/collectors.rst
+++ b/doc/source/user/collectors.rst
@@ -39,3 +39,28 @@ Redis
value. Defaults to: 0.1 seconds
* sentinel_service_name: The name of the Sentinel service to use.
Defaults to: "mymaster"
+
+SQLAlchemy
+----------
+
+The SQLAlchemy collector allows you to store profiling data into a database
+supported by SQLAlchemy.
+
+Usage
+=====
+To use the driver, the `connection_string` in the `[osprofiler]` config section
+needs to be set to a connection string that `SQLAlchemy understands`_
+For example::
+
+ [osprofiler]
+ connection_string = mysql+pymysql://username:password@192.168.192.81/profiler?charset=utf8
+
+where `username` is the database username, `password` is the database password,
+`192.168.192.81` is the database IP address and `profiler` is the database name.
+
+The database (in this example called `profiler`) needs to be created manually and
+the database user (in this example called `username`) needs to have priviliges
+to create tables and select and insert rows.
+
+
+.. _SQLAlchemy understands: https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls
diff --git a/osprofiler/drivers/__init__.py b/osprofiler/drivers/__init__.py
index 37fdb69..022b094 100644
--- a/osprofiler/drivers/__init__.py
+++ b/osprofiler/drivers/__init__.py
@@ -5,3 +5,4 @@ from osprofiler.drivers import loginsight # noqa
from osprofiler.drivers import messaging # noqa
from osprofiler.drivers import mongodb # noqa
from osprofiler.drivers import redis_driver # noqa
+from osprofiler.drivers import sqlalchemy_driver # noqa
diff --git a/osprofiler/drivers/base.py b/osprofiler/drivers/base.py
index 6583a88..b85ffda 100644
--- a/osprofiler/drivers/base.py
+++ b/osprofiler/drivers/base.py
@@ -36,6 +36,12 @@ def get_driver(connection_string, *args, **kwargs):
connection_string)
backend = parsed_connection.scheme
+ # NOTE(toabctl): To be able to use the connection_string for as sqlalchemy
+ # connection string, transform the backend to the correct driver
+ # See https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls
+ if backend in ["mysql", "mysql+pymysql", "mysql+mysqldb",
+ "postgresql", "postgresql+psycopg2"]:
+ backend = "sqlalchemy"
for driver in _utils.itersubclasses(Driver):
if backend == driver.get_name():
return driver(connection_string, *args, **kwargs)
diff --git a/osprofiler/drivers/sqlalchemy_driver.py b/osprofiler/drivers/sqlalchemy_driver.py
new file mode 100644
index 0000000..c16a3ac
--- /dev/null
+++ b/osprofiler/drivers/sqlalchemy_driver.py
@@ -0,0 +1,119 @@
+# Copyright 2019 SUSE Linux GmbH
+# All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+import logging
+
+from oslo_serialization import jsonutils
+
+from osprofiler.drivers import base
+from osprofiler import exc
+
+LOG = logging.getLogger(__name__)
+
+
+class SQLAlchemyDriver(base.Driver):
+ def __init__(self, connection_str, project=None, service=None, host=None,
+ **kwargs):
+ super(SQLAlchemyDriver, self).__init__(connection_str, project=project,
+ service=service, host=host)
+
+ try:
+ from sqlalchemy import create_engine
+ from sqlalchemy import Table, MetaData, Column
+ from sqlalchemy import String, JSON, Integer
+ except ImportError:
+ raise exc.CommandError(
+ "To use this command, you should install 'SQLAlchemy'")
+
+ self._engine = create_engine(connection_str)
+ self._conn = self._engine.connect()
+ self._metadata = MetaData()
+ self._data_table = Table(
+ "data", self._metadata,
+ Column("id", Integer, primary_key=True),
+ # timestamp - date/time of the trace point
+ Column("timestamp", String(26), index=True),
+ # base_id - uuid common for all notifications related to one trace
+ Column("base_id", String(255), index=True),
+ # parent_id - uuid of parent element in trace
+ Column("parent_id", String(255), index=True),
+ # trace_id - uuid of current element in trace
+ Column("trace_id", String(255), index=True),
+ Column("project", String(255), index=True),
+ Column("host", String(255), index=True),
+ Column("service", String(255), index=True),
+ # name - trace point name
+ Column("name", String(255), index=True),
+ Column("data", JSON)
+ )
+
+ # FIXME(toabctl): Not the best idea to create the table on every
+ # startup when using the sqlalchemy driver...
+ self._metadata.create_all(self._engine, checkfirst=True)
+
+ @classmethod
+ def get_name(cls):
+ return "sqlalchemy"
+
+ def notify(self, info, context=None):
+ """Write a notification the the database"""
+ data = info.copy()
+ base_id = data.pop("base_id", None)
+ timestamp = data.pop("timestamp", None)
+ parent_id = data.pop("parent_id", None)
+ trace_id = data.pop("trace_id", None)
+ project = data.pop("project", self.project)
+ host = data.pop("host", self.host)
+ service = data.pop("service", self.service)
+ name = data.pop("name", None)
+
+ ins = self._data_table.insert().values(
+ timestamp=timestamp,
+ base_id=base_id,
+ parent_id=parent_id,
+ trace_id=trace_id,
+ project=project,
+ service=service,
+ host=host,
+ name=name,
+ data=jsonutils.dumps(data)
+ )
+ try:
+ self._conn.execute(ins)
+ except Exception:
+ LOG.exception("Can not store osprofiler tracepoint {} "
+ "(base_id {})".format(trace_id, base_id))
+
+ def get_report(self, base_id):
+ try:
+ from sqlalchemy.sql import select
+ except ImportError:
+ raise exc.CommandError(
+ "To use this command, you should install 'SQLAlchemy'")
+ stmt = select([self._data_table]).where(
+ self._data_table.c.base_id == base_id)
+ results = self._conn.execute(stmt).fetchall()
+ for n in results:
+ timestamp = n["timestamp"]
+ trace_id = n["trace_id"]
+ parent_id = n["parent_id"]
+ name = n["name"]
+ project = n["project"]
+ service = n["service"]
+ host = n["host"]
+ data = jsonutils.loads(n["data"])
+ self._append_results(trace_id, parent_id, name, project, service,
+ host, timestamp, data)
+ return self._parse_results()
--
2.21.0

92
0001-Don-t-fail-if-sqlalchemy-driver-fails-to-initialize.patch
View File

@ -1,92 +0,0 @@
From 1a024e60a796bb11936ea1d29c2c70e4160f53f1 Mon Sep 17 00:00:00 2001
From: Thomas Bechtold <tbechtold@suse.com>
Date: Mon, 11 Mar 2019 11:20:19 +0100
Subject: [PATCH] Don't fail if sqlalchemy driver fails to initialize
Given that the driver is initialized during the service
startup (eg. like Keystone, Nova, ...) a osprofiler driver failure has
a huge impact on the service (when the driver fails, the whole service
is not usable).
We want to avoid that and just log error/exceptions but keep the
services running.
Change-Id: I5688f10364884a74b7eb44c0c8bda15730ccd424
Closes-Bug: 1819433
---
osprofiler/drivers/sqlalchemy_driver.py | 58 ++++++++++++++-----------
1 file changed, 32 insertions(+), 26 deletions(-)
diff --git a/osprofiler/drivers/sqlalchemy_driver.py b/osprofiler/drivers/sqlalchemy_driver.py
index c16a3ac..06f3746 100644
--- a/osprofiler/drivers/sqlalchemy_driver.py
+++ b/osprofiler/drivers/sqlalchemy_driver.py
@@ -34,34 +34,40 @@ class SQLAlchemyDriver(base.Driver):
from sqlalchemy import Table, MetaData, Column
from sqlalchemy import String, JSON, Integer
except ImportError:
- raise exc.CommandError(
- "To use this command, you should install 'SQLAlchemy'")
+ LOG.exception("To use this command, install 'SQLAlchemy'")
+ else:
+ self._metadata = MetaData()
+ self._data_table = Table(
+ "data", self._metadata,
+ Column("id", Integer, primary_key=True),
+ # timestamp - date/time of the trace point
+ Column("timestamp", String(26), index=True),
+ # base_id - uuid common for all notifications related to
+ # one trace
+ Column("base_id", String(255), index=True),
+ # parent_id - uuid of parent element in trace
+ Column("parent_id", String(255), index=True),
+ # trace_id - uuid of current element in trace
+ Column("trace_id", String(255), index=True),
+ Column("project", String(255), index=True),
+ Column("host", String(255), index=True),
+ Column("service", String(255), index=True),
+ # name - trace point name
+ Column("name", String(255), index=True),
+ Column("data", JSON)
+ )
- self._engine = create_engine(connection_str)
- self._conn = self._engine.connect()
- self._metadata = MetaData()
- self._data_table = Table(
- "data", self._metadata,
- Column("id", Integer, primary_key=True),
- # timestamp - date/time of the trace point
- Column("timestamp", String(26), index=True),
- # base_id - uuid common for all notifications related to one trace
- Column("base_id", String(255), index=True),
- # parent_id - uuid of parent element in trace
- Column("parent_id", String(255), index=True),
- # trace_id - uuid of current element in trace
- Column("trace_id", String(255), index=True),
- Column("project", String(255), index=True),
- Column("host", String(255), index=True),
- Column("service", String(255), index=True),
- # name - trace point name
- Column("name", String(255), index=True),
- Column("data", JSON)
- )
+ # we don't want to kill any service that does use osprofiler
+ try:
+ self._engine = create_engine(connection_str)
+ self._conn = self._engine.connect()
- # FIXME(toabctl): Not the best idea to create the table on every
- # startup when using the sqlalchemy driver...
- self._metadata.create_all(self._engine, checkfirst=True)
+ # FIXME(toabctl): Not the best idea to create the table on every
+ # startup when using the sqlalchemy driver...
+ self._metadata.create_all(self._engine, checkfirst=True)
+ except Exception:
+ LOG.exception("Failed to create engine/connection and setup "
+ "intial database tables")
@classmethod
def get_name(cls):
--
2.21.0

4
_service
View File

@ -1,8 +1,8 @@
<services> <services>
<service mode="disabled" name="renderspec"> <service mode="disabled" name="renderspec">
<param name="input-template">https://raw.githubusercontent.com/openstack/rpm-packaging/stable/rocky/openstack/osprofiler/osprofiler.spec.j2</param> <param name="input-template">https://raw.githubusercontent.com/openstack/rpm-packaging/stable/stein/openstack/osprofiler/osprofiler.spec.j2</param>
<param name="output-name">python-osprofiler.spec</param> <param name="output-name">python-osprofiler.spec</param>
<param name="requirements">https://raw.githubusercontent.com/openstack/osprofiler/stable/rocky/requirements.txt</param> <param name="requirements">https://raw.githubusercontent.com/openstack/osprofiler/stable/stein/requirements.txt</param>
<param name="changelog-email">cloud-devel@suse.de</param> <param name="changelog-email">cloud-devel@suse.de</param>
<param name="changelog-provider">gh,openstack,osprofiler</param> <param name="changelog-provider">gh,openstack,osprofiler</param>
</service> </service>

3
osprofiler-2.3.0.tar.gz
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c183a706d820991a46202c98190c8d32a794d3cad676898f696670988f15b84d
size 80950

3
osprofiler-2.6.0.tar.gz Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:67d9abdbb08d739c532002ca412195036b9faff672ef2d297c550a75fea476c9
size 80518

31
python-osprofiler.changes
View File

@ -1,3 +1,34 @@
-------------------------------------------------------------------
Mon Apr 8 14:04:49 UTC 2019 - cloud-devel@suse.de
- update to version 2.6.0
- Switch to stestr
- When shortening span-ids, check if they're already short
- Do not insert osprofiler filter into Neutron api-paste pipeline
- Change python3.5 job to python3.7 job on Stein+
- import zuul job settings from project-config
- [devstack] Add support for elasticsearch backend
- Change openstack-dev to openstack-discuss
- Update reno for stable/rocky
- Update min tox version to 2.0
- Don't quote {posargs} in tox.ini
- add python 3.6 unit test job
- Make tracing of SQL statements configurable in DevStack plugin
- add lib-forward-testing-python3 test job
- build universal wheels
- Reload keystone to apply osprofiler config
- Allow test path to be overridden
- Add sqlalchemy collector
- Configure Jaeger collector in DevStack
- add password for connecting redis-sentinel
- Add support for mongodb backend in devstack plugin
- In DevStack install Redis client library via pip, not as system package
- Use $STACK_USER variable in install_jaeger function
- Change http to https in reference link
- In case of an error, always add message
- Use templates for cover and lower-constraints
- remove 0001-Add-sqlalchemy-collector.patch and 0001-Don-t-fail-if-sqlalchemy-driver-fails-to-initialize.patch
------------------------------------------------------------------- -------------------------------------------------------------------
Tue Mar 12 09:50:53 UTC 2019 - Thomas Bechtold <tbechtold@suse.com> Tue Mar 12 09:50:53 UTC 2019 - Thomas Bechtold <tbechtold@suse.com>

21
python-osprofiler.spec
View File

@ -17,22 +17,19 @@
Name: python-osprofiler Name: python-osprofiler
Version: 2.3.0 Version: 2.6.0
Release: 0 Release: 0
Summary: OpenStack Profiler Library Summary: OpenStack Profiler Library
License: Apache-2.0 License: Apache-2.0
Group: Development/Languages/Python Group: Development/Languages/Python
URL: https://launchpad.net/osprofiler URL: https://launchpad.net/osprofiler
Source0: https://files.pythonhosted.org/packages/source/o/osprofiler/osprofiler-2.3.0.tar.gz Source0: https://files.pythonhosted.org/packages/source/o/osprofiler/osprofiler-2.6.0.tar.gz
# backport sql collector -- https://github.com/openstack/osprofiler/commit/032a21861854c5f63a039c997a58b4a979e62750
Patch0: 0001-Add-sqlalchemy-collector.patch
# https://review.openstack.org/#/c/642407/
Patch1: 0001-Don-t-fail-if-sqlalchemy-driver-fails-to-initialize.patch
BuildRequires: openstack-macros BuildRequires: openstack-macros
BuildRequires: python-devel BuildRequires: python-devel
BuildRequires: python2-PrettyTable >= 0.7.2 BuildRequires: python2-PrettyTable >= 0.7.2
BuildRequires: python2-WebOb >= 1.7.1 BuildRequires: python2-WebOb >= 1.7.1
BuildRequires: python2-ddt BuildRequires: python2-ddt
BuildRequires: python2-docutils
BuildRequires: python2-elasticsearch BuildRequires: python2-elasticsearch
BuildRequires: python2-mock BuildRequires: python2-mock
BuildRequires: python2-oslo.concurrency >= 3.26.0 BuildRequires: python2-oslo.concurrency >= 3.26.0
@ -43,12 +40,13 @@ BuildRequires: python2-pymongo
BuildRequires: python2-python-subunit BuildRequires: python2-python-subunit
BuildRequires: python2-redis BuildRequires: python2-redis
BuildRequires: python2-six >= 1.10.0 BuildRequires: python2-six >= 1.10.0
BuildRequires: python2-testrepository BuildRequires: python2-stestr
BuildRequires: python2-testtools BuildRequires: python2-testtools
BuildRequires: python3-PrettyTable >= 0.7.2 BuildRequires: python3-PrettyTable >= 0.7.2
BuildRequires: python3-WebOb >= 1.7.1 BuildRequires: python3-WebOb >= 1.7.1
BuildRequires: python3-ddt BuildRequires: python3-ddt
BuildRequires: python3-devel BuildRequires: python3-devel
BuildRequires: python3-docutils
BuildRequires: python3-elasticsearch BuildRequires: python3-elasticsearch
BuildRequires: python3-mock BuildRequires: python3-mock
BuildRequires: python3-oslo.concurrency >= 3.26.0 BuildRequires: python3-oslo.concurrency >= 3.26.0
@ -59,7 +57,7 @@ BuildRequires: python3-pymongo
BuildRequires: python3-python-subunit BuildRequires: python3-python-subunit
BuildRequires: python3-redis BuildRequires: python3-redis
BuildRequires: python3-six >= 1.10.0 BuildRequires: python3-six >= 1.10.0
BuildRequires: python3-testrepository BuildRequires: python3-stestr
BuildRequires: python3-testtools BuildRequires: python3-testtools
Requires: python-PrettyTable >= 0.7.2 Requires: python-PrettyTable >= 0.7.2
Requires: python-WebOb >= 1.7.1 Requires: python-WebOb >= 1.7.1
@ -97,9 +95,8 @@ BuildRequires: python-openstackdocstheme
Documentation for OSProfiler. Documentation for OSProfiler.
%prep %prep
%autosetup -p1 -n osprofiler-2.3.0 %autosetup -p1 -n osprofiler-2.6.0
%py_req_cleanup %py_req_cleanup
sed -i 's/^warning-is-error.*/warning-is-error = 0/g' setup.cfg
%build %build
%{python_build} %{python_build}
@ -120,9 +117,7 @@ rm -rf doc/build/html/.{doctrees,buildinfo}
%python_uninstall_alternative osprofiler %python_uninstall_alternative osprofiler
%check %check
%{python_expand rm -rf .testrepository %python_exec -m stestr.cli run --black-regex '(^osprofiler.tests.unit.drivers.test_jaeger.JaegerTestCase.*$)'
$python setup.py testr --testr-args '(?!^osprofiler.tests.unit.drivers.test_jaeger.JaegerTestCase.*$)(^.*$)'
}
%files %{python_files} %files %{python_files}
%license LICENSE %license LICENSE