Loading

×Sorry to interrupt
This page has an error. You might just need to refresh it.
[NoErrorObjectAvailable] Uncaught ReferenceError: addCTracking is not defined
Failing descriptor: {markup://c:KM_ArticleThemeLayout}

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

VMware Integrated Openstack Glance API Performance (58711)

Last Updated: 7/17/2023Categories: How toTotal Views: 781Language:
English

Subscribe to get an email
newsletter with new updates

Please provide article feedback:

Symptoms

Glance API requests through Horizon or through command line are hanging on:

GET https://[VIO-VIP/FQDN]:9292/v2/images

​For example, running the following API command:

time glance --insecure image-list Produces the following results:

+---------------------------------------------------------------------------------------+-------------------+
| URL | Seconds |
+---------------------------------------------------------------------------------------+-------------------+
| GET https://[VIO-VIP/FQDN]:5000/v3 | 0.03961 |
| POST https://[VIO-VIP/FQDN]:5000/v3/auth/tokens | 0.127194 |
| POST https://[VIO-VIP/FQDN]:5000/v3/auth/tokens | 0.099244 |
| GET https://[VIO-VIP/FQDN]:9292/v2/images | 41.269679 |
| GET https://[VIO-VIP/FQDN]:9292/v2/images?marker=00c32e86-85c6-4059-9e4d-4579dcf1b1d1 | 11.435379 |
| Total | 52.97110599999999 |
+---------------------------------------------------------------------------------------+-------------------+

Note:  Soft deleted records could be one of the reasons for performance problems.

Cause

Like most OpenStack systems, Glance performs soft deletions when it deletes records from its database. Depending on usage, cleanup of these records is periodically needed.

Checking the number of soft deleted records in these tables can be done by running the following.

osctl exec -it mariadb-server-0 bash
mysql --defaults-file=/etc/mysql/admin_user.cnf --host=localhost --connect-timeout 2

select count(*) from task_info;
select count(*) from image_properties where deleted_at !='NULL';
select count(*) from image_tags where deleted_at !='NULL';
select count(*) from image_locations where deleted_at !='NULL';
select count(*) from image_members where deleted_at !='NULL';
select count(*) from tasks where deleted_at !='NULL';

select count(*) from images where deleted_at !='NULL';

Impact / Risks

Warning: This procedure modifies the database. Ensure to take a backup of the deployment before proceeding.  See Backup and Restore

Resolution

Login to the glance-api pod

#osctl get pods | grep glance-api
#osctl exec -it <pod name from above> bash

Use the glance-manage tool to purge the soft-deleted rows from all tables except the images table:

bash-4.4$ glance-manage db purge
>2023-02-03 17:10:57.455 177 DEBUG oslo_db.sqlalchemy.engines [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3.7/site-packages/oslo_db/sqlalchemy/engines.py:305
>2023-02-03 17:10:57.519 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Purging deleted rows older than 30 day(s) from table task_info
>2023-02-03 17:10:57.564 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Deleted 0 row(s) from table task_info
>2023-02-03 17:10:57.589 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Purging deleted rows older than 30 day(s) from table image_properties
>2023-02-03 17:10:57.606 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Deleted 3 row(s) from table image_properties
>2023-02-03 17:10:57.615 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Purging deleted rows older than 30 day(s) from table image_tags
>2023-02-03 17:10:57.623 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Deleted 0 row(s) from table image_tags
>2023-02-03 17:10:57.641 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Purging deleted rows older than 30 day(s) from table image_locations
>2023-02-03 17:10:57.656 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Deleted 1 row(s) from table image_locations
>2023-02-03 17:10:57.668 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Purging deleted rows older than 30 day(s) from table image_members
>2023-02-03 17:10:57.678 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Deleted 0 row(s) from table image_members
>2023-02-03 17:10:57.678 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Purging deleted rows older than 30 day(s) from table tasks
>2023-02-03 17:10:57.684 177 INFO glance.db.sqlalchemy.api [req-7ea0d761-259d-42a6-8b9e-5f87dbbda9ea - - - - -] Deleted 0 row(s) from table tasks

Usage: This command takes two optional parameters:

--age_in_days NUM
Only purge rows that have been deleted for longer than NUM days. The default is 30 days.
--max_rows NUM
Purge a maximum of NUM rows from each table. The default is 100.

Use the glance-manage tool to purge the soft-deleted rows from the images table:

Note:  Openstack recommends that this is done less frequently than the above tables.  See Purging the Images Table

bash-4.4$ glance-manage db purge_images_table
2023-02-03 17:26:43.808 180 DEBUG oslo_db.sqlalchemy.engines [req-0c65abfe-f26b-4456-88c5-964d98c5c0d5 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3.7/site-packages/oslo_db/sqlalchemy/engines.py:305
2023-02-03 17:26:43.841 180 INFO glance.db.sqlalchemy.api [req-0c65abfe-f26b-4456-88c5-964d98c5c0d5 - - - - -] Purging deleted rows older than 180 day(s) from table images
2023-02-03 17:26:43.850 180 INFO glance.db.sqlalchemy.api [req-0c65abfe-f26b-4456-88c5-964d98c5c0d5 - - - - -] Deleted 0 row(s) from table images

Usage: This command takes two optional parameters:

--age_in_days NUM
Only purge rows that have been deleted for longer than NUM days. The default is 180 days.
--max_rows NUM
Purge a maximum of NUM rows from the images table. The default is 100.

Note:  This command can fail with an error similar to "Cannot delete or update a parent row: a foreign key constraint fails" This can happen when you try to purge records from the images table and related records have not been purged from other tables.  The purge_images_table command should only be used after the the regular purge command in Step 2 above.

Related Information

VIO purge Nova database of deleted instances

VIO Cleanup soft deleted volumes

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

No results for undefined

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware Integrated OpenStack

Related Versions:

VMware Integrated Openstack 7.x

Take Our Survey

Support Assistant