python – 分析和改进Django的ORM(SORT)生成的查询的性能
内容导读
互联网集市收集整理的这篇技术教程文章主要介绍了python – 分析和改进Django的ORM(SORT)生成的查询的性能，小编现在分享给大家，供广大互联网技能从业者学习和参考。文章包含13378字，纯文字阅读大概需要20分钟。
内容图文
基于我的Django模型中的几个条件,我有一个不那么复杂的(imho)过滤逻辑.有一个特定的查询需要花费很多时间才能完成.
查询是基于这两个查询集构建的：
queryset = self.serializer_class.Meta.model.valid_pricelist_objects.filter(
Q(drug_prices__pricelist__price_destination__to_all_insurances=True
) |
# pylint: disable=line-too-long
Q(
drug_prices__pricelist__price_destination__to_organization_data__organization__uuid
=self.kwargs.get('i_uuid')))
return queryset
和
return super().get_queryset().filter(
Q(active=True),
Q(drug_prices__pricelist__active=True), # Lista de precios activa
# Q(drug_pictures__is_main=True), # Que tenga una imagen
# TODO: Hacer filtros por pais PriceListCountries
Q(
Q(drug_prices__pricelist__expires=False) | # Que tenga precios que no caducan o
Q(
Q(drug_prices__pricelist__expires=True), # Que tenga precios que si caducan Y
Q(drug_prices__pricelist__datestart__date__lte=timezone.now()), # Fecha de inicio menor que hoy Y
Q(drug_prices__pricelist__dateend__date__gte=timezone.now()) # Fecha final mayor que hoy
)
)
).distinct().prefetch_related(
'categories__contexts',
'brands',
'drug_prices__pricelist',
'drug_pictures',
'drug_prices__pricelist__price_destination',
)
第二个查询集包装第一个(通过super()调用).生成的查询如下所示：
SELECT DISTINCT
"phdrug_phdrug"."id",
"phdrug_phdrug"."uuid",
"phdrug_phdrug"."default_description",
"phdrug_phdrug"."ean",
"phdrug_phdrug"."parent_ean",
"phdrug_phdrug"."reg_num",
"phdrug_phdrug"."atc_iv",
"phdrug_phdrug"."product_type",
"phdrug_phdrug"."fraction",
"phdrug_phdrug"."active",
"phdrug_phdrug"."loyal",
"phdrug_phdrug"."patent",
"phdrug_phdrug"."chronics",
"phdrug_phdrug"."recipe",
"phdrug_phdrug"."deal",
"phdrug_phdrug"."specialized",
"phdrug_phdrug"."armored",
"phdrug_phdrug"."hight_speciality",
"phdrug_phdrug"."temp_8_15",
"phdrug_phdrug"."temp_15_25",
"phdrug_phdrug"."temp_2_8",
"phdrug_phdrug"."temp_less_15",
"phdrug_phdrug"."new",
"phdrug_phdrug"."mdk_internal_code",
"phdrug_phdrug"."mdk_single_id",
"phdrug_phdrug"."is_from_mdk_db",
"phdrug_phdrug"."top",
"phdrug_phdrug"."laboratory_id",
"phdrug_phdrug"."specialty_id"
FROM
"phdrug_phdrug"
INNER JOIN "monetary_drugprice" ON ( "phdrug_phdrug"."id" = "monetary_drugprice"."drug_id" )
INNER JOIN "monetary_pricelist" ON ( "monetary_drugprice"."pricelist_id" = "monetary_pricelist"."id" )
INNER JOIN "monetary_drugprice" T4 ON ( "phdrug_phdrug"."id" = T4."drug_id" )
INNER JOIN "monetary_pricelist" T5 ON ( T4."pricelist_id" = T5."id" )
INNER JOIN "monetary_pricelistdestinations" ON ( T5."id" = "monetary_pricelistdestinations"."pricelist_id" )
LEFT OUTER JOIN "organization_organizationdata" ON ( "monetary_pricelistdestinations"."to_organization_data_id" = "organization_organizationdata"."id" )
LEFT OUTER JOIN "organization_organization" ON ( "organization_organizationdata"."organization_id" = "organization_organization"."id" )
WHERE
(
"phdrug_phdrug"."active" = TRUE
AND "monetary_pricelist"."active" = TRUE
AND (
"monetary_pricelist"."expires" = FALSE
OR (
"monetary_pricelist"."expires" = TRUE
AND ( "monetary_pricelist"."datestart" AT TIME ZONE'UTC' ) :: DATE <= '2019-01-22'
AND ( "monetary_pricelist"."dateend" AT TIME ZONE'UTC' ) :: DATE >= '2019-01-22'
))
AND (
"monetary_pricelistdestinations"."to_all_insurances" = TRUE
OR "organization_organization"."uuid" = 'b51773d4-05f8-43a2-86ef-0098b31725d8'
))
ORDER BY
"phdrug_phdrug"."default_description" ASC
使用EXPLAIN ANALYZE运行查询我得到：
Unique (cost=10412.31..12666.32 rows=29084 width=143) (actual time=3373.496..3620.090 rows=6442 loops=1)
-> Sort (cost=10412.31..10485.02 rows=29084 width=143) (actual time=3373.494..3460.790 rows=228667 loops=1)
Sort Key: phdrug_phdrug.default_description, phdrug_phdrug.id, phdrug_phdrug.uuid, phdrug_phdrug.ean, phdrug_phdrug.parent_ean, phdrug_phdrug.reg_num, phdrug_phdrug.medika_code, phdrug_phdrug.atc_iv, phdrug_phdrug.product_type, phdrug_phdrug.fraction, phdrug_phdrug.active, phdrug_phdrug.loyal, phdrug_phdrug.patent, phdrug_phdrug.chronics, phdrug_phdrug.recipe, phdrug_phdrug.deal, phdrug_phdrug.specialized, phdrug_phdrug.armored, phdrug_phdrug.hight_speciality, phdrug_phdrug.temp_8_15, phdrug_phdrug.temp_15_25, phdrug_phdrug.temp_2_8, phdrug_phdrug.temp_less_15, phdrug_phdrug.new, phdrug_phdrug.mdk_internal_code, phdrug_phdrug.mdk_single_id, phdrug_phdrug.is_from_mdk_db, phdrug_phdrug.top, phdrug_phdrug.laboratory_id, phdrug_phdrug.specialty_id
Sort Method: external merge Disk: 31192kB
-> Hash Join (cost=704.51..6166.54 rows=29084 width=143) (actual time=23.648..507.099 rows=228667 loops=1)
Hash Cond: (monetary_drugprice.pricelist_id = monetary_pricelist.id)
-> Nested Loop (cost=696.92..5604.95 rows=44105 width=147) (actual time=22.881..416.630 rows=457692 loops=1)
Join Filter: (phdrug_phdrug.id = monetary_drugprice.drug_id)
-> Hash Join (cost=696.51..1177.21 rows=4583 width=147) (actual time=22.864..38.841 rows=23577 loops=1)
Hash Cond: (phdrug_phdrug.id = t4.drug_id)
-> Seq Scan on phdrug_phdrug (cost=0.00..359.94 rows=11992 width=143) (actual time=0.438..3.593 rows=11992 loops=1)
Filter: active
Rows Removed by Filter: 2
-> Hash (cost=639.21..639.21 rows=4584 width=4) (actual time=22.339..22.339 rows=23577 loops=1)
Buckets: 32768 (originally 8192) Batches: 1 (originally 1) Memory Usage: 1085kB
-> Nested Loop (cost=3.99..639.21 rows=4584 width=4) (actual time=1.785..16.702 rows=23577 loops=1)
-> Nested Loop (cost=3.58..9.11 rows=5 width=8) (actual time=1.756..1.874 rows=7 loops=1)
-> Hash Left Join (cost=3.43..7.57 rows=5 width=4) (actual time=1.733..1.797 rows=7 loops=1)
Hash Cond: (monetary_pricelistdestinations.to_organization_data_id = organization_organizationdata.id)
Filter: (monetary_pricelistdestinations.to_all_insurances OR (organization_organization.uuid = 'b51773d4-05f8-43a2-86ef-0098b31725d8'::uuid))
Rows Removed by Filter: 130
-> Seq Scan on monetary_pricelistdestinations (cost=0.00..3.37 rows=137 width=9) (actual time=0.626..0.643 rows=137 loops=1)
-> Hash (cost=3.12..3.12 rows=25 width=20) (actual time=1.076..1.076 rows=25 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Hash Left Join (cost=1.56..3.12 rows=25 width=20) (actual time=1.040..1.053 rows=25 loops=1)
Hash Cond: (organization_organizationdata.organization_id = organization_organization.id)
-> Seq Scan on organization_organizationdata (cost=0.00..1.25 rows=25 width=8) (actual time=0.501..0.504 rows=25 loops=1)
-> Hash (cost=1.25..1.25 rows=25 width=20) (actual time=0.513..0.513 rows=25 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Seq Scan on organization_organization (cost=0.00..1.25 rows=25 width=20) (actual time=0.484..0.501 rows=25 loops=1)
-> Index Only Scan using monetary_pricelist_pkey on monetary_pricelist t5 (cost=0.14..0.31 rows=1 width=4) (actual time=0.007..0.007 rows=1 loops=7)
Index Cond: (id = monetary_pricelistdestinations.pricelist_id)
Heap Fetches: 7
-> Index Scan using monetary_drugprice_pricelist_id_1ce160ce on monetary_drugprice t4 (cost=0.42..110.21 rows=1581 width=8) (actual time=0.010..1.236 rows=3368 loops=7)
Index Cond: (pricelist_id = t5.id)
-> Index Scan using monetary_drugprice_drug_id_c2f278e5 on monetary_drugprice (cost=0.42..0.78 rows=15 width=8) (actual time=0.002..0.009 rows=19 loops=23577)
Index Cond: (drug_id = t4.drug_id)
-> Hash (cost=6.45..6.45 rows=91 width=4) (actual time=0.757..0.757 rows=93 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on monetary_pricelist (cost=0.00..6.45 rows=91 width=4) (actual time=0.655..0.713 rows=93 loops=1)
Filter: (active AND ((NOT expires) OR (expires AND ((timezone('UTC'::text, datestart))::date <= '2019-01-22'::date) AND ((timezone('UTC'::text, dateend))::date >= '2019-01-22'::date))))
Rows Removed by Filter: 45
Planning time: 25.871 ms
Execution time: 3638.544 ms
如果我将它粘贴在explain.depesz.com中,我会得到this result
几乎整个查询完成的时间都用于排序,如下所示：
……这就是我完全迷失的地方.什么是排序？在查询结束时不是ORDER BY,我已经尝试删除它.
如何提高此查询的性能？
编辑：我在删除ORDER BY后添加了EXPLAIN ANALYZE.查询需要更长时间才能完成.这是为问题添加另一层wtf.
Unique (cost=10412.31..12666.32 rows=29084 width=143) (actual time=4229.508..4427.560 rows=6442 loops=1)
-> Sort (cost=10412.31..10485.02 rows=29084 width=143) (actual time=4229.506..4274.698 rows=228667 loops=1)
Sort Key: phdrug_phdrug.id, phdrug_phdrug.uuid, phdrug_phdrug.default_description, phdrug_phdrug.ean, phdrug_phdrug.parent_ean, phdrug_phdrug.reg_num, phdrug_phdrug.medika_code, phdrug_phdrug.atc_iv, phdrug_phdrug.product_type, phdrug_phdrug.fraction, phdrug_phdrug.active, phdrug_phdrug.loyal, phdrug_phdrug.patent, phdrug_phdrug.chronics, phdrug_phdrug.recipe, phdrug_phdrug.deal, phdrug_phdrug.specialized, phdrug_phdrug.armored, phdrug_phdrug.hight_speciality, phdrug_phdrug.temp_8_15, phdrug_phdrug.temp_15_25, phdrug_phdrug.temp_2_8, phdrug_phdrug.temp_less_15, phdrug_phdrug.new, phdrug_phdrug.mdk_internal_code, phdrug_phdrug.mdk_single_id, phdrug_phdrug.is_from_mdk_db, phdrug_phdrug.top, phdrug_phdrug.laboratory_id, phdrug_phdrug.specialty_id
Sort Method: external merge Disk: 31160kB
-> Hash Join (cost=704.51..6166.54 rows=29084 width=143) (actual time=21.814..605.830 rows=228667 loops=1)
Hash Cond: (monetary_drugprice.pricelist_id = monetary_pricelist.id)
-> Nested Loop (cost=696.92..5604.95 rows=44105 width=147) (actual time=21.195..501.337 rows=457692 loops=1)
Join Filter: (phdrug_phdrug.id = monetary_drugprice.drug_id)
-> Hash Join (cost=696.51..1177.21 rows=4583 width=147) (actual time=21.180..43.205 rows=23577 loops=1)
Hash Cond: (phdrug_phdrug.id = t4.drug_id)
-> Seq Scan on phdrug_phdrug (cost=0.00..359.94 rows=11992 width=143) (actual time=0.491..5.225 rows=11992 loops=1)
Filter: active
Rows Removed by Filter: 2
-> Hash (cost=639.21..639.21 rows=4584 width=4) (actual time=20.589..20.589 rows=23577 loops=1)
Buckets: 32768 (originally 8192) Batches: 1 (originally 1) Memory Usage: 1085kB
-> Nested Loop (cost=3.99..639.21 rows=4584 width=4) (actual time=1.252..15.098 rows=23577 loops=1)
-> Nested Loop (cost=3.58..9.11 rows=5 width=8) (actual time=1.182..1.292 rows=7 loops=1)
-> Hash Left Join (cost=3.43..7.57 rows=5 width=4) (actual time=1.164..1.226 rows=7 loops=1)
Hash Cond: (monetary_pricelistdestinations.to_organization_data_id = organization_organizationdata.id)
Filter: (monetary_pricelistdestinations.to_all_insurances OR (organization_organization.uuid = 'b51773d4-05f8-43a2-86ef-0098b31725d8'::uuid))
Rows Removed by Filter: 130
-> Seq Scan on monetary_pricelistdestinations (cost=0.00..3.37 rows=137 width=9) (actual time=0.347..0.364 rows=137 loops=1)
-> Hash (cost=3.12..3.12 rows=25 width=20) (actual time=0.794..0.794 rows=25 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Hash Left Join (cost=1.56..3.12 rows=25 width=20) (actual time=0.774..0.786 rows=25 loops=1)
Hash Cond: (organization_organizationdata.organization_id = organization_organization.id)
-> Seq Scan on organization_organizationdata (cost=0.00..1.25 rows=25 width=8) (actual time=0.317..0.319 rows=25 loops=1)
-> Hash (cost=1.25..1.25 rows=25 width=20) (actual time=0.432..0.432 rows=25 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Seq Scan on organization_organization (cost=0.00..1.25 rows=25 width=20) (actual time=0.411..0.415 rows=25 loops=1)
-> Index Only Scan using monetary_pricelist_pkey on monetary_pricelist t5 (cost=0.14..0.31 rows=1 width=4) (actual time=0.006..0.006 rows=1 loops=7)
Index Cond: (id = monetary_pricelistdestinations.pricelist_id)
Heap Fetches: 7
-> Index Scan using monetary_drugprice_pricelist_id_1ce160ce on monetary_drugprice t4 (cost=0.42..110.21 rows=1581 width=8) (actual time=0.012..1.127 rows=3368 loops=7)
Index Cond: (pricelist_id = t5.id)
-> Index Scan using monetary_drugprice_drug_id_c2f278e5 on monetary_drugprice (cost=0.42..0.78 rows=15 width=8) (actual time=0.002..0.012 rows=19 loops=23577)
Index Cond: (drug_id = t4.drug_id)
-> Hash (cost=6.45..6.45 rows=91 width=4) (actual time=0.609..0.609 rows=93 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on monetary_pricelist (cost=0.00..6.45 rows=91 width=4) (actual time=0.539..0.582 rows=93 loops=1)
Filter: (active AND ((NOT expires) OR (expires AND ((timezone('UTC'::text, datestart))::date <= '2019-01-22'::date) AND ((timezone('UTC'::text, dateend))::date >= '2019-01-22'::date))))
Rows Removed by Filter: 45
Planning time: 25.288 ms
Execution time: 4440.406 ms
解决方法:
您在EXPLAIN ANALYZE结果中排序的原因是因为在您的查询中您使用的是SELECT DISTINCT,因此db按行对所有列进行排序并删除重复项.
查看您的查询可能您不需要对sql表达式中的所有列执行不同的操作.
将DISTINCT应用于您需要的列
尝试用.distinct(‘pk’)替换.distinct()
请参阅Django distinct()文档
On PostgreSQL only, you can pass positional arguments (*fields) in order to specify the names of fields to which the DISTINCT should apply. This translates to a SELECT DISTINCT ON SQL query. Here’s the difference. For a normal distinct() call, the database compares each field in each row when determining which rows are distinct. For a distinct() call with specified field names, the database will only compare the specified field names.
尝试更改排序方法
您的查询计划表明行的排序发生在磁盘上而不是内存中.
Sort Method: external merge Disk: 31160kB
您可以尝试another answer的解决方案
使用LIMIT
此外,您是否真的需要一批中的所有数据.你有没有尝试添加分页？
使用GROUP BY
我见过人们试图将他们的查询从DISTINCT转换为GROUP BY,但我不确定你的情况是否可行.
内容总结
以上是互联网集市为您收集整理的python – 分析和改进Django的ORM(SORT)生成的查询的性能全部内容，希望文章能够帮你解决python – 分析和改进Django的ORM(SORT)生成的查询的性能所遇到的程序开发问题。 如果觉得互联网集市技术教程内容还不错，欢迎将互联网集市网站推荐给程序员好友。
内容备注
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 gblab@vip.qq.com 举报，一经查实，本站将立刻删除。
内容手机端
扫描二维码推送至手机访问。
来源：【匿名】