From: Zi Yan <zi.yan@sent•com>
To: David Hildenbrand <david@redhat•com>, linux-mm@kvack•org
Cc: Mel Gorman <mgorman@techsingularity•net>, Zi Yan <ziy@nvidia•com>,
Robin Murphy <robin.murphy@arm•com>,
linux-kernel@vger•kernel.org, iommu@lists•linux-foundation.org,
Eric Ren <renzhengeek@gmail•com>,
virtualization@lists•linux-foundation.org,
linuxppc-dev@lists•ozlabs.org, Christoph Hellwig <hch@lst•de>,
Vlastimil Babka <vbabka@suse•cz>,
Marek Szyprowski <m.szyprowski@samsung•com>
Subject: [RFC PATCH v3 1/8] mm: page_alloc: avoid merging non-fallbackable pageblocks with others.
Date: Wed, 5 Jan 2022 16:47:49 -0500 [thread overview]
Message-ID: <20220105214756.91065-2-zi.yan@sent.com> (raw)
In-Reply-To: <20220105214756.91065-1-zi.yan@sent.com>
From: Zi Yan <ziy@nvidia•com>
This is done in addition to MIGRATE_ISOLATE pageblock merge avoidance.
It prepares for the upcoming removal of the MAX_ORDER-1 alignment
requirement for CMA and alloc_contig_range().
MIGRARTE_HIGHATOMIC should not merge with other migratetypes like
MIGRATE_ISOLATE and MIGRARTE_CMA[1], so this commit prevents that too.
Also add MIGRARTE_HIGHATOMIC to fallbacks array for completeness.
[1] https://lore.kernel.org/linux-mm/20211130100853.GP3366@techsingularity.net/
Signed-off-by: Zi Yan <ziy@nvidia•com>
---
include/linux/mmzone.h | 6 ++++++
mm/page_alloc.c | 28 ++++++++++++++++++----------
2 files changed, 24 insertions(+), 10 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index aed44e9b5d89..0aa549653e4e 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -83,6 +83,12 @@ static inline bool is_migrate_movable(int mt)
return is_migrate_cma(mt) || mt == MIGRATE_MOVABLE;
}
+/* See fallbacks[MIGRATE_TYPES][3] in page_alloc.c */
+static inline bool migratetype_has_fallback(int mt)
+{
+ return mt < MIGRATE_PCPTYPES;
+}
+
#define for_each_migratetype_order(order, type) \
for (order = 0; order < MAX_ORDER; order++) \
for (type = 0; type < MIGRATE_TYPES; type++)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 8dd6399bafb5..5193c953dbf8 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1042,6 +1042,12 @@ buddy_merge_likely(unsigned long pfn, unsigned long buddy_pfn,
return page_is_buddy(higher_page, higher_buddy, order + 1);
}
+static inline bool has_non_fallback_pageblock(struct zone *zone)
+{
+ return has_isolate_pageblock(zone) || zone_cma_pages(zone) != 0 ||
+ zone->nr_reserved_highatomic != 0;
+}
+
/*
* Freeing function for a buddy system allocator.
*
@@ -1117,14 +1123,15 @@ static inline void __free_one_page(struct page *page,
}
if (order < MAX_ORDER - 1) {
/* If we are here, it means order is >= pageblock_order.
- * We want to prevent merge between freepages on isolate
- * pageblock and normal pageblock. Without this, pageblock
- * isolation could cause incorrect freepage or CMA accounting.
+ * We want to prevent merge between freepages on pageblock
+ * without fallbacks and normal pageblock. Without this,
+ * pageblock isolation could cause incorrect freepage or CMA
+ * accounting or HIGHATOMIC accounting.
*
* We don't want to hit this code for the more frequent
* low-order merging.
*/
- if (unlikely(has_isolate_pageblock(zone))) {
+ if (unlikely(has_non_fallback_pageblock(zone))) {
int buddy_mt;
buddy_pfn = __find_buddy_pfn(pfn, order);
@@ -1132,8 +1139,8 @@ static inline void __free_one_page(struct page *page,
buddy_mt = get_pageblock_migratetype(buddy);
if (migratetype != buddy_mt
- && (is_migrate_isolate(migratetype) ||
- is_migrate_isolate(buddy_mt)))
+ && (!migratetype_has_fallback(migratetype) ||
+ !migratetype_has_fallback(buddy_mt)))
goto done_merging;
}
max_order = order + 1;
@@ -2484,6 +2491,7 @@ static int fallbacks[MIGRATE_TYPES][3] = {
[MIGRATE_UNMOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, MIGRATE_TYPES },
[MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_TYPES },
[MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, MIGRATE_TYPES },
+ [MIGRATE_HIGHATOMIC] = { MIGRATE_TYPES }, /* Never used */
#ifdef CONFIG_CMA
[MIGRATE_CMA] = { MIGRATE_TYPES }, /* Never used */
#endif
@@ -2795,8 +2803,8 @@ static void reserve_highatomic_pageblock(struct page *page, struct zone *zone,
/* Yoink! */
mt = get_pageblock_migratetype(page);
- if (!is_migrate_highatomic(mt) && !is_migrate_isolate(mt)
- && !is_migrate_cma(mt)) {
+ /* Only reserve normal pageblock */
+ if (migratetype_has_fallback(mt)) {
zone->nr_reserved_highatomic += pageblock_nr_pages;
set_pageblock_migratetype(page, MIGRATE_HIGHATOMIC);
move_freepages_block(zone, page, MIGRATE_HIGHATOMIC, NULL);
@@ -3545,8 +3553,8 @@ int __isolate_free_page(struct page *page, unsigned int order)
struct page *endpage = page + (1 << order) - 1;
for (; page < endpage; page += pageblock_nr_pages) {
int mt = get_pageblock_migratetype(page);
- if (!is_migrate_isolate(mt) && !is_migrate_cma(mt)
- && !is_migrate_highatomic(mt))
+ /* Only change normal pageblock */
+ if (migratetype_has_fallback(mt))
set_pageblock_migratetype(page,
MIGRATE_MOVABLE);
}
--
2.34.1
next prev parent reply other threads:[~2022-01-05 21:51 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-05 21:47 [RFC PATCH v3 0/8] Use pageblock_order for cma and alloc_contig_range alignment Zi Yan
2022-01-05 21:47 ` Zi Yan [this message]
2022-01-12 10:54 ` [RFC PATCH v3 1/8] mm: page_alloc: avoid merging non-fallbackable pageblocks with others David Hildenbrand
2022-01-13 11:36 ` Mike Rapoport
2022-01-13 12:28 ` David Hildenbrand
2022-01-13 14:50 ` Zi Yan
2022-01-13 14:49 ` Zi Yan
2022-01-05 21:47 ` [RFC PATCH v3 2/8] mm: compaction: handle non-lru compound pages properly in isolate_migratepages_block() Zi Yan
2022-01-12 11:01 ` David Hildenbrand
2022-01-13 14:57 ` Zi Yan
2022-01-13 16:23 ` Zi Yan
2022-01-05 21:47 ` [RFC PATCH v3 3/8] mm: migrate: allocate the right size of non hugetlb or THP compound pages Zi Yan
2022-01-12 11:04 ` David Hildenbrand
2022-01-13 15:46 ` Zi Yan
2022-01-13 15:50 ` David Hildenbrand
2022-01-05 21:47 ` [RFC PATCH v3 4/8] mm: make alloc_contig_range work at pageblock granularity Zi Yan
2022-01-05 21:47 ` [RFC PATCH v3 5/8] mm: page_isolation: check specified range for unmovable pages during isolation Zi Yan
2022-01-14 13:38 ` David Hildenbrand
2022-01-14 15:14 ` Zi Yan
2022-01-05 21:47 ` [RFC PATCH v3 6/8] mm: cma: use pageblock_order as the single alignment Zi Yan
2022-01-05 21:47 ` [RFC PATCH v3 7/8] drivers: virtio_mem: use pageblock size as the minimum virtio_mem size Zi Yan
2022-01-14 13:44 ` David Hildenbrand
2022-01-14 15:15 ` Zi Yan
2022-01-05 21:47 ` [RFC PATCH v3 8/8] arch: powerpc: adjust fadump alignment to be pageblock aligned Zi Yan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220105214756.91065-2-zi.yan@sent.com \
--to=zi.yan@sent$(echo .)com \
--cc=david@redhat$(echo .)com \
--cc=hch@lst$(echo .)de \
--cc=iommu@lists$(echo .)linux-foundation.org \
--cc=linux-kernel@vger$(echo .)kernel.org \
--cc=linux-mm@kvack$(echo .)org \
--cc=linuxppc-dev@lists$(echo .)ozlabs.org \
--cc=m.szyprowski@samsung$(echo .)com \
--cc=mgorman@techsingularity$(echo .)net \
--cc=renzhengeek@gmail$(echo .)com \
--cc=robin.murphy@arm$(echo .)com \
--cc=vbabka@suse$(echo .)cz \
--cc=virtualization@lists$(echo .)linux-foundation.org \
--cc=ziy@nvidia$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox