[Amazon FSx for NetApp ONTAP] FlexGroupのリバランシング機能を試してみた

[Amazon FSx for NetApp ONTAP] FlexGroupのリバランシング機能を試してみた

FlexGroupのコンスティチュエント間のデータ使用量の偏りが原因でパフォーマンスに影響が出ている時に
Clock Icon2024.04.08

FlexGroupのコンスティチュエント間のデータ使用量の偏りが気になる

こんにちは、のんピ(@non____97)です。

皆さんはFlexGroupのコンスティチュエント間のデータ使用量の偏りが気になったことはありますか? 私はあります。

FlexGroupの内部では複数のFlexVolumeのコンスティチュエント(メンバーボリューム)を束ねて、1つのボリュームとして見せています。FlexGroupの詳細は以下記事をご覧ください。

FlexGroupを運用しているとのコンスティチュエント間のデータ使用量の偏りが出てくることがあります。特にコンスティチュエントを追加した場合は偏ります。ONTAP側で良きようにバランスを取って書き込みをしてくれますが、大きいファイルの追加や削除をした場合は不均衡になりがちです。

  • ONTAP は常にメンバーボリュームの運用状態を監視し、 FlexGroup ボリュームの現在の状態に基づいて配置の決定を調整します。
    • ONTAP 9.8 よりも前のバージョンでは、 1 つのメンバーボリュームが他のメンバーボリュームとのバランスが少し取れていない場合、取り込みの調整が行われない可能性があります。
    • しかし、このメンバーボリュームが容量の 90% に近づくか、 512GB を超える容量不一致が発生する場合、 ONTAP は新しいデータをより積極的に配置して不均衡を解消します。
    • この調整によって、他のメンバーボリュームへのリモートファイル配置が増えることになり、 FlexGroup ボリュームのパフォーマンスが低下する(ただし顕著ではない場合もある - 約 5~10% )ことがあります。

FlexGroup コンスティチュエントのバランスが取れないのはなぜですか? - NetApp

コンスティチュエントのデータが偏るとCPU使用率が高まることがあるようです。

今までは時の流れに身を委ねるしかありませんでしたが、ONTAP 9.12.1以降では、コンスティチュエント間でファイルを無停止で移動することができるようになりました。

実際に試してみました。

いきなりまとめ

  • 無停止でFlexGroupのコンスティチュエント間のファイルを移動できる
  • リバランシング可能なファイルはSnapshotに含まれないものだけなど、注意点はいくつかある
  • スケジュール実行することも可能
    • 定期実行ではなく、指定したタイミングに一回だけ実行
  • 運用の中で頻繁に実行するものではなく、必要性に応じて実行するもの

やってみた

FlexGroupの作成

まず、FlexGroupボリュームを作成します。コンスティチュエントは2です。

::*> volume create -vserver svm -volume fg -aggr-list aggr1 -aggr-list-multiplier 2 -size 200GB

Notice: The FlexGroup volume "fg" will be created with the following number of constituents of size 100GB: 2.
Do you want to continue? {y|n}: y

Warning: You are attempting to create a FlexGroup volume in Vserver "svm", which currently uses 32-bit NFSv3 FSIDs and file IDs. This could result in collisions between different file IDs on the FlexGroup
         volume. Modify the Vserver to use 64-bit NFSv3 identifiers before using the FlexGroup volume with the commands "set advanced; vserver nfs modify -vserver svm -v3-64bit-identifiers enabled".
Do you want to continue? {y|n}: y
[Job 58] Job succeeded: Successful

::*> volume show -volume fg*
Vserver   Volume       Aggregate    State      Type       Size  Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
svm       fg           -            online     RW        200GB    189.9GB    0%
svm       fg__0001     aggr1        online     RW        100GB    94.94GB    0%
svm       fg__0002     aggr1        online     RW        100GB    94.94GB    0%
3 entries were displayed.

::*> volume mount -volume fg -junction-path /fg
Queued private job: 29

テストファイルの作成

FlexGroupボリューム上にテストファイルを作成します。

$ sudo mkdir -p /mnt/fsxn/fg
$ sudo mount -t nfs svm-0365ba78d7ad91348.fs-009351b227391d1f1.fsx.us-east-1.amazonaws.com:/fg /mnt/fsxn/fg
$ df -hT -t nfs4
Filesystem                                                                               Type  Size  Used Avail Use% Mounted on
svm-0365ba78d7ad91348.fs-009351b227391d1f1.fsx.us-east-1.amazonaws.com:/fg               nfs4  190G  973M  190G   1% /mnt/fsxn/fg

$ for i in {33..48}; do
    sudo dd if=/dev/urandom of=/mnt/fsxn/fg/random_pattern_binary_block_${i} bs=1M count=512
done

$ ls -l /mnt/fsxn/fg
total 8421888
-rw-r--r--. 1 root root 268435456 Apr  7 06:59 random_pattern_binary_block_1
-rw-r--r--. 1 root root 268435456 Apr  7 06:59 random_pattern_binary_block_10
-rw-r--r--. 1 root root 268435456 Apr  7 07:00 random_pattern_binary_block_11
.
.
(中略)
.
.
-rw-r--r--. 1 root root 268435456 Apr  7 06:59 random_pattern_binary_block_7
-rw-r--r--. 1 root root 268435456 Apr  7 06:59 random_pattern_binary_block_8
-rw-r--r--. 1 root root 268435456 Apr  7 06:59 random_pattern_binary_block_9

$ df -hT -t nfs4
Filesystem                                                                               Type  Size  Used Avail Use% Mounted on
svm-0365ba78d7ad91348.fs-009351b227391d1f1.fsx.us-east-1.amazonaws.com:/fg               nfs4  190G  9.1G  181G   5% /mnt/fsxn/fg

テストファイル作成後、ボリュームの状態を確認します。fg__0001というコンスティチュエントにほとんどのファイルが保存されていることが分かります。

::*> volume show -volume fg* -fields available, filesystem-size, total, used, percent-used, performance-tier-inactive-user-data, performance-tier-inactive-user-data-percent, size, dedupe-space-saved, dedupe-space-saved-percent, dedupe-space-shared,logical-used, logical-used-percent,logical-used-by-afs, logical-available, physical-used, physical-used-percent, tiering-policy
vserver volume size  available filesystem-size total used   percent-used dedupe-space-saved dedupe-space-saved-percent dedupe-space-shared physical-used physical-used-percent logical-used logical-used-percent logical-available logical-used-by-afs tiering-policy performance-tier-inactive-user-data performance-tier-inactive-user-data-percent
------- ------ ----- --------- --------------- ----- ------ ------------ ------------------ -------------------------- ------------------- ------------- --------------------- ------------ -------------------- ----------------- ------------------- -------------- ----------------------------------- -------------------------------------------
svm     fg     200GB 180.9GB   200GB           190GB 9.06GB 4%           0B                 0%                         0B                  9.06GB        5%                    9.06GB       5%                   -                 9.06GB              none           0B                                  0%
svm     fg__0001
               100GB 86.92GB   100GB           95GB  8.08GB 8%           0B                 0%                         0B                  8.08GB        8%                    8.08GB       9%                   -                 8.08GB              none           0B                                  0%
svm     fg__0002
               100GB 94.02GB   100GB           95GB  1005MB 1%           0B                 0%                         0B                  1005MB        1%                    1005MB       1%                   -                 1005MB              none           0B                                  0%
3 entries were displayed.

リバランシング

リバランシングをします。

いくつか注意点があるので気をつけながら実行しましょう。Snapshotや重複排除周りは特に気になりますね。リバランシングは頻繁にやるものではなさそうです。

  • FlexVol からFlexGroup への変換
    FlexVol からFlexGroup への変換後は、FlexGroup の自動リバランシングを使用しないことを推奨します。代わりに、ONTAP 9.10.1以降で使用可能なシステム停止を伴う逆アクティブファイル移動機能を使用するには、を入力します volume rebalance file-move コマンドを実行しますコマンド構文については、を参照してください volume rebalance file-move start のマニュアルページ。
    FlexGroupの自動リバランシング機能を使用したリバランシングでは、FlexVolからFlexGroupへの変換を実行し、FlexVolボリューム上のデータの50~85%が新しいコンスティチュエントに移動されるなど、多数のファイルを移動する際のパフォーマンスが低下する可能性があります。
  • ファイルの最小サイズと最大サイズ
    LIFの自動リバランシングで選択されるファイルは、保存されたブロックに基づいています。 リバランシングのために考慮される最小ファイルサイズはデフォルトで100MB(下記のmin-file-sizeパラメータを使用して20MBまで設定可能)で、最大ファイルサイズは100GBです。
  • Snapshotコピー内のファイル
    FlexGroup のリバランシングを設定して、Snapshotコピーに現在存在しないファイルのみを移動することができます。 リバランシングが開始されると、リバランシング処理中にいつでもSnapshotコピー処理がスケジュールされているかどうかが通知されます。
    ファイルの移動中で、デスティネーションでフレーミングが実行されている場合、Snapshotコピーは制限されます。 ファイルのリバランシングが実行中の場合、Snapshotコピーのリストア処理は実行できません。
  • SnapMirror 処理
    FlexGroup のリバランシングは、スケジュールされたSnapMirror処理の間に行う必要があります。SnapMirror処理の開始前にファイルを再配置している場合、そのファイルの移動が24分間のSnapMirror再試行期間内に完了しないと、SnapMirror処理が失敗することがあります。 SnapMirror転送の開始後に開始される新しいファイルの再配置は失敗しません。
  • ファイルベースの圧縮のストレージ効率化
    ファイルベースの圧縮によるストレージ効率化では、ファイルはデスティネーションに移動する前に解凍されるため、圧縮による削減が失われます。リバランシング後に手動で開始したバックグラウンドスキャナをFlexGroup で実行した場合、圧縮による削減効果が再び得られます。 ただし、いずれかのボリューム上のSnapshotコピーに関連付けられたファイルは、圧縮の対象として無視されます。
  • 重複排除
    重複排除されたファイルを移動すると、原因 でFlexGroup ボリュームの全体的な使用量が増加する可能性がファイルのリバランシング時には、一意のブロックのみがデスティネーションに移動され、ソースの容量が解放されます。 共有ブロックはソースに保持され、デスティネーションにコピーされます。 このため、ほぼフルのソースコンスティチュエントで使用済み容量を減らすことは目標ですが、新しいデスティネーションに共有ブロックがコピーされるため、FlexGroup ボリューム全体の使用量が増加することもあります。 また、Snapshotコピーの一部であるファイルを移動する場合にも使用できます。Snapshotコピースケジュールがリサイクルされるまでスペース削減は完全には認識されず、Snapshotコピー内のファイルのコピーも削除されます。
  • FlexClone ボリューム
    FlexCloneボリュームの作成時にファイルのリバランシングが実行中の場合、FlexCloneボリュームではリバランシングは実行されません。FlexCloneボリュームでのリバランシングは、FlexCloneボリュームの作成後に実行する必要があります。
  • ファイル移動
    FlexGroup のリバランシング処理中にファイルが移動されると、ソースとデスティネーションの両方のコンスティチュエントについて、クォータアカウンティングの一部としてファイルサイズが報告されます。 移動が完了すると、クォータアカウンティングは通常に戻り、ファイルサイズは新しいデスティネーションでのみ報告されます。
  • 自律的なランサムウェア防御
    ONTAP 9.13.1以降では、システムの停止を伴うリバランシング処理と無停止のリバランシング処理で自律型ランサムウェア対策がサポートされます。
  • オブジェクトストアボリューム
    ボリューム容量のリバランシングは、S3バケットなどのオブジェクトストアボリュームではサポートされていません。

FlexGroup ボリュームのリバランシング

リバランシング前に、リバランシングの状態と設定を確認します。

::*> volume rebalance show

Vserver: svm
                                         Used      Size
                                          For   Used by  Target     Imbalance
Volume      State             Total Imbalance Snapshots    Used    Size      %
----------- -------------- -------- --------- -------- -------- -------- -----
fg          not-running       200GB    8.59GB       0B        -   3.55GB   41%
fg__0001    not-running       100GB    7.84GB       0B   4.29GB   3.55GB   82%
fg__0002    not-running       100GB   762.6MB       0B   4.29GB  -3.55GB  -82%
3 entries were displayed.

::*> volume rebalance show -config

Vserver: svm
                     Max      Threshold     Max        Min     Exclude
Volume             Runtime    Min   Max  File Moves File Size  Snapshot
--------------- ------------ ----- ----- ---------- --------- ---------
fg                    6h0m0s    5%   20%         25     100MB true
fg__0001              6h0m0s    5%   20%         25     100MB true
fg__0002              6h0m0s    5%   20%         25     100MB true
3 entries were displayed.

::*> volume rebalance show -instance

                                     Vserver Name: svm
                                      Volume Name: fg
                                   Is Constituent: false
                                  Rebalance State: not-running
                        Rebalance Notice Messages: -
                                       Total Size: 200GB
              Used Size for Calculating Imbalance: 8.59GB
              Volume Size Used by Snapshot Copies: 332KB
                     Constituent Target Used Size: -
                                   Imbalance Size: 3.55GB
                             Imbalance Percentage: 41%
                                  Moved Data Size: -
         Maximum Constituent Imbalance Percentage: 82%
                             Rebalance Start Time: -
                              Rebalance Stop Time: -
                                Rebalance Runtime: -
                        Rebalance Maximum Runtime: 6h0m0s
      Maximum Imbalance Threshold per Constituent: 20%
      Minimum Imbalance Threshold per Constituent: 5%
    Maximum Concurrent File Moves per Constituent: 25
                                Minimum File Size: 100MB
           Exclude Files Stuck in Snapshot Copies: true
                  Most Recent File Move Timestamp: -
                               File Moves Started: 0
                                    Files Scanned: 0
                              Small Files Skipped: 0
                              Large Files Skipped: 0
                      Fast Truncate Files Skipped: 0
                 Files in Snapshot Copies Skipped: 0
 Files Skipped Due to Efficiency (by Block Count): 0
     Files Skipped Due to Efficiency (by Percent): 0
                       Incompatible Files Skipped: 0
                           Metadata Files Skipped: 0
                            Remote Caches Skipped: 0
                       Write Fenced Files Skipped: 0
              On Demand Destination Files Skipped: 0
                  Invalid Footprint Files Skipped: 0
                          All Other Files Skipped: 0
                                   Blocks Scanned: 0
                Blocks Skipped Due to Small Files: 0
                Blocks Skipped Due to Large Files: 0
        Blocks Skipped Due to Fast Truncate Files: 0
   Blocks Skipped Due to Files in Snapshot Copies: 0
Blocks Skipped Due to Efficiency (by Block Count): 0
    Blocks Skipped Due to Efficiency (by Percent): 0
         Blocks Skipped Due to Incompatible Files: 0
             Blocks Skipped Due to Metadata Files: 0
              Blocks Skipped Due to Remote Caches: 0
         Blocks Skipped Due to Write Fenced Files: 0
Blocks Skipped Due to On Demand Destination Files: 0
    Blocks Skipped Due to Invalid Footprint Files: 0
          Blocks Skipped Due to All Other Reasons: 0
                   Last File Move Error Timestamp: -
                     Last File Move Error File ID: -
Last File Move Error Destination Constituent DSID: -
                        Last File Move Error Code: -

                                     Vserver Name: svm
                                      Volume Name: fg__0001
                                   Is Constituent: true
                                  Rebalance State: not-running
                        Rebalance Notice Messages: -
                                       Total Size: 100GB
              Used Size for Calculating Imbalance: 7.84GB
              Volume Size Used by Snapshot Copies: 168KB
                     Constituent Target Used Size: 4.29GB
                                   Imbalance Size: 3.55GB
                             Imbalance Percentage: 82%
                                  Moved Data Size: -
         Maximum Constituent Imbalance Percentage: -
                             Rebalance Start Time: -
                              Rebalance Stop Time: -
                                Rebalance Runtime: -
                        Rebalance Maximum Runtime: 6h0m0s
      Maximum Imbalance Threshold per Constituent: 20%
      Minimum Imbalance Threshold per Constituent: 5%
    Maximum Concurrent File Moves per Constituent: 25
                                Minimum File Size: 100MB
           Exclude Files Stuck in Snapshot Copies: true
                  Most Recent File Move Timestamp: -
                               File Moves Started: 0
                                    Files Scanned: 0
                              Small Files Skipped: 0
                              Large Files Skipped: 0
                      Fast Truncate Files Skipped: 0
                 Files in Snapshot Copies Skipped: 0
 Files Skipped Due to Efficiency (by Block Count): 0
     Files Skipped Due to Efficiency (by Percent): 0
                       Incompatible Files Skipped: 0
                           Metadata Files Skipped: 0
                            Remote Caches Skipped: 0
                       Write Fenced Files Skipped: 0
              On Demand Destination Files Skipped: 0
                  Invalid Footprint Files Skipped: 0
                          All Other Files Skipped: 0
                                   Blocks Scanned: 0
                Blocks Skipped Due to Small Files: 0
                Blocks Skipped Due to Large Files: 0
        Blocks Skipped Due to Fast Truncate Files: 0
   Blocks Skipped Due to Files in Snapshot Copies: 0
Blocks Skipped Due to Efficiency (by Block Count): 0
    Blocks Skipped Due to Efficiency (by Percent): 0
         Blocks Skipped Due to Incompatible Files: 0
             Blocks Skipped Due to Metadata Files: 0
              Blocks Skipped Due to Remote Caches: 0
         Blocks Skipped Due to Write Fenced Files: 0
Blocks Skipped Due to On Demand Destination Files: 0
    Blocks Skipped Due to Invalid Footprint Files: 0
          Blocks Skipped Due to All Other Reasons: 0
                   Last File Move Error Timestamp: -
                     Last File Move Error File ID: -
Last File Move Error Destination Constituent DSID: -
                        Last File Move Error Code: -

                                     Vserver Name: svm
                                      Volume Name: fg__0002
                                   Is Constituent: true
                                  Rebalance State: not-running
                        Rebalance Notice Messages: -
                                       Total Size: 100GB
              Used Size for Calculating Imbalance: 762.6MB
              Volume Size Used by Snapshot Copies: 164KB
                     Constituent Target Used Size: 4.29GB
                                   Imbalance Size: -3.55GB
                             Imbalance Percentage: -82%
                                  Moved Data Size: -
         Maximum Constituent Imbalance Percentage: -
                             Rebalance Start Time: -
                              Rebalance Stop Time: -
                                Rebalance Runtime: -
                        Rebalance Maximum Runtime: 6h0m0s
      Maximum Imbalance Threshold per Constituent: 20%
      Minimum Imbalance Threshold per Constituent: 5%
    Maximum Concurrent File Moves per Constituent: 25
                                Minimum File Size: 100MB
           Exclude Files Stuck in Snapshot Copies: true
                  Most Recent File Move Timestamp: -
                               File Moves Started: 0
                                    Files Scanned: 0
                              Small Files Skipped: 0
                              Large Files Skipped: 0
                      Fast Truncate Files Skipped: 0
                 Files in Snapshot Copies Skipped: 0
 Files Skipped Due to Efficiency (by Block Count): 0
     Files Skipped Due to Efficiency (by Percent): 0
                       Incompatible Files Skipped: 0
                           Metadata Files Skipped: 0
                            Remote Caches Skipped: 0
                       Write Fenced Files Skipped: 0
              On Demand Destination Files Skipped: 0
                  Invalid Footprint Files Skipped: 0
                          All Other Files Skipped: 0
                                   Blocks Scanned: 0
                Blocks Skipped Due to Small Files: 0
                Blocks Skipped Due to Large Files: 0
        Blocks Skipped Due to Fast Truncate Files: 0
   Blocks Skipped Due to Files in Snapshot Copies: 0
Blocks Skipped Due to Efficiency (by Block Count): 0
    Blocks Skipped Due to Efficiency (by Percent): 0
         Blocks Skipped Due to Incompatible Files: 0
             Blocks Skipped Due to Metadata Files: 0
              Blocks Skipped Due to Remote Caches: 0
         Blocks Skipped Due to Write Fenced Files: 0
Blocks Skipped Due to On Demand Destination Files: 0
    Blocks Skipped Due to Invalid Footprint Files: 0
          Blocks Skipped Due to All Other Reasons: 0
                   Last File Move Error Timestamp: -
                     Last File Move Error File ID: -
Last File Move Error Destination Constituent DSID: -
                        Last File Move Error Code: -
3 entries were displayed.

リバランシング後の見込みボリューム使用量が表示されるのはありがたいですね。

リバランシングをします。

::*> volume rebalance start -volume fg

Error: command failed: Failed to start volume capacity rebalancing for volume "fg" in Vserver "svm". Reason: The next scheduled Snapshot copy for volume "fg" in Vserver "svm" is scheduled for Sun Apr  7
       08:05:00 2024, in 0h58m33s. This is within the specified 6h0m0s maximum runtime for the volume capacity rebalancing operation. To run the operation, either reduce the "-max-runtime" or disable the
       Snapshot policy for the volume.

リバランシングの最大実行時間に対して、Snapshotの取得間隔が短いためエラーとなりました。

Snapshotの取得間隔が6時間未満であることはよくあると思います。実際にリバランシングの運用をする際には最大実行時間をデフォルトの6時間から調整すると良さそうです。

今回はSnapshotポリシーをnoneに変更して、定期的にSnapshotを取得しないようにします。

::*> volume modify -volume fg -snapshot-policy none

Warning: You are changing the Snapshot policy on volume "fg" to "none". Snapshot copies on this volume that do not match any of the prefixes of the new Snapshot policy will not be deleted. However, when the
         new Snapshot policy takes effect, depending on the new retention count, any existing Snapshot copies that continue to use the same prefixes might be deleted. See the 'volume modify' man page for more
         information.
Do you want to continue? {y|n}: y
[Job 61] Job succeeded: volume modify succeeded

再度リバランシングをしようとします。

::*> volume rebalance start -volume fg

Warning: Granular data will be enabled for volume "fg" in Vserver "svm". Once enabled, this property can only be disabled by restoring a Snapshot copy. Earlier versions of ONTAP are not compatible with this
         property, therefore revert to an earlier version of ONTAP will not be possible unless this volume is deleted or restored to a Snapshot copy that was taken before this property was enabled.
Do you want to continue? {y|n}:

今度はGranular dataを有効化すると警告されました。

Granular Dataはリバランシングをするためには必ず有効にする必要があるようです。

Granular Data

ONTAP 9.12.1 introduced the Granular Data management feature for FlexGroups. This feature allows files on a FlexGroup to be moved non-disruptively from one constituent to another.

Granular Data is primarily used by FlexGroup rebalancing. Files that are moved during the rebalancing process will gain an additional management layer just as with remote hardlinks. A file can have both the remote hardlink and Granular Data layers. The rebalancing feature has controls that can limit what kind of files will be chosen for rebalancing. Depending on the settings used for rebalancing, there can be many files that gain the additional layer. Like with remote hardlinks, a file that has this layer will increase the scan time.

TR-4867: Best Practice Guidelines for ONTAP FIle System Analytics

確かにGranular Dataはデフォルトでは有効ではないですね。

::*> volume show -volume fg* -fields granular-data, is-autobalance-eligible
vserver volume is-autobalance-eligible granular-data
------- ------ ----------------------- -------------
svm     fg     false                   disabled
svm     fg__0001
               true                    disabled
svm     fg__0002
               true                    disabled
3 entries were displayed.

特に追加コストがかかったり、パフォーマンスに影響が出たりという記載は見つけられませんでした。

気を取り直して、再度リバランシングをします。

::*> volume rebalance start -volume fg

Warning: Granular data will be enabled for volume "fg" in Vserver "svm". Once enabled, this property can only be disabled by restoring a Snapshot copy. Earlier versions of ONTAP are not compatible with this
         property, therefore revert to an earlier version of ONTAP will not be possible unless this volume is deleted or restored to a Snapshot copy that was taken before this property was enabled.
Do you want to continue? {y|n}: y

リバランシング中の状態を確認します。

::*> volume rebalance show -instance

                                     Vserver Name: svm
                                      Volume Name: fg
                                   Is Constituent: false
                                  Rebalance State: rebalancing
                        Rebalance Notice Messages: -
                                       Total Size: 200GB
              Used Size for Calculating Imbalance: 8.59GB
              Volume Size Used by Snapshot Copies: 332KB
                     Constituent Target Used Size: -
                                   Imbalance Size: 3.55GB
                             Imbalance Percentage: 41%
                                  Moved Data Size: -
         Maximum Constituent Imbalance Percentage: 82%
                             Rebalance Start Time: Sun Apr 07 07:16:47 2024
                              Rebalance Stop Time: -
                                Rebalance Runtime: 0h0m37s
                        Rebalance Maximum Runtime: 6h0m0s
      Maximum Imbalance Threshold per Constituent: 20%
      Minimum Imbalance Threshold per Constituent: 5%
    Maximum Concurrent File Moves per Constituent: 25
                                Minimum File Size: 100MB
           Exclude Files Stuck in Snapshot Copies: true
                  Most Recent File Move Timestamp: -
                               File Moves Started: 0
                                    Files Scanned: 90
                              Small Files Skipped: 0
                              Large Files Skipped: 0
                      Fast Truncate Files Skipped: 0
                 Files in Snapshot Copies Skipped: 90
 Files Skipped Due to Efficiency (by Block Count): 0
     Files Skipped Due to Efficiency (by Percent): 0
                       Incompatible Files Skipped: 0
                           Metadata Files Skipped: 0
                            Remote Caches Skipped: 0
                       Write Fenced Files Skipped: 0
              On Demand Destination Files Skipped: 0
                  Invalid Footprint Files Skipped: 0
                          All Other Files Skipped: 0
                                   Blocks Scanned: 5921640
                Blocks Skipped Due to Small Files: 0
                Blocks Skipped Due to Large Files: 0
        Blocks Skipped Due to Fast Truncate Files: 0
   Blocks Skipped Due to Files in Snapshot Copies: 5921640
Blocks Skipped Due to Efficiency (by Block Count): 0
    Blocks Skipped Due to Efficiency (by Percent): 0
         Blocks Skipped Due to Incompatible Files: 0
             Blocks Skipped Due to Metadata Files: 0
              Blocks Skipped Due to Remote Caches: 0
         Blocks Skipped Due to Write Fenced Files: 0
Blocks Skipped Due to On Demand Destination Files: 0
    Blocks Skipped Due to Invalid Footprint Files: 0
          Blocks Skipped Due to All Other Reasons: 0
                   Last File Move Error Timestamp: -
                     Last File Move Error File ID: -
Last File Move Error Destination Constituent DSID: -
                        Last File Move Error Code: -

                                     Vserver Name: svm
                                      Volume Name: fg__0001
                                   Is Constituent: true
                                  Rebalance State: scanning
                        Rebalance Notice Messages: -
                                       Total Size: 100GB
              Used Size for Calculating Imbalance: 7.84GB
              Volume Size Used by Snapshot Copies: 168KB
                     Constituent Target Used Size: 4.29GB
                                   Imbalance Size: 3.55GB
                             Imbalance Percentage: 82%
                                  Moved Data Size: 0B
         Maximum Constituent Imbalance Percentage: -
                             Rebalance Start Time: Sun Apr 07 07:16:47 2024
                              Rebalance Stop Time: -
                                Rebalance Runtime: 0h0m37s
                        Rebalance Maximum Runtime: 6h0m0s
      Maximum Imbalance Threshold per Constituent: 20%
      Minimum Imbalance Threshold per Constituent: 5%
    Maximum Concurrent File Moves per Constituent: 25
                                Minimum File Size: 100MB
           Exclude Files Stuck in Snapshot Copies: true
                  Most Recent File Move Timestamp: -
                               File Moves Started: 0
                                    Files Scanned: 90
                              Small Files Skipped: 0
                              Large Files Skipped: 0
                      Fast Truncate Files Skipped: 0
                 Files in Snapshot Copies Skipped: 90
 Files Skipped Due to Efficiency (by Block Count): 0
     Files Skipped Due to Efficiency (by Percent): 0
                       Incompatible Files Skipped: 0
                           Metadata Files Skipped: 0
                            Remote Caches Skipped: 0
                       Write Fenced Files Skipped: 0
              On Demand Destination Files Skipped: 0
                  Invalid Footprint Files Skipped: 0
                          All Other Files Skipped: 0
                                   Blocks Scanned: 5921640
                Blocks Skipped Due to Small Files: 0
                Blocks Skipped Due to Large Files: 0
        Blocks Skipped Due to Fast Truncate Files: 0
   Blocks Skipped Due to Files in Snapshot Copies: 5921640
Blocks Skipped Due to Efficiency (by Block Count): 0
    Blocks Skipped Due to Efficiency (by Percent): 0
         Blocks Skipped Due to Incompatible Files: 0
             Blocks Skipped Due to Metadata Files: 0
              Blocks Skipped Due to Remote Caches: 0
         Blocks Skipped Due to Write Fenced Files: 0
Blocks Skipped Due to On Demand Destination Files: 0
    Blocks Skipped Due to Invalid Footprint Files: 0
          Blocks Skipped Due to All Other Reasons: 0
                   Last File Move Error Timestamp: -
                     Last File Move Error File ID: -
Last File Move Error Destination Constituent DSID: -
                        Last File Move Error Code: -

                                     Vserver Name: svm
                                      Volume Name: fg__0002
                                   Is Constituent: true
                                  Rebalance State: idle
                        Rebalance Notice Messages: -
                                       Total Size: 100GB
              Used Size for Calculating Imbalance: 762.6MB
              Volume Size Used by Snapshot Copies: 164KB
                     Constituent Target Used Size: 4.29GB
                                   Imbalance Size: -3.55GB
                             Imbalance Percentage: -82%
                                  Moved Data Size: 0B
         Maximum Constituent Imbalance Percentage: -
                             Rebalance Start Time: Sun Apr 07 07:16:47 2024
                              Rebalance Stop Time: -
                                Rebalance Runtime: 0h0m38s
                        Rebalance Maximum Runtime: 6h0m0s
      Maximum Imbalance Threshold per Constituent: 20%
      Minimum Imbalance Threshold per Constituent: 5%
    Maximum Concurrent File Moves per Constituent: 25
                                Minimum File Size: 100MB
           Exclude Files Stuck in Snapshot Copies: true
                  Most Recent File Move Timestamp: -
                               File Moves Started: 0
                                    Files Scanned: 0
                              Small Files Skipped: 0
                              Large Files Skipped: 0
                      Fast Truncate Files Skipped: 0
                 Files in Snapshot Copies Skipped: 0
 Files Skipped Due to Efficiency (by Block Count): 0
     Files Skipped Due to Efficiency (by Percent): 0
                       Incompatible Files Skipped: 0
                           Metadata Files Skipped: 0
                            Remote Caches Skipped: 0
                       Write Fenced Files Skipped: 0
              On Demand Destination Files Skipped: 0
                  Invalid Footprint Files Skipped: 0
                          All Other Files Skipped: 0
                                   Blocks Scanned: 0
                Blocks Skipped Due to Small Files: 0
                Blocks Skipped Due to Large Files: 0
        Blocks Skipped Due to Fast Truncate Files: 0
   Blocks Skipped Due to Files in Snapshot Copies: 0
Blocks Skipped Due to Efficiency (by Block Count): 0
    Blocks Skipped Due to Efficiency (by Percent): 0
         Blocks Skipped Due to Incompatible Files: 0
             Blocks Skipped Due to Metadata Files: 0
              Blocks Skipped Due to Remote Caches: 0
         Blocks Skipped Due to Write Fenced Files: 0
Blocks Skipped Due to On Demand Destination Files: 0
    Blocks Skipped Due to Invalid Footprint Files: 0
          Blocks Skipped Due to All Other Reasons: 0
                   Last File Move Error Timestamp: -
                     Last File Move Error File ID: -
Last File Move Error Destination Constituent DSID: -
                        Last File Move Error Code: -
3 entries were displayed.

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state       notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     rebalancing -       8.59GB             -           3.55GB         41%               -          Sun Apr 07 07:16:47 2024 0h4m53s -                                0                  55268640
svm     fg__0001
               scanning    -       7.84GB             4.29GB      3.55GB         82%               0B         Sun Apr 07 07:16:47 2024 0h4m53s -                                0                  55268640
svm     fg__0002
               idle        -       762.6MB            4.29GB      -3.55GB        -82%              0B         Sun Apr 07 07:16:47 2024 0h4m53s -                                0                  0
3 entries were displayed.

::*> volume rebalance file-move statistics show

Vserver: svm
Volume: fg
                                 Source                     Destination
Constituent                  Files         Bytes          Files         Bytes
-------------------    -----------   -----------    -----------   -----------
fg__0001
       In Progress:              0            0B              0            0B
         Completed:              0            0B              0            0B

fg__0002
       In Progress:              0            0B              0            0B
         Completed:              0            0B              0            0B

2 entries were displayed.
::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state       notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     rebalancing -       8.59GB             -           3.55GB         41%               -          Sun Apr 07 07:16:47 2024 6h0m0s  -                          0                  4214233800
svm     fg__0001
               scanning    -       7.84GB             4.29GB      3.55GB         82%               0B         Sun Apr 07 07:16:47 2024 6h0m0s  -                          0                  4214233800
svm     fg__0002
               idle        -       762.6MB            4.29GB      -3.55GB        -82%              0B         Sun Apr 07 07:16:47 2024 6h0m0s  -                          0                  0
3 entries were displayed.

::*> volume show -volume fg* -fields available, filesystem-size, total, used, percent-used, performance-tier-inactive-user-data, performance-tier-inactive-user-data-percent, size, dedupe-space-saved, dedupe-space-saved-percent, dedupe-space-shared,logical-used, logical-used-percent,logical-used-by-afs, logical-available, physical-used, physical-used-percent, tiering-policy
vserver volume size  available filesystem-size total used   percent-used dedupe-space-saved dedupe-space-saved-percent dedupe-space-shared physical-used physical-used-percent logical-used logical-used-percent logical-available logical-used-by-afs tiering-policy performance-tier-inactive-user-data performance-tier-inactive-user-data-percent
------- ------ ----- --------- --------------- ----- ------ ------------ ------------------ -------------------------- ------------------- ------------- --------------------- ------------ -------------------- ----------------- ------------------- -------------- ----------------------------------- -------------------------------------------
svm     fg     200GB 180.9GB   200GB           190GB 9.08GB 4%           0B                 0%                         0B                  9.08GB   5%                    9.08GB       5%                   -                 9.08GB              none           0B                                  0%
svm     fg__0001
               100GB 86.90GB   100GB           95GB  8.10GB 8%           0B                 0%                         0B                  8.10GB   8%                    8.10GB       9%                   -                 8.10GB              none           0B                                  0%
svm     fg__0002
               100GB 94.02GB   100GB           95GB  1005MB 1%           0B                 0%                         0B                  1005MB   1%                    1005MB       1%                   -                 1005MB              none           0B                                  0%
3 entries were displayed.

結局6時間待ってもリバランシングはされませんでした。

再リバランシング

偏りが足りないのか? と思い、もう少しテストファイルを追加します。

$ for i in {33..48}; do
    sudo dd if=/dev/urandom of=/mnt/fsxn/fg/random_pattern_binary_block_${i} bs=1M count=512
done

$ df -hT -t nfs4
Filesystem                                                                 Type  Size  Used Avail Use% Mounted on
svm-0365ba78d7ad91348.fs-009351b227391d1f1.fsx.us-east-1.amazonaws.com:/fg nfs4  190G   18G  173G  10% /mnt/fsxn/fg

テストファイル追加後のリバランシングの状態を確認します。

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state       notices                                                                              used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     not-running "The system scanned 64530 files, totalling 15.82TB. 100% of those files, totalling 100% of the size, have data in Snapshot copies. Files with data in Snapshot copies are not moved by default. To have the volume rebalancer move files with data in Snapshot copies, try the operation again using "-exclude-snapshots false"." 16.66GB            -           6.57GB         39%               -          Sun Apr 07 07:16:47 2024 6h0m0s  -                                0                  4245815880
svm     fg__0001
               not-running -                                                                              14.90GB            8.33GB      6.57GB         78%               0B         Sun Apr 07 07:16:47 2024 6h0m0s  -                                0                  4245815880
svm     fg__0002
               not-running -                                                                              1.76GB             8.33GB      -6.57GB        -78%              0B         Sun Apr 07 07:16:47 2024 6h0m0s  -                                0                  0
3 entries were displayed.

::*> volume show -volume fg* -fields available, filesystem-size, total, used, percent-used, performance-tier-inactive-user-data, performance-tier-inactive-user-data-percent, size, dedupe-space-saved, dedupe-space-saved-percent, dedupe-space-shared,logical-used, logical-used-percent,logical-used-by-afs, logical-available, physical-used, physical-used-percent, tiering-policy
vserver volume size  available filesystem-size total used    percent-used dedupe-space-saved dedupe-space-saved-percent dedupe-space-shared physical-used physical-used-percent logical-used logical-used-percent logical-available logical-used-by-afs tiering-policy performance-tier-inactive-user-data performance-tier-inactive-user-data-percent
------- ------ ----- --------- --------------- ----- ------- ------------ ------------------ -------------------------- ------------------- ------------- --------------------- ------------ -------------------- ----------------- ------------------- -------------- ----------------------------------- -------------------------------------------
svm     fg     200GB 172.7GB   200GB           190GB 17.25GB 9%           0B                 0%                         0B                  17.25GB       9%                    17.25GB      9%                   -                 17.25GB             none           0B                            0%
svm     fg__0001
               100GB 79.74GB   100GB           95GB  15.26GB 16%          0B                 0%                         0B                  15.26GB       15%                   15.26GB      16%                  -                 15.26GB             none           0B                            0%
svm     fg__0002
               100GB 93.00GB   100GB           95GB  2.00GB  2%           0B                 0%                         0B                  2.00GB        2%                    2.00GB       2%                   -                 2.00GB              none           0B                            0%
3 entries were displayed.

100%のデータがSnapshotにデータを保持していると記載されています。リバランシングの対象はSnapshotに現在存在しないファイルのみです。どうやらSnapshotを気づかないうちに取得していたみたいです。

実際に確認すると、確かにSnapshotが取得されていました。Snapshotポリシーをnoneに設定する前に取得されていたようです。

::*> snapshot show -volume fg*
                                                                 ---Blocks---
Vserver  Volume   Snapshot                                  Size Total% Used%
-------- -------- ------------------------------------- -------- ------ -----
svm      fg
                  hourly.2024-04-07_0705                  8.11GB     4%   33%
         fg__0001
                  hourly.2024-04-07_0705                   564KB     0%    0%
         fg__0002
                  hourly.2024-04-07_0705                   220KB     0%    0%
3 entries were displayed.

テストファイルを追加した後はSnapshotを取得していません。この状態であればリバランシングされるのではないでしょうか。

再度リバランシングをします。

:*> volume rebalance start -volume fg

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state       notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     rebalancing -       16.66GB            -           6.57GB         39%               -          Mon Apr 08 00:24:00 2024 0h0m5s  -                                0                  0
svm     fg__0001
               scanning    -       14.90GB            8.33GB      6.57GB         78%               0B         Mon Apr 08 00:24:00 2024 0h0m5s  -                                0                  0
svm     fg__0002
               scanning    -       1.76GB             8.33GB      -6.57GB        -78%              0B         Mon Apr 08 00:24:00 2024 0h0m5s  -                                0                  0
3 entries were displayed.

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state       notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     rebalancing -       16.66GB            -           6.57GB         39%               -          Mon Apr 08 00:24:00 2024 0h0m13s -                                0                  3816140
svm     fg__0001
               scanning    -       14.90GB            8.33GB      6.57GB         78%               0B         Mon Apr 08 00:24:00 2024 0h0m13s -                                0                  3816140
svm     fg__0002
               idle        -       1.76GB             8.33GB      -6.57GB        -78%              0B         Mon Apr 08 00:24:00 2024 0h0m13s -                                0                  0
3 entries were displayed.

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state       notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     rebalancing -       16.86GB            -           6.47GB         38%               -          Mon Apr 08 00:24:00 2024 0h0m19s Mon Apr 08 00:24:18 2024         3                  3816140
svm     fg__0001
               rebalancing-source
                           -       14.90GB            8.43GB      6.47GB         76%               0B         Mon Apr 08 00:24:00 2024 0h0m19s Mon Apr 08 00:24:18 2024         3                  3816140
svm     fg__0002
               rebalancing-dest
                           -       1.95GB             8.43GB      -6.47GB        -76%              0B         Mon Apr 08 00:24:00 2024 0h0m19s -                                0                  0
3 entries were displayed.

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state       notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     rebalancing -       19.17GB            -           5.32GB         27%               -          Mon Apr 08 00:24:00 2024 0h0m22s Mon Apr 08 00:24:21 2024         6                  3816140
svm     fg__0001
               rebalancing-source
                           -       14.90GB            9.58GB      5.32GB         55%               0B         Mon Apr 08 00:24:00 2024 0h0m22s Mon Apr 08 00:24:22 2024         7                  3816140
svm     fg__0002
               rebalancing-dest
                           -       4.27GB             9.58GB      -5.32GB        -55%              0B         Mon Apr 08 00:24:00 2024 0h0m22s -                                0                  0
3 entries were displayed.

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state       notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     rebalancing -       21.21GB            -           4.30GB         20%               -          Mon Apr 08 00:24:00 2024 0h0m35s Mon Apr 08 00:24:24 2024         9                  3816140
svm     fg__0001
               rebalancing-source
                           -       14.90GB            10.60GB     4.30GB         40%               0B         Mon Apr 08 00:24:00 2024 0h0m35s Mon Apr 08 00:24:24 2024         9                  3816140
svm     fg__0002
               rebalancing-dest
                           -       6.30GB             10.60GB     -4.30GB        -40%              0B         Mon Apr 08 00:24:00 2024 0h0m35s -                                0                  0
3 entries were displayed.

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state       notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     rebalancing -       16.92GB            -           2.14GB         12%               -          Mon Apr 08 00:24:00 2024 0h1m37s Mon Apr 08 00:24:24 2024         9                  3816140
svm     fg__0001
               idle        -       10.59GB            8.46GB      2.14GB         25%               -4.52GB    Mon Apr 08 00:24:00 2024 0h1m37s Mon Apr 08 00:24:24 2024         9                  3816140
svm     fg__0002
               idle        -       6.32GB             8.46GB      -2.14GB        -25%              4.52GB     Mon Apr 08 00:24:00 2024 0h1m37s -                                0                  0
3 entries were displayed.

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     idle  -       16.71GB            -           2.03GB         12%               -          Mon Apr 08 00:24:00 2024 0h1m58s Mon Apr 08 00:24:24 2024         9                  3816140
svm     fg__0001
               idle  -       10.38GB            8.35GB      2.03GB         24%               -4.52GB    Mon Apr 08 00:24:00 2024 0h1m58s Mon Apr 08 00:24:24 2024         9                  3816140
svm     fg__0002
               idle  -       6.32GB             8.35GB      -2.03GB        -24%              4.52GB     Mon Apr 08 00:24:00 2024 0h1m58s -                                0                  0
3 entries were displayed.

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state       notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     rebalancing -       17.21GB            -           1.78GB         10%               -          Mon Apr 08 00:24:00 2024 0h2m31s Mon Apr 08 00:26:31 2024         11                 3816140
svm     fg__0001
               rebalancing-source
                           -       10.38GB            8.60GB      1.78GB         20%               -4.52GB    Mon Apr 08 00:24:00 2024 0h2m31s Mon Apr 08 00:26:31 2024         11                 3816140
svm     fg__0002
               rebalancing-dest
                           -       6.82GB             8.60GB      -1.78GB        -20%              4.52GB     Mon Apr 08 00:24:00 2024 0h2m31s -                                0                  0
3 entries were displayed.

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state       notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     rebalancing -       17.21GB            -           1.78GB         10%               -          Mon Apr 08 00:24:00 2024 0h2m40s Mon Apr 08 00:26:32 2024         12                 3816140
svm     fg__0001
               rebalancing-source
                           -       10.38GB            8.60GB      1.78GB         20%               -4.52GB    Mon Apr 08 00:24:00 2024 0h2m40s Mon Apr 08 00:26:32 2024         12                 3816140
svm     fg__0002
               rebalancing-dest
                           -       6.82GB             8.60GB      -1.78GB        -20%              4.52GB     Mon Apr 08 00:24:00 2024 0h2m40s -                                0                  0
3 entries were displayed.

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state       notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     rebalancing -       17.36GB            -           858.6MB        4%                -          Mon Apr 08 00:24:00 2024 0h3m6s  Mon Apr 08 00:26:32 2024         12                 3816140
svm     fg__0001
               idle        -       9.52GB             8.68GB      858.6MB        9%                -6.02GB    Mon Apr 08 00:24:00 2024 0h3m6s  Mon Apr 08 00:26:32 2024         12                 3816140
svm     fg__0002
               idle        -       7.84GB             8.68GB      -858.6MB       -9%               6.02GB     Mon Apr 08 00:24:00 2024 0h3m6s  -                                0                  0
3 entries were displayed.

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     idle  -       17.36GB            -           858.6MB        4%                -          Mon Apr 08 00:24:00 2024 0h3m10s Mon Apr 08 00:26:32 2024         12                 3816140
svm     fg__0001
               idle  -       9.52GB             8.68GB      858.6MB        9%                -6.02GB    Mon Apr 08 00:24:00 2024 0h3m10s Mon Apr 08 00:26:32 2024         12                 3816140
svm     fg__0002
               idle  -       7.84GB             8.68GB      -858.6MB       -9%               6.02GB     Mon Apr 08 00:24:00 2024 0h3m10s -                                0                  0
3 entries were displayed.

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state notices used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----- ------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     idle  -       16.72GB            -           529.7MB        3%                -          Mon Apr 08 00:24:00 2024 0h5m18s Mon Apr 08 00:26:32 2024         12                 3816140
svm     fg__0001
               idle  -       8.88GB             8.36GB      529.7MB        6%                -6.02GB    Mon Apr 08 00:24:00 2024 0h5m18s Mon Apr 08 00:26:32 2024         12                 3816140
svm     fg__0002
               idle  -       7.84GB             8.36GB      -529.7MB       -6%               6.02GB     Mon Apr 08 00:24:00 2024 0h5m18s -                                0                  0
3 entries were displayed.

::*> volume rebalance file-move statistics show

Vserver: svm
Volume: fg
                                 Source                     Destination
Constituent                  Files         Bytes          Files         Bytes
-------------------    -----------   -----------    -----------   -----------
fg__0001
       In Progress:              0            0B              0            0B
         Completed:             12        6.02GB              0            0B

fg__0002
       In Progress:              0            0B              0            0B
         Completed:              0            0B             12        6.02GB

2 entries were displayed.

12個のファイルが3分弱でリバランシングされました。

Imbalance PercentageMaximum Imbalance Threshold per ConstituentMinimum Imbalance Threshold per Constituentの間なのでこれ以上待ってもリバランシングは進まなさそうです。停止させましょう。

::*> volume rebalance stop -volume fg

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state       notices                                                                         used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     not-running "The system scanned 44 files, totalling 14.56GB. 68% of those files, totalling 51% of the size, have data in Snapshot copies. Files with data in Snapshot copies are not moved by default. To have the volume rebalancer move files with data in Snapshot copies, try the operation again using "-exclude-snapshots false"." 16.72GB            -           529.7MB        3%                -          Mon Apr 08 00:24:00 2024 0h9m9s  Mon Apr 08 00:26:32 2024         12                 3816140
svm     fg__0001
               not-running -                                                                         8.88GB             8.36GB      529.7MB        6%                -6.02GB    Mon Apr 08 00:24:00 2024 0h9m9s  Mon Apr 08 00:26:32 2024         12                 3816140
svm     fg__0002
               not-running -                                                                         7.84GB             8.36GB      -529.7MB       -6%               6.02GB     Mon Apr 08 00:24:00 2024 0h9m9s  -                                0                  0
3 entries were displayed.

ファイル単位のリバランシング

リバランシングはファイル単位でも行えます。

8GiBのファイルを作成します。

$ sudo dd if=/dev/urandom of=/mnt/fsxn/fg/random_pattern_binary_block_49 bs=1M count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 58.1159 s, 148 MB/s

$ df -hT -t nfs4
Filesystem                                                                 Type  Size  Used Avail Use% Mounted on
svm-0365ba78d7ad91348.fs-009351b227391d1f1.fsx.us-east-1.amazonaws.com:/fg nfs4  190G   26G  165G  14% /mnt/fsxn/fg

fg__0002の使用量が増加しています。fg__0002に保存されたようですね。

::*> volume rebalance show -fields state, notices, used-for-imbalance, target-used, imbalance-size, imbalance-percent, data-moved, start-time, runtime, most-recent-file-move-start-time, file-moves-started, blocks-scanned
vserver volume state       notices                                                                         used-for-imbalance target-used imbalance-size imbalance-percent data-moved start-time               runtime most-recent-file-move-start-time file-moves-started blocks-scanned
------- ------ ----------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------ ----------- -------------- ----------------- ---------- ------------------------ ------- -------------------------------- ------------------ --------------
svm     fg     not-running "The system scanned 44 files, totalling 14.56GB. 68% of those files, totalling 51% of the size, have data in Snapshot copies. Files with data in Snapshot copies are not moved by default. To have the volume rebalancer move files with data in Snapshot copies, try the operation again using "-exclude-snapshots false"." 24.79GB            -           3.52GB         14%               -          Mon Apr 08 00:24:00 2024 0h9m9s  Mon Apr 08 00:26:32 2024         12                 3816140
svm     fg__0001
               not-running -                                                                         8.88GB             12.40GB     -3.52GB        -28%              -6.02GB    Mon Apr 08 00:24:00 2024 0h9m9s  Mon Apr 08 00:26:32 2024         12                 3816140
svm     fg__0002
               not-running -                                                                         15.92GB            12.40GB     3.52GB         28%               6.02GB     Mon Apr 08 00:24:00 2024 0h9m9s  -                                0                  0
3 entries were displayed.

先ほど作成したファイルをfg__0002からfg__0001にリバランシングします。

::*> volume rebalance file-move start -volume fg -path /random_pattern_binary_block_49 -destination-constituent fg__0001

File move started with operation-uuid b2f920e6-6e38-4916-aa8b-4491da7e5577.
Use the "volume rebalance file-move show" command to view the status of this operation.

::*> volume rebalance file-move statistics show

Vserver: svm
Volume: fg
                                 Source                     Destination
Constituent                  Files         Bytes          Files         Bytes
-------------------    -----------   -----------    -----------   -----------
fg__0001
       In Progress:              0            0B              1        8.03GB
         Completed:             12        6.02GB              0            0B

fg__0002
       In Progress:              1        8.03GB              0            0B
         Completed:              0            0B             12        6.02GB

2 entries were displayed.

::*> volume rebalance file-move show

Vserver: svm
Volume: fg
Path                       Source        Destination   Status          Progress
-------------------------- ------------- ------------  --------------- --------
/random_pattern_binary_block_49
                           fg__0002      fg__0001      Preparing       100%

::*> volume rebalance file-move show -instance

                        Vserver: svm
                    Volume Name: fg
 Destination Constituent Volume: fg__0001
                 Operation UUID: b2f920e6-6e38-4916-aa8b-4491da7e5577
   Destination Constituent Node: FsxId009351b227391d1f1-01
   Destination Constituent DSID: 1033
      Source Constituent Volume: fg__0002
          Source Operation UUID: b2f920e6-6e38-4916-aa8b-4491da7e5577
        Source Constituent Node: FsxId009351b227391d1f1-01
        Source Constituent DSID: 1034
               Operation Status: Data-Transfer
    Scanner Progress Percentage: 5%
            Last Failure Reason: -
                      File Path: /random_pattern_binary_block_49
   Destination Constituent MSID: 2163179387
             Destination Fileid: 143
         Destination Generation: 8112694
        Source Constituent MSID: 2163179388
                  Source Fileid: 510
              Source Generation: 8026273
Maximum Scanner Speed (per sec): -
                 Scanner Paused: false
                 Scanner Status: Data
               Scanner Progress: 371.0MB
                  Scanner Total: 8GB
          Data Scanner Priority: low
                   Elapsed Time: 0h0m17s
                   Cutover Time: 4s
             Is Snapshot Fenced: false
           Is Destination Ready: true
              Last Failure Time: -

::*> volume rebalance file-move show

Vserver: svm
Volume: fg
Path                       Source        Destination   Status          Progress
-------------------------- ------------- ------------  --------------- --------
/random_pattern_binary_block_49
                           fg__0002      fg__0001      Data-Transfer   17%

::*> volume rebalance file-move show

Vserver: svm
Volume: fg
Path                       Source        Destination   Status          Progress
-------------------------- ------------- ------------  --------------- --------
/random_pattern_binary_block_49
                           fg__0002      fg__0001      Data-Transfer   31%

::*> volume rebalance file-move show

Vserver: svm
Volume: fg
Path                       Source        Destination   Status          Progress
-------------------------- ------------- ------------  --------------- --------
/random_pattern_binary_block_49
                           fg__0002      fg__0001      Data-Transfer   55%

::*> volume rebalance file-move show

Vserver: svm
Volume: fg
Path                       Source        Destination   Status          Progress
-------------------------- ------------- ------------  --------------- --------
/random_pattern_binary_block_49
                           fg__0002      fg__0001      Data-Transfer   68%

::*> volume rebalance file-move show

Vserver: svm
Volume: fg
Path                       Source        Destination   Status          Progress
-------------------------- ------------- ------------  --------------- --------
/random_pattern_binary_block_49
                           fg__0002      fg__0001      Data-Transfer   100%

::*> volume rebalance file-move show -instance

                        Vserver: svm
                    Volume Name: fg
 Destination Constituent Volume: fg__0001
                 Operation UUID: b2f920e6-6e38-4916-aa8b-4491da7e5577
   Destination Constituent Node: FsxId009351b227391d1f1-01
   Destination Constituent DSID: 1033
      Source Constituent Volume: fg__0002
          Source Operation UUID: b2f920e6-6e38-4916-aa8b-4491da7e5577
        Source Constituent Node: FsxId009351b227391d1f1-01
        Source Constituent DSID: 1034
               Operation Status: Data-Transfer
    Scanner Progress Percentage: 100%
            Last Failure Reason: -
                      File Path: /random_pattern_binary_block_49
   Destination Constituent MSID: 2163179387
             Destination Fileid: 143
         Destination Generation: 8112694
        Source Constituent MSID: 2163179388
                  Source Fileid: 510
              Source Generation: 8026273
Maximum Scanner Speed (per sec): -
                 Scanner Paused: false
                 Scanner Status: Data
               Scanner Progress: 7.98GB
                  Scanner Total: 8GB
          Data Scanner Priority: low
                   Elapsed Time: 0h1m4s
                   Cutover Time: 4s
             Is Snapshot Fenced: false
           Is Destination Ready: true
              Last Failure Time: -

完了しました。簡単ですね。特大ファイルが特定できている場合は役立ちそうです。

FlexGroupのコンスティチュエント間のデータ使用量の偏りが原因でパフォーマンスに影響が出ている時に

FlexGroupのリバランシング機能を試してみました。

FlexGroupのコンスティチュエント間のデータ使用量の偏りが原因でパフォーマンスに影響が出ている時に実行しましょう。

スケジューリング機能もありますが、こちらは定期実行するものではなく指定した時間に一回実行するものです。普段の運用に組み込みものというよりかは、必要性に応じてピンポイントで実行するものかと思います。特にSnapshotと相性はよろしくないので注意が必要です。

この記事が誰かの助けになれば幸いです。

以上、AWS事業本部 コンサルティング部の のんピ(@non____97)でした!

Share this article

facebook logohatena logotwitter logo

© Classmethod, Inc. All rights reserved.