====== I received a warning message about the Kohn Sham matrix not 100% occupied when doing Hartree-Fock or hybrid calculations, what is going on, and what should I do?  ======

<note important> **Since version 2023.2** The occupancy of the Kohn-Sham matrix depends on whether atomic pairs are neighbours. Two atoms are considered neighbours if their basis functions (Gaussians) are overlapping, which is controlled through the ''EPS_PFG_ORB'' keyword. In large cells, the KS matrix can be sparse because atomic pairs can be further apart. Setting the ''MIN_PAIR_LIST_RADIUS'' to -1 makes sure all atomic pairs within the unit cell are considered neighbours, therefore ensuring a full KS matrix. This is considerably more efficient than reducing ''EPS_PFG_ORB'' to unreasonably small numbers. </note> 

When you do a calculation that involves the calculation of the Hartree-Fock exchange (e.g. calculations with hybrid functionals, including using the ADMM method), you may occasionally encounter the following warning message (more often if ADMM is used):

<code>
The Kohn Sham matrix is not 100% occupied. 
This may result in incorrect Hartree-Fock results. 
Setting MIN_PAIR_LIST_RADIUS to -1 in the QS section 
ensures a fully occupied KS matrix. 
</code>

===== Why is this warning message produced =====

As you may already know, CP2K extensively uses the sparsity of matrices to increase efficiency of calculations. In a normal DFT (LSDA/GGA) calculation, the sparsity of the Kohn Sham (KS) matrix is the same as the overlap matrix, and therefore only parts of the density matrix that are non-zero in the corresponding parts of the overlap matrix are needed for the calculation.  In other words, the density matrix and Kohn Sham matrix are stored using the same sparse pattern as the overlap matrix.  This makes the calculation efficient by avoiding unnecessary multiplication with zero matrices blocks, and the accuracy of the calculation as well as the electron density function are unaffected.

For Hartree Fock exchange calculations, however, this is no longer the case. The sparsity of the KS matrix is no longer that of the overlap matrix. For the HF exchange term, we need all non-zero blocks of the density matrix and not only those that matched with the overlap.  This contributes to more non-zero elements in the KS matrix.

Due to anything outside the sparsity pattern of the KS matrix---which is based on that of the overlap matrix---is not stored, the Hartree Fock code performs a screening of the HF exchange terms based on the sparsity pattern of KS, and has to //assume// that if a block is not present in the KS matrix then the HF exchange contribution to the block is also zero.  This is not always true, depending on the property of the density matrix.  Therefore, when the code detects that non-zero contributions of the HF exchange terms are being screened out, it produces the above warning message.

===== What is the EPS_PGF_ORB parameter =====

''EPS_PGF_ORB'' controls the sparse pattern of the overlap matrix.  Any element in the overlap matrix whose absolute value is smaller than ''EPS_PGF_ORB'' is treated as zero.  And if a whole atomic block of the overlap matrix is less than ''EPS_PGF_ORB'' then the block will not be included in the sparse matrix.

The default value for ''EPS_PFG_ORB'' is set as the SQRT of the value of ''EPS_DEFAULT''.

===== What is the EPS_FILTER_MATRIX =====

On top of KS matrix having sparse pattern of the overlap matrix, if ''EPS_FILTER_MATRIX'' is not zero, then a filtering process is performed so that any atomic blocks in the KS matrix having all elements with their absolute values less than ''EPS_FILTER_MATRIX'' is then removed from the KS matrix.

The default value of ''EPS_FILTER_MATRIX'' is zero, so this extra filter step is not used unless specified by the user.

===== What should I do when I see this warning message =====

If ''EPS_FILTER_MATRIX'' is already zero, then setting ''EPS_PGF_ORB'' to a smaller value will eventually remove the warning message.  If ''EPS_FILTER_MATRIX'' is not zero, then the first course of action is to reduce it.

As usual with screening, the typical error you make is on the order of EPS, except the case when the calculation becomes unstable, which yields results that can be essentially unrelated to EPS (e.g. wrong by O(1)). In practice, despite of the warning, if a calculation is stable it should be accurate up to the value of ''EPS_PGF_ORB'' or ''EPS_FILTER_MATRIX''---whichever is larger. You can always check by running a single point calculation with a smaller EPS value, and see if the difference in the total energy is in the order of magnitude as the larger of the EPS value you have tested.

The warning message is therefore more for the case when the calculation becomes unstable due to the forced screening of the HF exchange. So if your calculation is unstable, and if you see this warning message, then ''EPS_PGF_ORB'' and ''EPS_FILTER_MATRIX'' are the first places to look.

Note that by decreasing ''EPS_PGF_ORB'', you will be making the overlap matrix more dense, and correspondingly also the density and KS matrices stored in CP2K.  This will increase your computational cost.  However, for calculations with hybrid functionals the cost of HF exchange term usually dominates and hence the associated cost increase due to more dense overlap, KS and density matrices may not be significant in comparison.