Why is the HOMO-LUMO gap not a good guess of the excitation energy? The HOMO energy is a good guess for electron detachment and (under certain idealized conditions) the LUMO for electron attachment. Excitation can formally be considered as an electron detachment with a subsequent attachment, which makes a consideration of the HOMO-LUMO gap sound reasonable. But the difference is that the LUMO energy is initially determined under the condition that the HOMO is occupied. To get the correct first order excitation energy one has to subtract the "electron-hole interaction" or the "exciton binding energy" (from last post).
The Hartree-Fock energy in terms of the occupied MOs (indexed i,j) of the neutral system is given by
If we remove an electron out of orbital k (for example the HOMO but this holds for any occupied orbital), the new total energy becomes
The detachment energy (or ionization potential) is simply defined as the difference
And considering the permutation symmetry of two-electron integrals this reduces to
which is nothing but the negative orbital energy of orbital k with respect to the original n-electron system. This is just Koopmans' theorem.
What happens if we take away an electron from the occupied orbital k and put it into the virtual orbital a? Then the energy of the resulting Slater determinant is given by
Subtracting the ground state Hartree-Fock energy from this (and considering permutation symmetry)
Part of this is of course just εk. But the other half is not exactly εa, which also includes the interaction with orbital k
However, we can still write the first order expression for the excitation energy in the following way
First, there is the orbital energy gap, as we expected. But second also Coulomb and exchange integrals between the occupied and virtual orbitals have to be considered, yielding a term, which can be identified with the exciton binding energy from last post. The first part can be identified with an attractive Coulomb interaction between the "electron" and the "hole", the second one with an exchange interaction, which is present for singlet excited states (or more generally if the ground and excited states are of the same multiplicity).
Since the Coulomb integral is larger than the exchange integral (which I believe it is always) the first order correction term is negative and the excitation energy is lower than the gap between the corresponding orbitals. The first order energy (i.e. CIS) is usually still too high because it neglects orbital relaxation in the excited state.
For DFT the story is different because of self-interaction and these rigorous equalities do not hold. On the one hand this is part of the reason why TDDFT is so unexpectedly good in many cases. On the other hand the failure to rigorously include the electron-hole interaction is the reason why TDDFT fails so badly for charge transfer states and does not retain the correct 1/r asymptotics [see e.g. this Ref.].
Continue at: HOMO-LUMO gaps and spin eigenfunctions