We need to look at this from investment point of view. Treat paying down a loan like a bond investment and compare to other options.

Advantages of early pay-off

• One less bill to worry about. Peace of mind
• Simplified life. No need to budget for mortgage payment
• Guaranteed, zero-risk return for bond equivalent to mortgage interest rate.

Disadvantages of early pay-off

• If you are taking standard deduction, then interest rate is the actual rate of your mortgage. No tax benefit for interest deduction.
• Current rates are low. If you can make after-tax returns that are better than mortgage rate, you will come out ahead.
• Current rates are historically low, they may increase in future. It is better to have fixed rate loan for long term, when you factor in inflation.
• Liquidity issue. Money is stuck in home equity. Instead of payoff, you can buy bonds and then sell them if immediate cash is needed.
• Once you have paid it, you cannot get it back without taking out new loan or HELOC which will be at higher rate than first mortgage.