My view is that there is no harm in allowing them to conduct an inspection.  If the inspection confirms that all is well, there's no problem.  If the inspection shows that there is something wrong, you can discuss it with them.  But if you refuse to let them inspect they will only conclude that there's a problem anyway.

If there is a problem with the property it may be too late for the purchasers to cancel.

Of course, you should discuss the matter with your solicitor and follow his/her advice so that it doesn't appear that you are extending the purchasers' right to cancel.