Yes, you can delete a task immediately by tapping or clicking on the "..." button in the top right corner of an open task, then on "Delete Task".

Please Note: Deleting a task is a permanent action. You cannot undo this action. If you think you may like to reuse this task in the future, please archive it instead.