Then only the component of the force that's parallel to the displacement is used to calculate the work. The component of force that's perpendicular to the displacement doesn't move through any distance at all, so its contribution to the total work is zero.