• that'll be be putting extra stress on the forks too

    That'll be (more or less) the only place it will be putting any stress on the forks.
    The weight of the rack and luggage will sit on the axle and go through the wheel.

About