Grease a 9-inch square pan.

Beat sugar into goat milk in a large, heavy pot; bring to a rapid boil, stirring continually to prevent burning, and cook, stirring constantly, until dark and thick, 30 to 45 minutes. An instant-read thermometer inserted into the center of the mixture should read at least 245 degrees F (118 degrees C).

Stir peanut butter, butter, and vanilla extract into the milk mixture immediately upon reaching temperature. Continue to stir until the peanut butter and butter melt completely, the color is uniform, and the mixture appears smooth and glossy; spread into the prepared pan. Cool fudge to room temperature before cutting.