I don't know if there is a formula, but I'd want the screws to be in the brick at least as deep as the thickness of trellis they're going through. So, (trellis thickness) * 2
As for thickness of screw, probably depends on how many you will use. From your description, doesn't sound like they will be under much shear stress so more likely to be pulled out than sheared off. I would go for more of them but not too beefy.
I don't know if there is a formula, but I'd want the screws to be in the brick at least as deep as the thickness of trellis they're going through. So, (trellis thickness) * 2
As for thickness of screw, probably depends on how many you will use. From your description, doesn't sound like they will be under much shear stress so more likely to be pulled out than sheared off. I would go for more of them but not too beefy.