I think the screams would be muffled but still audible. Until the bag is closed, the extradimensional space inside is connected to the outside world. However, since sound can only escape from the opening and not the sides of the bag, I would rule that it is much quieter, granting disadvantage on checks to hear the scream.

You could just kill the guard by leaving them in the bag

I would immediately shift your alignment to chaotic evil for it because you’re literally making them suffocate to death over a period of about ten minutes, but it’s still RAW.