If you are thinking of bringing young kids to the Summerfest grounds, I would suggest making the trip as the gates open. There are dedicated areas of the grounds, as well as activities created with families in mind.
Included are a children's theater, playground area, Children's Hospital helmet program, Target Story Time, puppets and so on. For the older kids, there is a sports zone which will have visits from various athletes throughout the week.
Summerfest runs along Milwaukee's Lake Michigan, but that doesn't mean it doesn't get super hot at times. Be sure to bring sunscreen, hats and some water to keep you and the kids cool. And kudos to you for even considering taking on such an activity! Good luck with the stroller!