Need advice - planting flower seeds in a high weed pressure garden plot.

Hi! I’m taking on a plot in a local community garden, to grow cut flowers, that has been vacant for a long time and is covered in weeds. The weed pressure is bonkers there. I’m hoping to get access to the plot very soon so that I can tarp the area but the reality is that I may not have enough time before my last frost to let it sit long enough. I will be using woven landscape fabric with holes burned in for each plant and will certainly weed around all my plugs. But, I’m worried about the seeds that I’m going to direct sow. My concern is that I will struggle to tell the difference between my germinating seed and the germinating weeds that will likely pop up in the fabric hole. My idea is to dig out a bit of the existing soil in each hole, refill with purchased garden soil and plant my seed in that to eliminate weed seed lurking directly around my planted seeds. That way, hopefully, I can tell what is what when the weeds ultimately start growing around my established seedlings. Is plain ol’ garden soil fine to use here? Can I use straight compost? A mix? Top soil? I welcome any and all advice and tips! Thank you!