The hard costs of starting an organic garden vary based on the scope and size of the garden, as well as availability and price of the materials in your area that you need, like compost and soil. But the consensus is that growing your own food is always cheaper than buying it from a grocery store. To give you an example, the Web site Get Rich Slowly profiled one couple who harvested their own vegetables. They spent $318.43 on their garden and harvested $606.97 worth of produce. They spent 60 hours working in their garden that summer, so you'll be putting in more labor than you would on your weekly shopping trip. But in terms of finances, that's a huge savings.
The cheapest way is to start from seeds. There are many great online seed companies that offer more varieties of every vegetable than you can possibly imagine, so you can pick a vegetable based on its ideal growing conditions and the way it tastes when it matures. Seeds can be tough for beginners, though. Ideally, you'd get them started before it's time to plant them outside, but this can require an investment in indoor growing equipment. You can sow many seeds directly into the soil once it's warm enough, but weeds also love organic soil and often it's tough to tell the difference between a sprouting seed and a growing weed. But we do have to say that watching a plant sprout from something that is microscopic in the palm of your hand makes you feel like an excited kindergartener, so if you're up for the challenge, it's worth it.
If that seems a little too daunting to try first, then buy a small plant that someone else started from seed and plant it right at the beginning of the season. They cost more than seeds, but still less than produce from the store, and they're easier to care for than baby plants.