It isn't what you asked, but on islands where they often import diesel for power generation, solar is probably cheaper across the board. Even if it only offsets fuel and not the capital cost of a diesel plant, imported fuel is stupid expensive.
The arguments that solar is cheaper than natural gas (or anything else) seem to center around retail costs of energy. So an argument might be that a homeowner can install rooftop solar for a cost of $4/watt. With a broad assumption of a thirty year lifetime and average production of 15%, and ignoring the investment cost, they can produce energy for about $0.10/kWh, a favorable price in most of the US when we're at a customer meter.
Where solar falls short is in broader system planning. There are minimum capacity requirements. Utilities, whether investor owned or non-profit municipals, must be able to meet load demands. Solar has limited ability to meet peak load demands, so a natural gas plant is needed anyway.
Since the power into the grid has to equal the power out, something controllable is necessary.
Again that falls to natural gas.
When we get up onto the wholesale side of things, typical energy prices are in the mid-$20/MWh. Xcel Energy recently finished a 100 MW solar plant in Minnesota at a cost of $180M. Making the same assumptions as before, we have a cost of $45/MWh ($0.045/kWh). On the wholesale side it's less competitive, though its immunity to fuel cost fluctuations are a pro. But again it lacks capacity and controllability.
So my take is one can make arguments based on fact that will show solar is cheaper than natural gas, but those arguments have a limited scope and are deliberately structured to give the desired result. That doesn't mean solar is bad; its cost reductions have been impressive. But there are challenges to continued growth.
On the topic of cost reductions, much of the savings has been in panel costs. It seems to be to the point where labor is the biggest cost. That explains part of the gap in cost between a utility scale plant like Xcel's (labor efficiencies) and rooftop solar.