I’m a full-time contractor. One of my clients is based in CA but also has an office in my state. Some random thoughts:

• Dhork’s post looks accurate to me regarding state income tax stuff. As a contractor, I haven’t done anything at all in CA - I just get paid by my client and count it as normal revenue that I earn where I live. If you go the contract route, I expect it to be similarly simple for you.
• Tax-wise, a person who makes money from work other than employment is by definition a sole proprietor. There is nothing to set up, it’s just the word used for that the type of entity that earns contracting income that isn’t set up as a corporation or LLC. Even if you made an LLC, unless you elect s corp or c corp status, that LLC is a “disregarded entity” as far as the IRS is concerned and tax treatment would be the same as for a sole proprietor.
• people get worked up about self-employment taxes, but in my case I actually come out ahead because I can dump way more into an individual 401k than I ever could as an employee (employer contribution can be around 18.5% of your profit as a sole prop, on top of the employee deferral). I also run as an s-corp and my state has a PTE tax, both of which let me avoid a bit of federal taxes. But s corp is only relevant if you make like $80k+ from self employment and don’t have much other w2 income.
• submitting bids is a function of your relationship with the client and the norms in your industry. I have two clients where I just have an open-ended contracting agreement (both clients had their own standard agreements and I negotiated small changes in one case) and I bill for however many hours i worked every month on projects that are constantly evolving. My standard though is to submit time-and-materials bids for fixed-scope projects. I keep an eye open for fixed-bid opportunities too but I haven’t submitted any fixed bids yet.
• contracting is not that big of an administrative hassle so don’t let that deter you. But do estimate the tax impact and do either make estimated payments or adjust your withholdings at your w2 job to cover them (this one is easier IMO).
• contractors get a higher hourly rate than employees. Don’t undervalue yourself. If you don’t have a good benchmark, do some research. I am in engineering services where big companies regularly charge $170-200/hr for my level - I charge a bit below that normally, with a further discount on my long-term open-ended contracts. But the most I’ve ever made as a w2 was about half that amount, after taking all the tax and benefits stuff into account. I’ve never fully understood why clients are willing to pay that much, but they are.

Advice you get on the Internet is worth what you pay for it. So what you should really do is talk to a local CPA. Everything I say below might be bullshit.

But, in general, you owe income taxes to the state where you live, not the state where your company is based. Two exceptions I know of are NY, which will insist on collecting non-resident income taxes from remote employees of NY companies, if their work duties do not require them to be there - it is confusingly called the “convenience of the employer” rule. The other exception is CA, which may require a remote employee to pay CA state taxes for work they do for a CA company while physically in CA. (So fully remote employees would be OK, but you might have to keep track of all the time you are present and working in CA).

However, this just covers who you owe taxes to. Smaller companies may not have the infrastructure to withhold for multiple states, and may be forced to withhold taxes for their state even though you technically don’t owe that state any tax. Which may force you to pay estimated taxes direct to the state you live in throughout the year (effectively double-taxing you), but then you should be able to file a non-resident return in your companies’ state to get a full refund.

My only experience with this was when I worked for a huge CA company with offices all over. They had an office in my state, so they were able to count me as a local employee there, even though I never worked in that office. They withheld the proper taxes and everything went smoothly. (Except for the fact that they never got my withholding right for my state, and withheld way too much…)