Tonight I went out to the shop to oil and tap the shaft again.
When I tapped the end it moved.
Now all I have to do is file off the setscrew marks and a little burr on the end and it will come right out.
Could have been the heat I put on the nut. Could have been the heat on the shaft. Could have been the Kroil. (I love the stuff.)
Bet it was the patience.
Thanks for all the ideas guys. If the shaft hadn't slipped loose tonight it was off with the end and drill it out this weekend.
Now where did I put the wood for the frame?