Add the crushed granite on top of the pea gravel and tamp it in. This should fill in the gaps between the stones.
Depends what is under the pea stone. Pea stone isn't the worst base. Straight sand is often used as a base for patio blocks. Have it tamped down very well, and as long as what is under it is trustworthy you should be fine. Adding sand and tamping so you have a smooth layer may not be bad idea either. If you don't think the base is trustworthy put down some chicken wire and cover it with a 1/2 inch of cheap mortar as you lay the patio stone. Then use polymorphic sand to fill the gaps between blocks.
I bought pea stone for a base to do my patio of advice from someone. Apparently, as I read up on it now; not the best idea. I spread them already but I need more for a base anyway. Is there something I can buy now to save it to have a good solid base that the patio blocks wont shift?