Can someone tell me how to hook up LEDs to a fuse box that comes on when the keys are in the ignition?

Wire them into your cigarette lighter. It's switched and fused

so you are saying use the wires and not directly into the fuse box?

If you can follow a wiring diagram, then this would be TONS of help.
http://www.xbsite.com/forum/wiring-diagram-t-868.html

Correct, you can just tap into the wires going to the cig lighter. Very easy and works just as good as if it were coming directly from the fuse box.

should I worry at all about pulling too much power from that one source? I was looking to hook up my under dash lights to that source. Can I also hook up the LEDs to that and not blow the fuse?

Shouldn't draw too much to hook that up.
I'm running a 400 watt power inverter to run my laptop in the car, plus my other electronics at the same time. Been working fine.