I'm not familiar with the modalbox library, but I can take a guess at what it's doing. When the browser requests an HTML page it does a bunch of stuff to evaluate JavaScript and stylesheets included in the response. However, modalbox is probably just intercepting that response and stuffing the result into the DOM. Unless it's using an iframe the modal dialog isn't a separate page, it just appears that way due to CSS trickery. If you load the JS files you need on the main page they'll be available in the modal box. The best way to make sure you load the JS everywhere you need it is to put it in your layout.

If the library does use an iframe for the modal box then the browser should parse the JavaScript you send in the page response, in which case someone who knows more about iframes will hopefully speak up.