Hi, Jetpack contact forms do not use AJAX for submission. Instead they use the old-fashioned way of POSTing directly to the page. This means the complete page is reloaded on form submission and it's not FancyBox that is closing but the complete web page is refreshed and opens (as always) without a FancyBox lightbox.
There is however, a way to make FancyBox open on page load and with the exact combination of triggers and ID's, it might be possible to launch FancyBox right after a Jetpack form has been submitted, displaying the submission result/feedback.
If you can share a link to your site, I might be able to give you a code snippet to try.