Sampling Validation using MC¶
We can also validate our SBI model by drawing samples from the posterior using MCMC methods. This allows us to assess how well our model captures the underlying distribution of parameters given the observed data.
We can do this using the SBI_Fitter.fit_observation_using_sampler method, which lets us choose from a few nested sampling or MCMC samplers. Here, we’ll use the dynesty sampler to draw samples from the posterior.
First we’ll load a trained model and choose a mock observation to fit.
[1]:
from synference import SBI_Fitter, load_unc_model_from_hdf5, test_data_dir
library_path = (
f"{test_data_dir}/grid_BPASS_Chab_DenseBasis_SFH_0.01_z_14_logN_2.7_Calzetti_v3_multinode.hdf5" # noqa: E501
)
fitter = SBI_Fitter.load_saved_model(
model_file=f"{test_data_dir}", library_path=library_path, device="cpu"
)
nm_path = f"{test_data_dir}/BPASS_DenseBasis_v4_final_nsf_0_params_empirical_noise_models.h5"
noise_models = load_unc_model_from_hdf5(nm_path)
fitter.feature_array_flags["empirical_noise_models"] = noise_models
/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
2026-06-24 09:51:00,919 | synference | INFO | Loaded model from /home/runner/.local/share/Synthesizer/data/synference/BPASS_DenseBasis_v4_final_nsf_0_posterior.pkl.
2026-06-24 09:51:00,920 | synference | INFO | Device: cpu
2026-06-24 09:51:00,968 | synference | WARNING | IndexError when trying to set train/test arrays.
First we can recreate the simulator from the information stored in the library. We will use the simulator in the log-likelihood calculation during sampling.
[2]:
fitter.recreate_simulator_from_library(
override_library_path=library_path, override_grid_path="test_grid.hdf5"
);
2026-06-24 09:51:01,011 | synference | INFO | Overriding internal library name from provided file path.
2026-06-24 09:51:01,304 | synference | WARNING | Failed to load cosmology from HDF5. Using Planck18 instead.
---------------------------------------------------------------------------
TimeoutError Traceback (most recent call last)
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:1348, in AbstractHTTPHandler.do_open(self, http_class, req, **http_conn_args)
1347 try:
-> 1348 h.request(req.get_method(), req.selector, req.data, headers,
1349 encode_chunked=req.has_header('Transfer-encoding'))
1350 except OSError as err: # timeout error
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/http/client.py:1303, in HTTPConnection.request(self, method, url, body, headers, encode_chunked)
1302 """Send a complete request to the server."""
-> 1303 self._send_request(method, url, body, headers, encode_chunked)
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/http/client.py:1349, in HTTPConnection._send_request(self, method, url, body, headers, encode_chunked)
1348 body = _encode(body, 'body')
-> 1349 self.endheaders(body, encode_chunked=encode_chunked)
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/http/client.py:1298, in HTTPConnection.endheaders(self, message_body, encode_chunked)
1297 raise CannotSendHeader()
-> 1298 self._send_output(message_body, encode_chunked=encode_chunked)
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/http/client.py:1058, in HTTPConnection._send_output(self, message_body, encode_chunked)
1057 del self._buffer[:]
-> 1058 self.send(msg)
1060 if message_body is not None:
1061
1062 # create a consistent interface to message_body
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/http/client.py:996, in HTTPConnection.send(self, data)
995 if self.auto_open:
--> 996 self.connect()
997 else:
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/http/client.py:962, in HTTPConnection.connect(self)
961 sys.audit("http.client.connect", self, self.host, self.port)
--> 962 self.sock = self._create_connection(
963 (self.host,self.port), self.timeout, self.source_address)
964 # Might fail in OSs that don't implement TCP_NODELAY
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/socket.py:857, in create_connection(address, timeout, source_address)
856 try:
--> 857 raise err
858 finally:
859 # Break explicitly a reference cycle
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/socket.py:845, in create_connection(address, timeout, source_address)
844 sock.bind(source_address)
--> 845 sock.connect(sa)
846 # Break explicitly a reference cycle
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
URLError Traceback (most recent call last)
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/synthesizer/instruments/filters.py:2041, in Filter._make_svo_filter(self)
2040 try:
-> 2041 with urllib.request.urlopen(self.svo_url) as f:
2042 # Get the root of the XML tree
2043 root = ElementTree.parse(f).getroot()
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:216, in urlopen(url, data, timeout, cafile, capath, cadefault, context)
215 opener = _opener
--> 216 return opener.open(url, data, timeout)
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:519, in OpenerDirector.open(self, fullurl, data, timeout)
518 sys.audit('urllib.Request', req.full_url, req.data, req.headers, req.get_method())
--> 519 response = self._open(req, data)
521 # post-process response
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:536, in OpenerDirector._open(self, req, data)
535 protocol = req.type
--> 536 result = self._call_chain(self.handle_open, protocol, protocol +
537 '_open', req)
538 if result:
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:496, in OpenerDirector._call_chain(self, chain, kind, meth_name, *args)
495 func = getattr(handler, meth_name)
--> 496 result = func(*args)
497 if result is not None:
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:1377, in HTTPHandler.http_open(self, req)
1376 def http_open(self, req):
-> 1377 return self.do_open(http.client.HTTPConnection, req)
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/urllib/request.py:1351, in AbstractHTTPHandler.do_open(self, http_class, req, **http_conn_args)
1350 except OSError as err: # timeout error
-> 1351 raise URLError(err)
1352 r = h.getresponse()
URLError: <urlopen error [Errno 110] Connection timed out>
During handling of the above exception, another exception occurred:
SVOInaccessible Traceback (most recent call last)
Cell In[2], line 1
----> 1 fitter.recreate_simulator_from_library(
2 override_library_path=library_path, override_grid_path="test_grid.hdf5"
3 );
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/synference/sbi_runner.py:5652, in SBI_Fitter.recreate_simulator_from_library(self, set_self, overwrite, override_library_path, override_grid_path, **kwargs)
5649 default_kwargs.update(kwargs)
5651 try:
-> 5652 simulator = GalaxySimulator.from_library(
5653 library_path, override_synthesizer_grid_dir=override_grid_path, **default_kwargs
5654 )
5655 except ValueError as e:
5656 logger.error(
5657 "Could not recreate simulator from grid. This model"
5658 " may not be compatible. A GalaxySimulator object can"
5659 " be provided manually to recover the SED."
5660 )
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/synference/library.py:5562, in GalaxySimulator.from_library(cls, library_path, override_synthesizer_grid_dir, override_emission_model, **kwargs)
5547 dict_create = dict(
5548 sfh_model=sfh_model,
5549 zdist_model=zdist_model,
(...)
5559 fixed_params=fixed_params,
5560 )
5561 dict_create.update(kwargs)
-> 5562 return cls(**dict_create)
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/synference/library.py:5108, in GalaxySimulator.__init__(self, sfh_model, zdist_model, grid, instrument, emission_model, emission_model_key, emitter_params, cosmo, param_order, param_units, param_transforms, out_flux_unit, required_keys, extra_functions, normalize_method, output_type, include_phot_errors, depths, depth_sigma, noise_models, fixed_params, photometry_to_remove, ignore_params, ignore_scatter, return_type, device)
5105 self.reported_unused = False
5107 if len(photometry_to_remove) > 0:
-> 5108 self.update_photo_filters(
5109 photometry_to_remove=photometry_to_remove, photometry_to_add=None
5110 )
5112 if noise_models is not None:
5113 assert isinstance(noise_models, dict), (
5114 f"Noise models must be a dictionary. Got {type(noise_models)} instead."
5115 )
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/synference/library.py:5224, in GalaxySimulator.update_photo_filters(self, photometry_to_remove, photometry_to_add)
5221 if filter_code not in new_filters:
5222 new_filters.append(filter_code)
-> 5224 self.instrument.filters = FilterCollection(filter_codes=new_filters)
5225 logger.info(f"Updated filters: {self.instrument.filters.filter_codes}")
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/synthesizer/utils/operation_timers.py:98, in timed.<locals>.decorator.<locals>.wrapped(*args, **kwargs)
94 tic(timer_name)
95 try:
96 # Return the wrapped function result unchanged so the decorator
97 # is transparent aside from its timing side effect.
---> 98 return func(*args, **kwargs)
99 finally:
100 # Always stop the timer, even if the wrapped function raises,
101 # so the timing stack remains balanced.
102 toc(timer_name)
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/synthesizer/instruments/filters.py:297, in FilterCollection.__init__(self, filter_codes, tophat_dict, generic_dict, filters, path, new_lam, fill_gaps, verbose)
295 # Let's make the filters
296 if filter_codes is not None:
--> 297 self._include_svo_filters(filter_codes)
298 if tophat_dict is not None:
299 self._include_top_hat_filters(tophat_dict)
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/synthesizer/instruments/filters.py:474, in FilterCollection._include_svo_filters(self, filter_codes)
471 # Loop over the given filter codes
472 for f in filter_codes:
473 # Get filter from SVO
--> 474 _filter = Filter(f, new_lam=self.lam)
476 # Store the filter and its code
477 self.filters[_filter.filter_code] = _filter
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/synthesizer/units.py:909, in accepts.<locals>.check_accepts.<locals>.wrapped(*args, **kwargs)
906 finally:
907 toc(f"accepts({func.__qualname__})")
--> 909 return func(*bound.args, **bound.kwargs)
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/synthesizer/utils/operation_timers.py:98, in timed.<locals>.decorator.<locals>.wrapped(*args, **kwargs)
94 tic(timer_name)
95 try:
96 # Return the wrapped function result unchanged so the decorator
97 # is transparent aside from its timing side effect.
---> 98 return func(*args, **kwargs)
99 finally:
100 # Always stop the timer, even if the wrapped function raises,
101 # so the timing stack remains balanced.
102 toc(timer_name)
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/synthesizer/instruments/filters.py:1770, in Filter.__init__(self, filter_code, transmission, lam_min, lam_max, lam_eff, lam_fwhm, new_lam, hdf)
1768 # Is this an SVO filter?
1769 elif "/" in filter_code and "." in filter_code:
-> 1770 self._make_svo_filter()
1772 # Otherwise we haven't got a valid combination of inputs.
1773 else:
1774 raise exceptions.InconsistentArguments(
1775 "Invalid combination of filter inputs. \n For a generic "
1776 "filter provide a transmission and wavelength array. "
(...)
1781 "wavelength or an effective wavelength and FWHM."
1782 )
File /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/synthesizer/instruments/filters.py:2052, in Filter._make_svo_filter(self)
2049 data = root.find(".//TABLEDATA")
2051 except URLError:
-> 2052 raise exceptions.SVOInaccessible(
2053 (
2054 f"The SVO Database at {self.svo_url} "
2055 "is not responding. Is it down?"
2056 )
2057 )
2059 # Throw an error if we didn't find the filter.
2060 if field is None:
SVOInaccessible: The SVO Database at http://svo2.cab.inta-csic.es/theory/fps/fps.php?ID=HST/ACS_WFC.F435W is not responding. Is it down?
Then we can proceed to fit the observation using the sampler.
The code will attempt to remove parameters which don’t affect the fit (e.g. supplemental parameters) before fitting, but you can also specify to remove specific parameters using the remove_params argument.
Let’s choose an observation from the validation set and fit it using the dynesty sampler.
[3]:
index = 2
data = f"{test_data_dir}/sbi_test_data_BPASS_DenseBasis_v4_final.npz"
import numpy as np
loaded = np.load(data)
X_test = loaded["X"][:, index]
y_test = loaded["y"][:, index]
print(X_test)
[24.33915 21.76936 28.048773 28.16009 28.237339 28.1432 28.001326
28.250029]
We’ll just recreate our prior for the model.
[4]:
fitter.create_priors(set_self=True);
2026-06-24 09:53:18,909 | synference | INFO | ---------------------------------------------
2026-06-24 09:53:18,910 | synference | INFO | Prior ranges:
2026-06-24 09:53:18,910 | synference | INFO | ---------------------------------------------
2026-06-24 09:53:18,911 | synference | INFO | log_mass: 4.43 - 11.83 [log10_Msun]
2026-06-24 09:53:18,911 | synference | INFO | log10metallicity: -3.96 - -1.84 [log10(Zmet)]
2026-06-24 09:53:18,912 | synference | INFO | log10_Av: -2.88 - 0.59 [log10(mag)]
2026-06-24 09:53:18,912 | synference | INFO | log_sfr: -6.10 - 2.98 [log10(Msun/yr)]
2026-06-24 09:53:18,914 | synference | INFO | sfh_quantile_25: 0.16 - 0.77 [dimensionless]
2026-06-24 09:53:18,914 | synference | INFO | sfh_quantile_50: 0.23 - 0.88 [dimensionless]
2026-06-24 09:53:18,915 | synference | INFO | sfh_quantile_75: 0.39 - 0.98 [dimensionless]
2026-06-24 09:53:18,916 | synference | INFO | log10_mass_weighted_age: 2.16 - 2.91 [log10(Myr)]
2026-06-24 09:53:18,916 | synference | INFO | log10_floor_sfr_10: -6.00 - 2.95 [log10_floor(Msun/yr)]
2026-06-24 09:53:18,918 | synference | INFO | log_surviving_mass: 4.23 - 11.67 [log10_Msun]
2026-06-24 09:53:18,918 | synference | INFO | beta: -2.38 - 2.34 [dimensionless]
2026-06-24 09:53:18,919 | synference | INFO | ---------------------------------------------
2026-06-24 09:53:18,922 | synference | INFO | Processing prior...
No we can run the sampler to fit the observation. This is too computationally expensive to run here, so we’ll just show the code you would use.
result = fitter.fit_observation_using_sampler(
observation=X_test,
sampler="dynesty",
sampler_kwargs={"bound": "multi", "sample": "rwalk", "run_kwargs": {"n_effective": 2000}},
plot_name=f"example_{index}",
min_flux_pc_error=0.05,
)
samples = result["samples"]
log_l = result["logl"]
log_w = result["logwt"]
We can sample using our trained SBI model as well for comparison.
[5]:
predicted_params = fitter.sample_posterior(X_test, num_samples=1000)
30s per sample.
Sampling from posterior: 0%| | 0/1 [00:00<?, ?it/s]
2026-06-24 09:53:18,934 | synference | ERROR | Timeout exceeded for sample 0.
Returning empty array for this sample.
Sampling from posterior: 100%|██████████| 1/1 [00:00<00:00, 565.19it/s]