The kine algorithm is a static or dynamic image reconstruction algorithm for interferometric data. It is a forward imaging method, based on a neural field representation⁴³ of the brightness distribution of the source. In this section, we describe our methodology, starting with a general overview of VLBI measurements followed by an explanation of the kine imaging algorithm, which is shown in Fig. 4. Then, we describe the synthetic data used to test the performance of the algorithm and show the corresponding reconstructions. In the Supplementary Information, we present extensive tests validating the dynamic range, resolution, time interpolation and motion tracking of our method on synthetic datasets, of which we report the results here. We also present a performance comparison with the CLEAN method on real observations and the details of the optical flow velocity analysis.

VLBI measurements

In radio interferometric observations, each antenna in the array records a signal proportional to the received electromagnetic flux density. By the van Cittert–Zernike theorem, the time-averaged correlation product of the signals recorded by any pair of antennas (called visibility) is the Fourier transform of the flux density spatial distribution on the sky plane, evaluated at a frequency proportional to the distance of the antennas. For a pair of antennas A and B, with projected baseline vector \(\barFor more tech updates, stay tuned to our blog.=(Check back often for more exciting news!_For more tech updates, stay tuned to our blog.,Keep following us for the latest insights._{y})\), observing at wavelength λ, the ideal complex visibility V_AB(u, v, t) is related to the flux density distribution \({\mathcal{I}}(x,y,t)\) by

where \((u,v)=\left(\frac{{b}_{x}}{\lambda },\frac{{b}_{y}}{\lambda }\right)\) are the x and y spatial frequencies (refer to ref. ⁴⁴ for a detailed description of radio interferometry observations and imaging). The above equation, written for Stokes \({\mathcal{I}}\), holds for all Stokes visibilities \(({V}_{\mathrm{AB}}^{({\mathcal{I}})},{V}_{\mathrm{AB}}^{({\mathcal{Q}})},{V}_{\mathrm{AB}}^{({\mathcal{U}})},{V}_{\mathrm{AB}}^{({\mathcal{V}})})\).

In practice, different sources of noise corrupt the measurement of visibilities. They are classified as baseline-dependent errors and site-dependent errors, so the measured visibilities can be expressed as

where \({G}_{{\rm{A}},{\rm{B}}}{{\rm{e}}}^{{\rm{i}}{\phi }_{{\rm{A}},{\rm{B}}}}\) are the site-dependent errors in amplitude and phase, referred to as complex gains, and ϵ_AB is the thermal noise, which is Gaussian-distributed with baseline-dependent standard deviation⁴⁴. The most problematic source of error is the complex gains, as they might be incorrectly estimated from the a priori calibration, whereas thermal noise can be fully characterized and incorporated in the loss function through the uncertainties σ_AB of the visibilities.

Therefore, although complex visibilities are the fundamental data product resulting from VLBI observations, the imaging process can use different data products that are constructed to be unaffected by site-dependent amplitude or phase corruptions. kine supports the following data products: complex visibilities V_AB, amplitudes of the visibilities |V_AB|, closure phases \({\varPhi }_{\mathrm{ABC}}:= \arg ({V}_{\mathrm{AB}}{V}_{\mathrm{BC}}{V}_{\mathrm{CA}})\), closure amplitudes A_ABCD ≔ |V_ABV_CD“https://www.nature.com/”V_ACV_BD| and the logarithm of closure amplitudes.

Model

kine is a forward imaging algorithm for interferometric data, which models the polarized flux density distribution of a continuous image or video using a neural field⁹, parameterized by weights \({\mathcal{W}}\). The neural field is implemented as a coordinate-based neural network, specifically a multi-layer perceptron (MLP)⁴⁵, which takes spatial coordinates (right ascension x and declination y) and time (t) as input, and outputs the total intensity flux density \(\hat{I}\), fractional linear polarization \({\widehat{m}}_{{\ell }}\), electric vector position angle \(\widehat{\chi }\) and fractional circular polarization \({\hat{m}}_{c}\) at each sky location and time, as shown in Fig. 4:

The hat symbol denotes estimated quantities, as opposed to the true or observed properties of the source. The Stokes parameters are then computed from the polarization quantities \((\hat{{\mathcal{I}}},{\hat{m}}_{{\ell }},\hat{\chi },{\hat{m}}_{c})\) using the transformation:

We compute VLBI measurements from the predicted video using a fully differentiable forward model and optimize the MLP weights by minimizing a χ² data-fit loss between the predicted and observed measurements. Within this forward model, estimated complex visibilities \(\widehat{V}\) and associated data products are obtained by evaluating the 2D Fourier transform of the network-estimated flux density \(\hat{{\mathcal{I}}}\) at the (u, v, t) points corresponding to the observation tracks. In practice, \(\hat{{\mathcal{I}}}\) is evaluated on a discrete set of (x, y, t) points, and the visibilities are therefore computed using a discrete Fourier transform F implemented as a matrix multiplication. For each observation time t_j and u–v point i, the estimated visibilities are

where k runs over all spatial points at which the video is estimated, and both the estimated total intensity and the estimated visibilities are dependent on the network parameters \({\mathcal{W}}\).

MLP networks are known to exhibit a ‘spectral bias’⁴⁶, by which training with gradient descent preferentially captures low spatial frequencies. In the context of VLBI imaging, this behaviour is advantageous, as it acts as an implicit regularizer that promotes smooth reconstructions and suppresses spurious high-frequency structure in both space and time. Moreover, the continuous nature of the neural representation enables the video reconstruction to be evaluated at times when no observations are available, by leveraging correlations learnt from neighbouring frames. This implicit regularization through neural representations has been successfully applied to other ill-posed inverse problems in astrophysical imaging, in which many possible solutions are consistent with a given set of measurements^47,48,49.

Optimization

We formulate the optimization as a noise-weighted data-fitting problem. Assuming Gaussian noise residuals in the data products, which is valid for visibilities and amplitudes and for closure quantities in the high-SNR regime, the natural loss function \({\mathcal{L}}\) is given by the sum of the χ² contributions from the selected data products:

where D indicates a generic data product, the index j runs over all the observed times, from 1 to the total number of observed times N_t; the index i runs over all data corresponding to the observed time t_j, from 1 to the total number of data N_D,j; and k_D is a normalization factor that takes into account the degrees of freedom of the data product. Any combination of one or more kinds of data products can be used in the loss function, and the loss function terms associated with each data product are weighted equally.

The optimization process consists of iterating the following steps until convergence. First, the algorithm estimates the video, with the current network parameters \({{\mathcal{W}}}_{k}\), on a coordinate grid with x and y regularly spaced and t spaced according to the observation times. Then it computes the loss function from the estimated video and the observed data. Finally, it updates the MLP parameters to \({{\mathcal{W}}}_{k+1}\), by applying a gradient descent optimization step of the loss function. More specifically, we use the Adamax optimizer⁵⁰. In this work, convergence is considered to be reached when the moving average of the loss function over a window of 100 iterations does not decrease by more than 1% for at least 1,000 iterations, provided that all data products in the loss function have reached a value of about 1. This convergence was achieved after approximately 2 × 10⁴ iterations, a value that we finally set as the total number of iterations.

Calibration-free imaging details

When imaging a dataset with no gain corruption, the best data products to use are complex visibilities, because they contain the full amount of available information. In that case, the imaging procedure in kine consists of running the optimization algorithm estimating all Stokes parameters simultaneously. However, if the dataset contains amplitude and/or phase gains, imaging must be performed using closure quantities. When imaging with closure or log-closure amplitudes, the information about the total flux of the frames is lost so a total flux regularizer term is added to the loss function, constraining the total flux in each frame to a value either provided by the user or computed from the instantaneous visibility of the shortest baseline. When imaging with closure phases, the information about the absolute location of the source in the frame is lost, so the reconstructed movie may show the source drifting smoothly across the frame. We solve this issue post-imaging by re-aligning the frames to a chosen feature, such as a jet core. The alignment procedure is detailed in the Supplementary Information.

The current version of kine does not model station gains simultaneously with the imaging. For the total intensity reconstruction, this problem is solved by relying on closure quantities. The closure quantities for the \({\mathcal{Q}}\) and \({\mathcal{U}}\) signals, however, are gain independent only in the absence of polarization leakage⁵¹, but, even in that case, the low SNR of the \({\mathcal{Q}}\) and \({\mathcal{U}}\) data, combined with the loss of information due to the use of closure quantities, may result in insufficient data constraints. Therefore, we found that the most effective procedure to produce a full-polarimetric video, in the presence of gain corruption, is to first image the data in total intensity only, using the appropriate closure quantities, then self-calibrate the data to the total intensity video reconstruction, and finally, image the self-calibrated data in full Stokes, using complex visibilities.

The kine imaging method was developed for video reconstruction of a variable source, but it can reduce to static imaging by removing the time dimension from the input coordinates.

Architecture

kine is implemented in Python using the JAX deep learning framework⁵². We use a four-layer MLP for total intensity imaging and a deeper six-layer MLP for full-Stokes imaging of the self-calibrated data, both with 256 nodes per layer and a residual skip connection⁵³ between the first and the last layers. The node outputs in all layers except the last are normalized with BatchNorm⁵⁴ across the full set of input coordinates to have zero mean and unit standard deviation. Hidden layers use the Gaussian Error Linear Unit activation⁵⁵, whereas the output layer uses SoftPlus for \(\hat{{\mathcal{I}}}\) and a sigmoid for \({\widehat{m}}_{{\ell }}\), \({\widehat{m}}_{c}\) and \(\widehat{\chi }\). Standard interferometry functions are handled by the eht-imaging library⁷. During optimization, spatial input coordinates are sampled on a regular grid, whereas the time coordinates follow the observation timestamps. For the imaging of 3C 345, we used 200 × 200 spatial input points with 75 μas spacing.

The runtime depends on the number of visibilities and epochs, the field-of-view to beam-size ratio, and the data products used. Optimizing the network on 116 epochs of 3C 345 data from the VLBA took approximately 1.3 h on four NVIDIA A100 GPUs, whereas imaging 4 days of M 87* EHT observations took around 20 min on a single A100.

Initialization

When the (u, v)-coverage of the VLBI data is sufficiently dense, such as for VLBA observations, kine is able to reconstruct the image or video starting from a random initialization of the network weights. However, if imaging with closure phases, it may be useful to initialize the network to a simple shape, such as a Gaussian, to constrain the image to the correct location in the frame. We found that, although it is important that the initialization image has the correct total flux and covers approximately the area in which most of the flux of the source should be located, the specific shape of the initialization image does not affect the resulting image or video obtained when convergence is reached.

The initialization is carried out by optimizing the network directly on the initialization image \({{\mathcal{I}}}_{\mathrm{init}}(x,y)\), with a pixel-to-pixel distance loss function:

without the computation of the image Fourier transform or other data products. After the weights of the network are optimized to output the initialization image, the optimization on the data products begins.

Synthetic data validation

We validated the algorithm on multiple synthetic datasets prepared to resemble the morphology, coverage and noise of the real 3C 345 data. The first ground truth video consists of a geometric jet model in which the emission direction of the jet is precessing along a circular orbit, emitting ballistic components, which expand and decrease in brightness as they move away from the core. This results in a model with a continuous winding structure, which we refer to as model1. The second model, model2A, is constructed in a similar way, but with longer spacing between the emitted components, so that the model is composed of discrete, separate components emitted at different angles. The components have an initial full width at half maximum (FWHM) = 80 μas, which increases slightly as they move away from the core at a constant apparent speed of β = 11.4. The third model, model3A, is similar to the second but with the addition of a parabolic jet profile as a background to simulate a more realistic jet structure. We used these three models for the main validation tests, but we also prepared variations of model2A and model3A, with components of different sizes—FWHM = 60 μas for model2B and model3B and FWHM = 160 μas for model2C—and moving at different speeds—β = 22.7 for model2D and β = 5.7 for model2E—to test the resolution and motion recovery. For each observed epoch, we simulated a synthetic dataset from the model, using the corresponding real (u, v)-coverage and applying the same amount of thermal noise. We did not need to introduce complex gain corruption, because the closure quantities used in the first step of the imaging process are invariant under gain errors.

We imaged the synthetic data with the same parameters used for real data. Here, as an example, we present and discuss the reconstructions of model1, model2A and model3A. The ground truth videos are shown in the first, third and fifth rows of Extended Data Fig. 3, whereas the reconstructions of the synthetic data are in the second, fourth and sixth rows of the same figure. These show that kine is able to recover the models with high fidelity, which is reflected by the high values of the average normalized cross-correlation between the ground truth frames and the reconstructed ones (Extended Data Table 1).

In the Supplementary Information, we present the details of additional validation tests that quantify the dynamic range, super-resolution, component characterization, time interpolation and motion tracking achieved by our method, and we report the quantitative results of the tests in the subsequent sections. The improvements in reconstruction quality achieved by kine over traditional imaging methods, especially the dynamic range and resolution improvements, are dependent on the quality and quantity of the observations, so they should not be intended as intrinsic performance gains independent of dataset quality and temporal coverage.

Forward-modelling imaging algorithms can achieve higher resolution than traditional inverse methods by incorporating prior information. From the validation tests, we see that kine dynamic reconstructions have effective resolutions between about 100 μas and 125 μas, with an average of 113 μas, whereas kine static reconstructions have effective resolutions between about 110 μas and 145 μas, with an average of 125 μas. This shows that the main resolution improvement is achieved already at the static reconstruction level, with an improvement factor of 3.8 over the nominal resolution of 475 μas, whereas the dynamic reconstruction provides a marginally higher improvement factor of 4.2.

The dynamic range improvement of kine over traditional methods is a result of the combination of two factors: the resolution improvement of kine and the simultaneous imaging of multiple epochs. From the validation tests, we conclude that kine static reconstructions achieve a dynamic range between 6 × 10³ and 8 × 10⁴, whereas dynamic ones achieve a dynamic range from 2 × 10⁵ to 7 × 10⁵. This shows that dynamic imaging provides a higher dynamic range than static imaging. In this case, the improvement factor is of the order of about 10, which is compatible with the \(\sqrt{N}=\sqrt{116}\simeq 11\) limit.

We tested the ability of kine to correctly resolve and locate small discrete components in the core region by fitting a double 2D Gaussian model on kine reconstructions. The tests show that components can be precisely located starting at about 110 μas from the core and tracked throughout their whole path. For a component fully detached from the core, the position angle can be recovered within 1° of the true value, whereas for those connected to the core, it is recovered correctly within 7°.

We tested the limit of the velocity that could be recovered with kine combined with the optical flow, and we conclude that we can accurately measure apparent velocities in the 3C 345 dataset up to at least approximately 23c, which exceeds the maximum speeds observed in the real data.

Owing to the continuous nature of neural field representations, kine reconstructions can be sampled at any arbitrary time, independent of the observed epochs. For data with the coverage and quality of 3C 345 observations, kine provides good, motion-preserving time interpolation for frames up to 6 months apart from the nearest observations. In the present work, we use time-interpolated frames in the optical flow analysis to have a video sampled at regular intervals. The longest separation between an interpolated frame and the nearest observation is 5.7 months, which is within the range of reliable interpolation.

Comparison of CLEAN and kine

As shown from the validation tests, even when the instantaneous coverage is sufficient for static imaging, dynamic reconstructions achieve higher resolution and dynamic range than static ones. Here, we estimate the effective resolution and dynamic range of the static and dynamic reconstructions of kine of the real 3C 345 observations and compare them with images obtained with CLEAN. Supplementary Fig. 8 shows an example frame comparing traditional static imaging with CLEAN, snapshot static imaging with kine and dynamic imaging with kine. Both kine reconstructions show consistency with the CLEAN image, although they are able to resolve considerably more structure in the jet. As we do not have a ground truth in this case, we can estimate the dynamic range only with the traditional definition, while we estimate the resolution using the power spectrum comparison with the CLEAN model convolved with the nominal beam.

We obtain an average dynamic range of 3.6 × 10³ for CLEAN, 4.9 × 10⁴ for kine static and 5.1 × 10⁵ for kine dynamic, resulting in an improvement of one order of magnitude for kine static over the traditional CLEAN imaging method and two orders of magnitude for kine dynamic. The dynamic ranges of kine static have a wide distribution of values, whereas that of kine dynamic varies less. This means that kine dynamic is effective in improving the reconstruction quality of epochs with lower data quality by leveraging the information from previous and subsequent epochs.

We obtain a resolution of 141 μas for kine static and a resolution of 106 μas for kine dynamic, representing an improvement factor of 3.4 for kine static and 4.5 for kine dynamic with respect to CLEAN, whose average nominal resolution of 475 μas is set by the diffraction limit. These results are consistent with the estimates from the tests on synthetic data, although we consider the resolution estimate from real data slightly less reliable than that from synthetic data because the comparison is not carried out with the ground truth but rather with the CLEAN reconstruction. Therefore, we prefer to rely on the more conservative estimate from synthetic data for our conclusions.