keras-self-attention | Attention mechanism for processing sequential data | Machine Learning library

One approach is to fetch the outputs of SeqSelfAttention for a given input, and organize them so to display predictions per-channel (see below). For something more advanced, have a look at the iNNvestigate library (usage examples included).

Update: I can also recommend See RNN, a package I wrote.

• input_data = single batch of data of shape (1, input_shape)
• prefetched_outputs = already-acquired layer outputs; overrides input_data
• max_timesteps = max # of timesteps to show
• max_col_subplots = max # of subplots along horizontal
• equate_axes = force all x- and y- axes to be equal (recommended for fair comparison)
• show_y_zero = whether to show y=0 as a red line
• channel_axis = layer features dimension (e.g. units for LSTM, which is last)
• scale_width, scale_height = scale displayed image width & height
• dpi = image quality (dots per inches)

Visuals (below) explanation:

• First is useful to see the shapes of extracted features, regardless of magnitude - giving information about e.g. frequency contents
• Second is useful to see feature relationships - e.g. relative magnitudes, biases, and frequencies. Below result stands in stark contrast with image above it, as, running print(outs_1) reveals that all magnitudes are very small and don't vary much, so including the y=0 point and equating axes yields a line-like visual, which can be interpreted as self-attention being bias-oriented.
• Third is useful for visualizing features too many to be visualized as above; defining model with batch_shape instead of input_shape removes all ? in printed shapes, and we can see that first output's shape is (10, 60, 240), second's (10, 240, 240). In other words, the first output returns LSTM channel attention, and the second a "timesteps attention". The heatmap result below can be interpreted as showing attention "cooling down" w.r.t. timesteps.

SeqWeightedAttention is a lot easier to visualize, but there isn't much to visualize; you'll need to rid of Flatten above to make it work. The attention's output shapes then become (10, 60) and (10, 240) - for which you can use a simple histogram, plt.hist (just make sure you exclude the batch dimension - i.e. feed (60,) or (240,)).