graphcore | python library which allows you to query a computational | Database library

Thanks to @albert I realised the function references need to be on a single line. But then I found another problem when I went back to the full version of the code.

Turns out that the problem is caused by being in a namespace.

The plain function name fn() is auto-linked to a version of the function.

If the arguments are included then no link is generated.

But if there is a namespace comment, then all versions of the function reference generate a link.

The full example is shown below (Run.hpp):

The suggested way to launch docker images that require access to Graphcore IPUs is using the gc-docker command line tool that you can read more about here. This command line tool is available in the Poplar SDK and wraps the system installed docker command line so that you don't need to worry about passing in devices manually like you've shown above.

For interested users you can see what gc-docker is calling under the hood by using the --echo arg, and this is where you will see something similar to what you've posted:

This error usually happens when the model protobuf you pass to the TrainingSession or InferenceSession objects doesn’t contain the loss tensor. A common reason for this is when you call builder->getModelProto() before you add the loss tensor to the graph. To ensure your loss tensor is part of the protobuf your calls should be in the following order:

Illegal instruction means that your program is generating instructions that your CPU can’t handle. The Graphcore TensorFlow wheel is compiled for Skylake class CPUs with the AVX-512 instruction set available, so processors that do not fit the requirements (i.e. a Skylake class CPU with AVX-512 capabilities) will not be able to run Graphcore Tensorflow code. (You can see the requirements in the “Requirements” section of the SDK Overview documentation here).

To see if your processors have AVX-512 capabilities, run cat /proc/cpuinfo and look at the flags field of any of the processors - they should all have the same flags. Here If you don’t see avx512f, your processors don’t fit the Graphcore requirements for running Tensorflow code. Here is an example of what the cat command returns on a machine that fits the requirements (result truncated to one processor):

Sharding consists in partitioning the model across multiple IPUs so that each IPU device computes part of the graph. However, this approach is generally recommended for niche use cases involving multiple models in a single graph e.g. ensembles.

A different approach to implement model parallelism across multiple IPUs is pipelining. The model is still split into multiple compute stages on multiple IPUs; the stages are executed in parallel and the outputs of a stage are the inputs to the next one. Pipelining ensures improved utilisation of the hardware during execution, which leads to better efficiency and performance in terms of throughput and latency, if compared to sharding.

Therefore, pipelining is the recommended method to parallelise a model across multiple IPUs.

You can find more details on pipelined training in this section of the Targeting the IPU from TensorFlow guide.

A more comprehensive review of those two model parallelism approaches is provided in this dedicated guide.

You could also consider using IPUPipelineEstimator : it is a variant of the IPUEstimator that automatically handles most aspects of running a (pipelined) program on an IPU. Here you can find a code example showing how to use the IPUPipelineEstimator to train a simple CNN on the CIFAR-10 dataset.

It is possible to run IPU programs as a non-root user. The reason you're seeing this behaviour is because switching user within a running Docker container (and any Ubuntu based environment) causes environment variables to be reset. These environment variables contain important IPU configuration settings required to attach to and run a program on an IPU. You can avoid this behaviour by instead doing your user management in a Dockerfile. Below is a sample snippet (where examples is a clone of https://github.com/graphcore/examples/):

This failure might be caused by the IPUs being busy running other processes or by an incorrect environment configuration.

1. The IPUs are busy

When you execute a Poplar program (or a framework specific model utilising IPU libraries) you request a certain number of IPUs. If, for instance, you request to run a program with 2 IPUs but somebody else is already using all the IPUs on a chassis, then your program will fail to attach and throw a similar error to the one you’ve seen. For this scenario, you should simply wait until the desired number of IPUs are available. You can verify whether the devices are busy using gc-monitor command line tool (see for reference IPU Command Line tools guide). This is what a busy machine looks like:

The XLA dumping is a TensorFlow native feature. It dumps one file per graph. The number of graphs produced depends on the number of TensorFlow to XLA to HLO modules produced. This can generally be predicted from the number of sess.run calls on distinct graphs you make. For example, if your program contains a variable initialisation then this initialisation will be compiled as a separate XLA graph and appear as a separate graph when dumped. If your program creates a report op, then that will also be compiled as a separate XLA graph.

Typically, ipu_compiler.compile forces the compilation into a single XLA graph. If you don't use ipu_compiler.compile, the native XLA scheduler will combine or split up parts of the TensorFlow graph as it sees fit, creating many XLA graphs - this is why you see far more graphs dumped when not using ipu_compiler.compile.

NOTE: There is no guarantee your compiled op will only produce one XLA graph. Sometimes, others are made, e.g. for casting.

As for the naming, it can be broken down as follows:

module_XXXX.YYYY.IPU.after_allocation-finder.before_forward-allocation.dot

We always have a module_ prefix, which is just to signal that this is the graph for a HLO Module.

The first XXXX is the HLO module's unique ID. There is no guarantee about the spacing between IDs, just that they are unique and increasing.

To understand the rest of the name - YYYY.IPU.......dot - we need to understand that the XLA graph is operated on by multiple different HLO passes, each modifying the XLA graph by optimizing, shuffling or otherwise rewriting it. After these passes, the graph is then lowered to Poplar. There are some TensorFlow native HLO passes, and there are some IPU specific ones. When dumping the XLA graphs, we can render the XLA graph before and after any HLO pass (e.g. to see the pass's effect on the graph) by supplying the argument --xla_dump_hlo_pass_re=XXX, where XXX is regex describing which passes you want. TensorFlow will then render the XLA graph before and after every pass that matches that regex (by its name). For example, if you wanted to see the effect of every XLA HLO IPU pass involving while loops, you could use xla_dump_hlo_pass_re=*While*. Finally, the number YYYY is the ID pertaining to the order in which these graphs are generated, and the passes which the graph was "between" when it was rendered are appended to the filename. The "before_optimizations" graph is always rendered if dumping XLA.

Unfortunately, there is no formal way of knowing which XLA graph is your main program, as the unique ids are somewhat arbitrary and the importance of the contents of each XLA graph is tacit knowledge of the user. The closest approximation possible is likely the file or visual sizes - the main program XLA graph should be much larger than others. As a crude way, you could include a very specific op in your main graph and search for it in the XLA graphs.

You can try something like this, hope that i understood your question