carmen | A repository of geographic regions for Ruby

I have two large CSV with data that I want to compare. I used pandas therefore I have two data frames to work with easier, but the program takes too long to finish and compare all the data.

I am comparing the data sent with the received, in order to get the latency time, for that I put a double loop and the program works fine. But I want to know if there is a faster way to do this, because for my heaviest files it takes days to finish the program.

I am working with large files, the first has 68001 rows and the second has 837190 rows. But the program takes too long. Thanks in advance.

Explanation of how my code works

I am doing some performance tests of the MQTT broker, for which I created the paho clients that send and receive messages, the data sent was stored in a csv to later calculate the latency. The csv of the Publishers (users who publish) contains the publisher's client ID, the account, the timestamp, and the topic to which it is subscribed.

While the subscribers (users who receive the message) contain the timestamp when the message is received, the message received, the count (counter for the number of messages), the publisher's client ID and the topic.

Now to calculate the latency, evaluate row by row with a loop that starts at 0. That's why I used a loop for df1 (dataframe of the publishers) and the first row of the df2 (dataframe of the message receivers).

With the conditional "if" I compare the client's ID, the count number, and the topics where the messages were sent from the first row of the dataframe df1, which corresponds to the first client to send a message and I compare it with the first row of the dataframe df2, to see if the client ID, count and topic match.

If it coincides, I proceed to subtract the times to calculate the Latency, locating myself in the column corresponding to the times, and then store them in another csv, which is almost the last line you can see.

If the conditional is not fulfilled, continue and goes to the next iteration of the loop corresponding to the df2. Therefore df1 will remain in the same position until the df2 loop has finished evaluating if there are matches in all the lines of its dataframe. I hope I have explained myself well.