Closing the socket connection

[drewbyron]

Hello mortenjc,

I am using your udp receiver (thanks!) but am having an issue where things seem to be left in the socket buffer. How can one close the receiver? For example one can typically call close(socket); but I can't get the syntax to work in this context. Any help would be greatly appreciated.

Best,
Drew

[mortenjc]

Hi Drew,

Thanks for reaching out.

It has been a while since I last paid this repo any attention. I will have to have a look at the code. But if you want to tell me more about your usage and what platform you are using then that might help speed things up.

Just out of curiosity how did you find this code in the first place?

Cheers
Morten

[mortenjc]

Just a quick followup.

There was no Close() method for the Socket wrapper class. I've added one now and I am now closing the socket in the UDPReceiver destructor. But it shouldn't be necessary because when the process is stopped the relevant file descriptors are automatically closed.

The intended use case is to let transmitter and receiver loop forever or until manually stopped.

Then stop the transmitter first and the receiver next.

There is a new branch that you could try and build where I've added the Close() method which is now called
from the destructor, but you could also call it explicitly if you are modifying the program for your needs.

Example from udprx.cpp woud be: Receive.Close();

Let me know if any of this is useful.

Best
Morten

[drewbyron]

Hello Morten, Thanks for the reply! Yes, this is very useful. One thing is that in the new branch your Socket.h file needs: `#include <unistd.h>` to build. My desire is to build a receiver that writes ~150k 4128-byte UDP packets to a single file (this corresponds to 1s of data) and then does that for many files. I want the individual files to be filled with consecutive packets but the different files need not be consecutive. The issue I am running into is that the first file is fine, with no dropped packets (I can tell because the transmitter assigns consecutive packet numbers). But all subsequent files have many sporadically dropped packets. I tried to address that by closing the socket after each file was filled to see if that would empty the socket buffer but it doesn't seem to help. I attach my implementation for context. Any advice would be greatly appreciated. I am new to this type of thing (as may be evident!). I am a graduate student at the University of Washington CENPA (experimental nuclear/particle physics). A postdoc in my group found this repo, not sure how. Thanks in advance for any advice, Drew

…

On Tue, Apr 5, 2022 at 11:47 PM Morten Jagd Christensen < ***@***.***> wrote: Just a quick followup. There was no Close() method for the Socket wrapper class. I've added one now and I am now closing the socket in the UDPReceiver destructor. But it shouldn't be necessary because when the process is stopped the relevant file descriptors are automatically closed. The intended use case is to let transmitter and receiver loop forever or until manually stopped. Then stop the transmitter first and the receiver next. There is a new branch that you could try and build where I've added the Close() method which is now called from the destructor, but you could also call it explicitly if you are modifying the program for your needs. Example from udprx.cpp woud be: Receive.Close(); Let me know if any of this is useful. Best Morten — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKOEW2EMHNVQIZMDGTNUGNLVDUXPPANCNFSM5SUQA7AQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[mortenjc]

Ok thanks, I built it on my Mac and there it didn't complain about the missing header file so I am guessing
you are building on some Linux flavor.

From what I hear the problem is probably not in the receiver/transmitter code. After all I was able to demonstrate
0% packet loss at 10Gbit/s. There are some necessary tweaks though for good performance. I recommend you to
read or skim my paper: https://iopscience.iop.org/article/10.1088/1748-0221/15/09/T09005

PS I couldn't immediately see any attached files.

Cheers
M

[drewbyron]

Hey Morten,

Yes, our DAQ runs Linux. The attached paper was very useful, thanks. I edited the net.core parameters on the daq and that seems to have helped a bit. One question I have is about the optimization of the intermediate buffer size. In your example, I see that the buffer size is larger than the datasize (static const int BUFFERSIZE{9200}; but int DataSize{9000};), is that an important step? My second question is about choosing the SocketBufferSize. I got better performance when I made this very large (larger than the total amount of data I am trying to write out to disk at one time) but this seems like it could be poor practice. How do you choose that parameter?

Here is a snippet from my implementation. If you wouldn't mind taking a look and seeing if there are any glaring issues I would greatly appreciate it.

struct {
  int UDPPort{4003};
  int DataSize{5000};
  int SocketBufferSize{700000000};
} Settings;

int main(int argc, char *argv[4]) {

  // Set this to highest priority: 
  setpriority(PRIO_PROCESS, 0, -20);

  char *nPtr = NULL;

  const int BUFFERSIZE = strtol(argv[1], &nPtr, 10); //Size of packets
  int tot_packets = strtol(argv[2], &nPtr, 10); //number of packets to be collected per acquisition
  int drive = strtol(argv[3], &nPtr, 10);

  int outputs = 0; //index variable for # of output files written

  const int B1M = 1000000;               //bytes per MB
  char buffer[BUFFERSIZE];               //buffer is declared as chars b/c they're 1 byte each

  uint64_t RxBytes{0};                   //bytes recieved in last interval
  uint64_t RxPackets{0};                 //packets received in last interval

  auto t = std::time(nullptr);           //time object for output file naming
  auto tm = *std::localtime(&t);         //system time object for output file naming
  std::ostringstream outFileStream;      //string for building output file name

  std::vector<char> sdb_datablock(700000000); //the large buffer for holding packets
  vector<char>::iterator sdb_iter = sdb_datablock.begin(); //iterator for navigating buffer vector

  std::chrono::milliseconds downtime(1000); // Currently 1s. How long to rest between threads. 

  int rec_time_s{1};
  int rec_time_us{0};

  //Set up socket and receiver to listen on it
  Socket::Endpoint local("10.66.192.33", 4003);

  // Set up Receiver. 
  UDPReceiver Receive(local);
  Receive.setBufferSizes(Settings.SocketBufferSize, Settings.SocketBufferSize);
  Receive.setRecvTimeout(rec_time_s, rec_time_us);
  while(RxPackets < tot_packets){
      //pull individual packet off the network interface

      int ReadSize = Receive.receive(buffer, BUFFERSIZE);

      if (ReadSize > 0) {
        //copy bytes from the buffer to the memmory position indicated by iter
        std::copy (buffer, buffer+BUFFERSIZE, sdb_iter);
        sdb_iter += BUFFERSIZE;  //advance the iterator to prep for buffering the next packet

        //increment counters for performance monitoring
        RxBytes += ReadSize;
        RxPackets++; 
      }

    } //end of while(RxPackets < tot_packets)

    // Close the socket connection. 
    Receive.Close();

Thanks!
Drew

[drewbyron]

BUFFERSIZE = 4128 (commandline argument)
One more comment, I then write the large data block to a file. I couldn't figure out a way to write directly into the large block (sdb datablock). Is that possible and would that result in better performance?

[mortenjc]

Hi Drew

I had a quick look and have a few comments.

Hope it can help a little.

Cheers
Morten

The socket buffer size should be set to 12M or so which was enough to not lose packets
at 10G speeds. I assume you are operating well below this. It is not related to the size
of the final buffer that you copy your data into.

It says in the comments that BUFFERSIZE is the number of files to write. However it is used
to create a buffer that you later use for packet reception. This confuses me as these two things
have no obvious relation.

The call
Receive.receive(buffer, BUFFERSIZE);

Receives at most BUFFERSIZE bytes of data. But can receive less. If you receive less,
the copying that you do with

std::copy (buffer, buffer+BUFFERSIZE, sdb_iter);

Will leave gaps in the sdb_iter vector.

3. To debug this you should maybe start by adding a
   printf("Rx packet size %d\n", ReadSize);
   after Receive.receive() or at least

assert(ReadSize == BUFFERSIZE); // requires #include <assert.h>

which will catch the cases where that assumption does not hold.

Generally you should use assert() as much as you can (general programming hint). It will
help you catch bugs early.

I would personally use a preallocated char buffer and initialize it to zero in the beginning
to start with clean buffer.

char finalbuffer[700'000'000];
memset(finalbuffer, 0, sizeof(finalbuffer));

I would then avoid using iterators so that after each receive you can copy
the data from rxbuffer to the final buffer

int ReadSize = Socket.receiver(buffer, RXBUFSIZE);
assert(ReadSize == RXBUFSIZE);
memcpy(finalbuffer + writeoffset, buffer, ReadSize)
writeoffset += ReadSize;

This will ensure that data is being written back to back without gaps.

[drewbyron]

Hey Morten, That was very helpful! Point 1) was just a typo. I am getting better performance now, but it seems like the copy operation from the small buffer to the large memory block is still a bottleneck. Is it possible to receive packets and write them directly to the large memory block? If not, is there a reason this is bad practice? Best, Drew

…

On Thu, Apr 7, 2022 at 8:36 AM Morten Jagd Christensen < ***@***.***> wrote: Hi Drew I had a quick look and have a few comments. Hope it can help a little. Cheers Morten 1. The socket buffer size should be set to 12M or so which was enough to not lose packets at 10G speeds. I assume you are operating well below this. It is not related to the size of the final buffer that you copy your data into. 1. It says in the comments that BUFFERSIZE is the number of files to write. However it is used to create a buffer that you later use for packet reception. This confuses me as these two things have no obvious relation. 1. The call Receive.receive(buffer, BUFFERSIZE); Receives *at most* BUFFERSIZE bytes of data. But can receive less. If you receive less, the copying that you do with std::copy (buffer, buffer+BUFFERSIZE, sdb_iter); Will leave gaps in the sdb_iter vector. 1. To debug this you should maybe start by adding a printf("Rx packet size %d\n", ReadSize); after Receive.receive() or at least assert(ReadSize == BUFFERSIZE); // requires #include <assert.h> which will catch the cases where that assumption does not hold. Generally you should use assert() as much as you can (general programming hint). It will help you catch bugs early. 1. I would personally use a preallocated char buffer and initialize it to zero in the beginning to start with clean buffer. char finalbuffer[700'000'000]; memset(finalbuffer, 0, sizeof(finalbuffer)); 1. I would then avoid using iterators so that after each receive you can copy the data from rxbuffer to the final buffer int ReadSize = Socket.receiver(buffer, RXBUFSIZE); assert(ReadSize == RXBUFSIZE); memcpy(finalbuffer + writeoffset, buffer, ReadSize) writeoffset += ReadSize; This will ensure that data is being written back to back without gaps. — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKOEW2D4XHKOK5EXVATDQI3VD36HFANCNFSM5SUQA7AQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[mortenjc]

Hi Drew
It is not bad practice at all. Initially I would do that just to separate things and make it work. Obviously the
extra copy adds overhead and can easily be avoided.

When you have verified it works correctly, just replace the buffer in Receive() with the
offset into the final buffer and omit the copy. That should speed things up.

Personally i would also count the received packets and received bytes and periodically print out the rates. This is really
good for building up intuition about the nature of the received data.