网易云信的一篇文章, 里面有干货.

1

https://zhuanlan.zhihu.com/p/37538078

udp connect

这篇文章讲的是webrtc SFU的多线程架构. 其中讲到了UDP fd connect的用法，让我涨见识了.

UDP fd connect用于服务端多线程架构中，可以不依赖于主线程做数据分发，减低单一线程负载，并且将对端同一ip端口的数据包收取到同一线程中， 类似于TCP一个fd一个连接.

udpserver

udp_server_test是我用c实现的一个简单示例，验证这篇文章讲的线程架构.

server端

  • main thread创建udp fd, 并绑定7000端口
  • main thread创建4个worker, 并创建4对socketpair, 分别用于与4个worker通信
  • woker thread创建epollfd, 将socketpair fd加入epoll中监听读事件
  • main thread创建epollfd, 将udp fd加入epoll中监听读事件
  • 有客户端udp packet到来时，根据客户端ip端口hash，选定一个worker
  • 将udp packet和它的ip port信息通过socketpair fd投递到woker线程中
  • worker线程接收到main thread投递的udp packet, 创建一个新的udp fd, 也绑定7000端口
  • 将创建的udp fd通过connect调用绑定这个udp packet的源ip port
  • 后续同一ip port来源的udp packet将直接被内核送到worker创建的那个udp fd里

client端

  • 创建4个udp fd，分别绑定4个端口
  • 分别使用它们发送消息到服务端7000端口, 每个fd发送内容包括固定的字符，源端口和发送计数
  • 重复发送3次

测试结果

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

[thread: 80176, file: server.c, line:   120] worker start, id: 0

[thread: 80177, file: server.c, line:   120] worker start, id: 1

[thread: 80178, file: server.c, line:   120] worker start, id: 2

[thread: 80179, file: server.c, line:   120] worker start, id: 3

[thread: 80175, file: server.c, line:   232] main thread, msg from remote port: 50000, msg: (content: A, client_port: 50000, count: 0), hash to thread: 0
[thread: 80176, file: server.c, line:    96] connect with remote addr, (workerid: 0 <---> fd: 17 <---> remote_port: 50000), msg from main thread, msg: (content: A, client_port: 50000, count: 0)
[thread: 80175, file: server.c, line:   232] main thread, msg from remote port: 50001, msg: (content: B, client_port: 50001, count: 0), hash to thread: 1
[thread: 80177, file: server.c, line:    96] connect with remote addr, (workerid: 1 <---> fd: 18 <---> remote_port: 50001), msg from main thread, msg: (content: B, client_port: 50001, count: 0)
[thread: 80175, file: server.c, line:   232] main thread, msg from remote port: 50002, msg: (content: C, client_port: 50002, count: 0), hash to thread: 2
[thread: 80178, file: server.c, line:    96] connect with remote addr, (workerid: 2 <---> fd: 19 <---> remote_port: 50002), msg from main thread, msg: (content: C, client_port: 50002, count: 0)
[thread: 80175, file: server.c, line:   232] main thread, msg from remote port: 50003, msg: (content: D, client_port: 50003, count: 0), hash to thread: 3
[thread: 80179, file: server.c, line:    96] connect with remote addr, (workerid: 3 <---> fd: 20 <---> remote_port: 50003), msg from main thread, msg: (content: D, client_port: 50003, count: 0)
[thread: 80176, file: server.c, line:    65] msg from remote 127.0.0.1:50000, fd: 17, msg: (content: A, client_port: 50000, count: 1)
[thread: 80177, file: server.c, line:    65] msg from remote 127.0.0.1:50001, fd: 18, msg: (content: B, client_port: 50001, count: 1)
[thread: 80178, file: server.c, line:    65] msg from remote 127.0.0.1:50002, fd: 19, msg: (content: C, client_port: 50002, count: 1)
[thread: 80179, file: server.c, line:    65] msg from remote 127.0.0.1:50003, fd: 20, msg: (content: D, client_port: 50003, count: 1)
[thread: 80176, file: server.c, line:    65] msg from remote 127.0.0.1:50000, fd: 17, msg: (content: A, client_port: 50000, count: 2)
[thread: 80177, file: server.c, line:    65] msg from remote 127.0.0.1:50001, fd: 18, msg: (content: B, client_port: 50001, count: 2)
[thread: 80178, file: server.c, line:    65] msg from remote 127.0.0.1:50002, fd: 19, msg: (content: C, client_port: 50002, count: 2)
[thread: 80179, file: server.c, line:    65] msg from remote 127.0.0.1:50003, fd: 20, msg: (content: D, client_port: 50003, count: 2)

第一次发送都发到了主线程，主线程根据ip port分别hash到了4个worker线程

第二次第三次发送都直接发送到了work线程创建的fd里.

源码：

udp_server_test

update 2021

开源的多线程媒体服务器ZLM就是多线程架构的，其中webrtc SFU功能就是基于多线程udpserver的，他也使用了UDP connect.